Biosurveillance: DHS Should Not Pursue BioWatch Upgrades or Enhancements Until System Capabilities Are Established
Released: 2015-11-23T00:00:00

Letter

Background

How the BioWatch Program Works

Currently, the BioWatch program collaborates with more than 30 BioWatch jurisdictions throughout the nation to operate approximately 600 Gen-2 aerosol collectors. These units rely on a vacuum-based collection system that draws air through a filter. These filters are manually collected and transported to state and local public health laboratories for analysis using a process called polymerase chain reaction (PCR). Sometimes also called molecular photocopying, PCR is a technique used to amplify (or copy) segments of deoxyribonucleic acid (DNA), the building blocks of genetic material. By targeting specific segments of genetic material, PCR can be used as the basis for a test, or assay, for the presence of genetic signatures associated with specific biological organisms, such as the five BioWatch threat agents. (The program monitors for six distinct biothreat agents, but two of these are closely related, although they cause different diseases, and the BioWatch program has treated them as a single agent. For consistency, we will treat them as a single agent and report that there are five BioWatch threat agents in total.) In the BioWatch Gen-2 system, multiple PCR assays are used for each threat agent. In an initial “screening” step, one assay is run for each threat agent. If any of these assays yields a positive result, suggesting the presence of one of the threat agents, then the analysis proceeds to a “verification” step in which multiple additional assays are run targeting different genetic signatures for that agent. If the verification step also yields a positive result, then a BioWatch Actionable Result (BAR) is declared. Using this manual process, the determination of a BAR can occur from 12 to 36 hours after an agent is initially captured by the aerosol collection unit. This 36-hour timeline consists of up to 24 hours for air sampling, up to 4 hours for retrieving the sample from an aerosol collection unit and transporting it to the laboratory, and up to 8 hours for laboratory testing.

Each BioWatch jurisdiction has either a BioWatch Advisory Committee or equivalent decision-making group in place, composed of public health officials, first responders, and other relevant stakeholders. The BioWatch Advisory Committee is responsible for the day-to-day BioWatch operations, including routine filter collection and laboratory analysis of filter samples. In the event of a BAR, the BioWatch Advisory Committee, in partnership with OHA and other stakeholders, is also responsible for determining whether that BAR poses a public health risk and deciding how to respond. The declaration of a BAR does not necessarily signal that a biological attack has occurred. BARs have been triggered by biological agents that occur naturally in numerous areas of the United States. From 2003 through 2014, 149 BARs were declared, but none was linked to an attack or to a public health threat. For a more detailed discussion of this issue, see appendix II. Figure 1 shows the process that local BioWatch jurisdictions are to follow when deciding how to respond to a BAR.

A History of BioWatch

In cooperation with other federal agencies, DHS created the BioWatch program in 2003. The goal of BioWatch is to provide early warning, detection, or recognition of a biological attack. When DHS was established in 2002, a perceived urgency to deploy useful—even if immature—technologies in the face of potentially catastrophic consequences catalyzed the rapid deployment of many technologies. In the initial deployment of BioWatch—known as Generation-1—DHS deployed aerosol collectors to 20 major metropolitan areas, known as BioWatch jurisdictions, to monitor primarily outdoor spaces. DHS completed the initial deployment quickly—within 80 days of the President’s announcement of the BioWatch program in his 2003 State of the Union Address. To accomplish this rapid deployment, DHS adapted an existing technology that was already used for other air monitoring missions. In 2005, DHS expanded BioWatch to an additional 10 jurisdictions, for a total of more than 30. The expanded deployment— referred to as Gen-2—also included the addition of indoor monitoring capabilities in three high-threat jurisdictions and provided additional capacity for events of national significance, such as major sporting events and political conventions. The technology used in Gen-1 and Gen-2 was deployed rapidly and, according to the National Academies in 2011, without sufficient testing, validation, and evaluation of its technical capabilities.

To reduce the time required to detect biothreat agents, DHS began to develop autonomous detection capability in 2003 for the BioWatch program—known as Gen-3. Envisioned as a laboratory-in-a-box, the autonomous detection system would automatically collect air samples, conduct analysis to detect the presence of biothreat agents every 4 to 6 hours, and communicate the results to public health officials via an electronic network without manual intervention. By automating the analysis, DHS anticipated that detection time could be reduced to 6 hours or less, making the technology more appropriate for monitoring indoor high-occupancy facilities such as transportation nodes and enabling a more rapid response to an attack. DHS also anticipated a reduction in operational costs by eliminating the program’s daily manual sample retrieval and laboratory analysis. In 2008, DHS OHA initiated a competitive bid process for the first testing phase of the Gen-3 acquisition, known as Gen-3 Phase I. Five vendors responded to the request for proposal, and DHS awarded contracts to two, for technologies known as the Bioagent Autonomous Network Detector (BAND, later named M-BAND) and the Next Gen Automated Detection System (NG- ADS).

From May 2010 through June 2011, the BioWatch program conducted Phase I testing on these candidate Gen-3 technologies. The testing goals included characterizing the state of available autonomous detection technology on the market and evaluating the candidate systems’ abilities to meet performance requirements developed by the BioWatch program. The Phase I testing consisted of testing of individual system components, such as the aerosol sampling component (the component that collects particles from the air) and the analytical subsystem (the component that detects and identifies biothreat agents), whole system chamber testing, and an operational field test in a BioWatch jurisdiction. Characterization testing did not demonstrate the system’s end-to-end ability to detect the five BioWatch threat agents in an operational environment because these agents cannot be released into the air in such environments.

Expressing concern in 2011 about the rigor of DHS’s effort to help guide its Gen-3 decision making, Members of the Congress asked us to examine issues related to the Gen-3 acquisition. We released a report that evaluated the acquisition decision-making process for Gen-3 in September 2012. We recommended that before continuing the Gen-3 acquisition, DHS should carry out key acquisition steps, including reevaluating the mission need and systematically analyzing alternatives based on cost-benefit and risk information. DHS subsequently commissioned an analysis of alternatives, which was interpreted by DHS as showing that any advantages of an autonomous system over the current manual system were insufficient to justify the cost of a full technology switch. DHS’s April 24, 2014, ADM announced the cancellation of the Gen-3 acquisition and made Gen-2 the official program of record for aerosol biological threat detection. The ADM also directed S&T to explore development and maturation of an effective and affordable automated aerosol biodetection capability, or other operational enhancements, that meets the operational requirements of the BioWatch system.

Methods and Measures for Assessing System Performance

The capabilities of the BioWatch system can be assessed at three different levels (fig. 2). At the highest level, BioWatch consists of an array of aerosol collectors deployed in an operational environment and the associated laboratory processes for analyzing samples. The operational environment might be outdoors, as in a metropolitan area (shown in the fig.); indoors, as in an airport; or a subway or other transit system. At this level, the capability of the system to detect an attack depends on factors that include the performance characteristics of the technology (including the aerosol collector and the technology used for laboratory analysis of samples), the number and locations of the collectors, the location of an attack (that is, where a biothreat agent is released into the air), and wind patterns (for an outdoor attack).

At the next level is the detector, which consists of the aerosol collector unit and the process by which samples collected by this unit are transferred to a laboratory and analyzed. Performance and effectiveness at this level depend on the technical performance characteristics of the aerosol collector itself, the extent to which the sample is preserved intact during the collection cycle and during transport to the laboratory, and the laboratory processes that are used to prepare and analyze the sample.

Finally, performance can also be assessed at the level of individual components of the detector. These include (1) the aerosol collection unit, which collects and retains aerosol particles on a filter; (2) the sample recovery process, by which samples are removed from the aerosol collector and transported to a laboratory; (3) the filter extraction process, in which aerosol particles are removed from the filter and put into a liquid solution; (4) the DNA extraction process, in which DNA is extracted from the aerosol particles in liquid solution in preparation for further analysis; and (5) the PCR assays, which are used to test for the presence of specific genetic signatures of biothreat agents (as described earlier).

At the highest level—an array of detectors deployed in an operational environment—measures of performance include the system’s probability of detection (Pd) for attacks of different types and sizes. The BioWatch program employs a variation on the Pd measure that is designed to assess the system’s ability to detect attacks that could cause large numbers of casualties. Because BioWatch threat agents cannot be released into the air in operational environments, the performance of an array of detectors cannot be tested directly. One method of testing that can be used to address this limitation is the use of simulants for biothreat agents. A simulant is a selected nonpathogenic organism that mimics all or some of the physical or biological characteristics of one or more pathogenic agents. Another method that has been used for BioWatch involves computer modeling and simulation of attack scenarios.

At the level of a single detector, key measures of system performance include limits of detection, probability of detection, and specificity. The limits of detection are the lowest aerosol concentrations at which the system can detect the presence of a biothreat agent with a defined level of reliability. Probability of detection is the likelihood that the system will correctly detect the presence of a biothreat agent when it is present at a given aerosol concentration in the immediate vicinity of the detector. Specificity is the probability that the system will correctly yield a negative result when a biothreat agent is not present (that is, the probability that the system will not generate a “false positive”). The term specificity may also refer to the system’s ability to distinguish a biothreat agent from other, similar organisms.

At this single detector level, limits of detection and probability of detection reflect the sensitivity of the system and help determine the ability of an array of detectors in an operational environment to detect attacks; all else being equal, a more sensitive system will have greater ability to detect attacks. (Note, however, that the sensitivity of a single detector reflects its ability to detect an aerosol at the location where the detector is placed; additional analysis must be done to say what this means for the ability of an array of detectors in different locations to detect attacks of defined types and sizes.) The specificity of the system may contribute to the confidence that stakeholders and decision makers have in a positive result; a system with higher specificity is less likely to generate false positives, so users can have greater confidence in a positive result.

Detector-level performance is assessed primarily through testing. Such tests may be conducted in laboratory chambers or in open air and may involve live biothreat agents or simulants. End-to-end tests using live agents are currently not possible for the BioWatch system, as the BioWatch threat agents cannot be released in open-air environments, and at present there is no indoor chamber in which testing the Gen-2 system with live agents is technically feasible. Consequently, end-to-end tests of BioWatch must rely on simulants, which may be inactivated, or killed, forms of the same agents that the system is designed to detect. Alternatively, simulants may be related organisms, either live or killed. Testing a biodetection system outdoors with killed related organisms presents the most realistic opportunity to evaluate performance in an operationally representative environment.

Individual components of a detection system, such as the aerosol collector or the assay component, may be tested under strictly defined conditions. This type of testing could support comparisons of components with live agents and simulants and provides for a tight control of test conditions and variables for a robust characterization of components. However, testing components in a laboratory or chamber setting typically excludes some factors that might affect system performance in an operational environment, such as meteorological factors and materials in the air that might interfere with system performance (called interferents). Important meteorological factors include relative humidity, temperature, and solar irradiance; important interferents include pollutants (e.g., nitrates or carbon monoxide), as well as smoke and dust, all of which can influence the performance of a biodetection system. Also, the operational environment is difficult to reproduce in a biological containment chamber in terms of the aerosol concentration, particle size distribution, aging of the agent, and dispersion dynamics. The metrics used for components of a system depend on the component being tested. For example, for an aerosol collection component, they might include efficiency (i.e., the percentage of aerosol particles successfully collected on the filter and retained intact for subsequent analysis). For laboratory tests or assays, other metrics are used, including sensitivity, limits of detection, and specificity, as well as the efficiency of the processes by which samples are removed from the filter and prepared for analysis.

Biodefense, BioWatch, and the Biosurveillance Enterprise

Concerned with the threat of bioterrorism, in 2004, the White House released Homeland Security Presidential Directive 10 (HSPD-10), which outlined four pillars of the biodefense enterprise and discussed various federal efforts and responsibilities to help support it. The biodefense enterprise is the whole combination of systems at every level of government and the private sector that can contribute to protecting the nation and its citizens from potentially catastrophic effects of a biological event. It is composed of a complex collection of federal, state, local, tribal, territorial, and private resources, programs, and initiatives, designed for different purposes and dedicated to mitigating various risks, both natural and intentional. The four pillars of biodefense outlined in HSPD-10 were (1) threat awareness, (2) prevention and protection, (3) surveillance and detection, and (4) response and recovery. The BioWatch program falls under the surveillance and detection pillar, as an example of an environmental monitoring activity. Biosurveillance also includes disease monitoring and reporting to protect humans, animals, and plants from potentially catastrophic effects of intentional or natural biological events.

However, in 2011, the National Academies evaluation of BioWatch noted considerable uncertainty about the likelihood and magnitude of a biological attack, and about how the risk of a release of an aerosolized pathogen compares with risks from other potential forms of terrorism or from natural diseases. BioWatch was deployed rapidly to meet a perceived need for a system to detect catastrophic attacks. More recently, as we reported in 2012, OHA officials told us they use the Bioterrorism Risk Assessment (BTRA) to inform BioWatch because it is the most relevant risk assessment available to them and because it allows OHA to focus BioWatch detection efforts on the biological agents of significant concern. However, in 2008, the National Academies raised concerns about the methods used to develop the BTRA, particularly the methods used to assess the probability of an attack. The last full BTRA was issued in 2010 and did not address all the recommendations made by the National Academies.

The National Academies’ evaluation of BioWatch in 2011 also stated that to achieve its health protection goals, the BioWatch system should be better linked to a broader and more effective national biosurveillance framework that will help provide state and local public health authorities, in collaboration with the health care system, with the information they need to determine the appropriate response to a possible or confirmed attack or disease outbreak. In our earlier work, we highlighted the uncertainty about the incremental risk-mitigating benefit of the kind of environmental monitoring offered by BioWatch because of its relatively limited scope and the challenges agencies face in making investment decisions. In our June 2010 report on federal biosurveillance efforts, we recommended the Homeland Security Council direct the National Security Staff to identify a focal point to lead the development of a national biosurveillance strategy. We made this recommendation because we recognized the difficulty that decision makers and program managers in individual federal agencies face prioritizing resources to help ensure a coherent effort across a vast and dispersed interagency, intergovernmental, and intersectoral network. Therefore, we called for a strategy that would, among other things, (1) define the scope and purpose of a national capability; (2) provide goals, objectives and activities, priorities, milestones, and performance measures; and (3) assess the costs and benefits and identify resource and investment needs, including investment priorities. In July 2012, the White House released the National Strategy for Biosurveillance to describe the U.S. government’s approach to strengthening biosurveillance, but it did not fully meet the intent of our prior recommendations, because it did not offer a mechanism to identify resource and investment needs, including investment priorities among various biosurveillance efforts. Further, in 2005, we reported that because the nation cannot afford to protect everything against all threats, choices must be made about protection priorities given the risk and how to best allocate available resources. The strategic implementation plan has not been publicly released, but according to the strategy, it will include specific actions and activity scope, designated roles and responsibilities, and a mechanism for evaluating progress. However, it is too soon to tell what effect, if any, it may have on determining resource allocation priorities across the agencies.

DHS Cannot Make Informed Decisions about Upgrades or Enhancements Because It Lacks Reliable Information about Gen-2’s Capability to Detect an Attack

DHS lacks reliable information about BioWatch Gen-2’s technical capabilities to detect a biological attack and therefore lacks the basis for informed cost-benefit decisions about possible upgrades or enhancements to the system. In order to assess Gen-2’s capability to detect a biological attack, DHS would have to link test results to its conclusions about the ability of arrays of deployed detectors to detect attacks in BioWatch operational environments. This would ordinarily be done by developing and validating technical performance requirements based on operational objectives, but DHS has not developed such requirements for Gen-2. In the absence of technical performance requirements, DHS officials said their assertion that the system can detect catastrophic attacks is supported by modeling and simulation studies. However, these studies have not directly and comprehensively assessed the capabilities of the Gen-2 system. Furthermore, in our review of the tests that have been conducted, we found there are limitations and uncertainties in the test results on the technical performance characteristics of the Gen-2 system.

DHS Has Not Developed Performance Requirements That Would Allow Conclusions about Gen-2’s Ability to Detect Attacks

DHS commissioned four key tests of Gen-2’s technical performance characteristics, but has not developed and validated performance requirements that would enable it to interpret the test results and draw conclusions about the ability of an array of detectors in an operational environment to detect attacks. One test focused on the sensitivity of the whole system (that is, the aerosol collection unit and subsequent laboratory analysis of samples collected by that unit), while others focused on components of the system (table 1). None of these four tests focused on the highest level that we identified earlier—that is, an array of detectors placed in different locations. The four tests are described briefly below.

Dugway Proving Ground conducted a test of the sensitivity of the whole system, from the unit that collects aerosol particles on a filter through the analysis that looks for genetic material from biothreat agents. This test was designed to assess the system’s ability to detect aerosols of different concentrations, and it produced estimates of the system’s limits of detection (that is, the lowest aerosol concentrations that the system could detect with defined levels of reliability).

Dugway also conducted a test of the efficiencies of particular components of the Gen-2 system—in particular, the filter wash component (where aerosol particles are recovered from the filter into a liquid solution) and the DNA extraction component (where genetic material is extracted from the aerosol particles for further analysis). Each of these components influences the overall sensitivity of the Gen-2 system.

Edgewood Chemical Biological Center conducted a test of the aerosol collection component of the system by aerosolizing particles and measuring the system’s efficiency at trapping these particles on the filter and transferring them into liquid solution, another component whose performance influences the overall sensitivity of the system.

Los Alamos National Laboratory conducted tests of the PCR assays, which included measuring the assays’ sensitivity (their ability to detect different amounts of genetic material from the BioWatch threat agents), as well as their specificity (their ability to detect various strains of the BioWatch threat agents while correctly “ignoring” genetic material from other agents and interfering substances and materials commonly found in the environment).

In addition to these four key tests, DHS commissioned a demonstration of the system in an outdoor environment and conducts quality assurance tests on an ongoing basis. Both of these provide additional information about the system’s capabilities; however, we do not include them in our list of key tests because neither was designed to produce estimates of key performance characteristics, including sensitivity, or to support conclusions about the types and sizes of attack the system can reliably detect. The outdoor demonstration, performed by the Naval Surface Warfare Center Dahlgren Division, involved releasing a simulant for one of the BioWatch threat agents and showed that the Gen-2 technology could successfully detect this simulant in an open-air environment. However, aerosol concentrations were not varied systematically and measured independently in such a way as to produce statistical estimates of the system’s sensitivity. Additionally, ongoing quality assurance tests of the laboratory component of the Gen-2 system include testing filters that (1) contain potential interferents from BioWatch operational environments and (2) have been “spiked” with samples of killed biothreat agents, to verify that the system correctly detects these agents. However, these tests challenge the system with just one concentration of agent on the filter and therefore do not involve the systematic variation in concentration that is required to produce statistical estimates of the system’s sensitivity. Rather than estimating the system’s performance characteristics, these quality assurance tests are designed to provide confidence that system performance meets or exceeds benchmarks based on past system performance.

Under both DHS guidance and standard practice in testing and evaluation of defense systems, test results would be compared with predefined technical performance requirements. Those requirements would specify the technical performance parameters that a system must achieve in order to meet its operational objectives. In other words, requirements would provide targets against which test results can be evaluated in order to assess whether the system will reliably achieve its intended purpose. Technical performance requirements for BioWatch could include the limits of detection and probability of detection that a detector needs in order for a deployed array of detectors to reliably detect attacks of particular types and sizes.

While DHS has commissioned some testing of the system’s performance characteristics, officials told us they have not developed technical performance requirements, which would enable them to interpret the test results and draw conclusions about the system’s ability to meet its operational objective. DHS officials told us that the system’s operational objective is to detect catastrophic attacks, which they define as attacks large enough to cause 10,000 casualties, and they stated that the system is able to meet this objective. However, as we have previously reported, the BioWatch system was deployed quickly in 2003 to address a perceived urgent need; it was deployed without performance requirements and, as the National Academies has reported, without sufficient testing. In keeping with Office of Management and Budget (OMB) guidance on making decisions about federal programs, decisions about upgrades for BioWatch will require comprehensive information about the benefits and costs associated with the current system, including its capability to meet its operational objective. However, DHS officials told us that in the 12 years since BioWatch’s initial deployment, they have not developed technical performance requirements against which to measure the system’s ability to meet its objective. Nevertheless, DHS has already taken steps to pursue enhancements to the Gen-2 system. Because DHS lacks targets for the current system’s performance characteristics, including limits of detection, that would enable conclusions about the system’s ability to detect attacks of defined types and sizes with specified probabilities, it also cannot ensure it has complete information to make decisions about upgrades or enhancements.

DHS-Commissioned Modeling and Simulation Studies Have Not Directly and Comprehensively Assessed Gen-2 Capabilities

In the absence of technical performance requirements for Gen-2, DHS officials said they have used modeling and simulation studies, commissioned from multiple national laboratories, to link test results to conclusions about the system’s ability to detect attacks. In particular, they said that these modeling and simulation studies support their assertion that the Gen-2 system can detect catastrophic attacks, defined as attacks large enough to cause 10,000 casualties. However, while DHS officials provided reports to illustrate the modeling and simulation work that has been done, none of the studies that were provided or described to us incorporated specific test results, accounted for uncertainties in those results, and drew specific conclusions about the Gen-2 system’s ability to achieve the defined operational objective. Further, according to officials, DHS has not prepared an analysis of its own that combines the modeling and simulation studies with the specific Gen-2 test results to demonstrate DHS’s assertions about the system’s capabilities to detect attacks of defined types and sizes.

The modeling and simulation studies were designed for purposes other than to directly and comprehensively assess Gen-2’s operational capabilities. For example, one set of modeling and simulation studies, conducted by Sandia National Laboratories (Sandia) in collaboration with other national laboratories, was designed to predict the capabilities of hypothetical biodetection systems (similar to BioWatch) with different performance characteristics and deployed in different ways. For instance, these studies, which Sandia researchers called trade-space studies, assessed possible trade-offs in deploying fewer detectors with higher sensitivity or deploying more detectors with lower sensitivity. Sandia constructed models of hypothetical biodetection systems and then analyzed how these hypothetical systems would respond to simulated attacks of different sizes, using different agents, in different locations, and under different conditions (e.g., outdoor attacks with different wind speeds and directions, which affect how an aerosol disperses over an area). Because the goal was to assess hypothetical biodetection systems, Sandia analyzed ranges of hypothetical system sensitivities rather than incorporating the results of the four key tests of the performance characteristics of Gen-2. These studies drew no conclusions about the actual capabilities of the deployed Gen-2 system.

Further, the trade-space studies did not incorporate information about the actual locations of Gen-2 collector units. Rather, these studies were designed to model hypothetical BioWatch deployments in which collectors were placed in optimal locations. If the Gen-2 collectors were not actually placed in these optimal locations, then model results might not accurately describe the capabilities of the system as currently deployed.

In addition to the trade-space studies, DHS officials described modeling and simulation work they commissioned for the purpose of selecting sites for Gen-2 collector units; however, this work also had limitations that prevent specific conclusions about the Gen-2 system’s operational capabilities. Unlike the trade-space studies, the collector-siting analyses do include a test result that is meant to describe the sensitivity of the Gen-2 system. However, the test result used in this work was for just one of the five BioWatch threat agents, as decisions about collector siting are based on just this one agent. Consequently, these collector-siting analyses contain no information about the system’s capabilities to detect attacks using any of the other four BioWatch threat agents. Further, the test result used in this work was not from the four key tests of Gen-2 described earlier, but from an older test from 2004. An internal DHS analysis in 2013 noted that there were differences between the system tested in 2004 and the currently deployed system that limit the ability to draw conclusions from the 2004 results. Finally, the collector-siting studies use a measure of operational capability that does not directly support conclusions about the BioWatch objective of detecting attacks large enough to cause 10,000 casualties. In general, these studies use a measure called fraction of population protected, or Fp. Roughly speaking, Fp represents a system’s probability of successfully detecting simulated attacks, but calculated in a way that gives more weight to attacks that infect more people and less weight to attacks that infect fewer people. We believe this metric does not directly support conclusions about the system’s ability to detect attacks causing more than 10,000 casualties. Such conclusions would be supported by another metric that has been used by Sandia but is not preferred by the BioWatch program: the probability of detection (Pd) for attacks causing more than 10,000 casualties. DHS officials told us that they use Fp because BioWatch has a public health mission and so the system should be assessed in a way that reflects its ability to detect attacks that infect more people. However, Pd for attacks causing more than 10,000 casualties also incorporates public health impact; unlike Fp, it could directly support conclusions about the BioWatch operational objective, and, as noted in a Sandia report, is straightforward to communicate. Sandia officials told us that Fp has certain strengths and is appropriate for certain purposes. However, because the collector-siting studies focus on Fp, their results are not straightforward to communicate and do not support conclusions that align directly with the BioWatch operational objective.

Finally, because none of the modeling and simulation work was designed to interpret Gen-2 test results and comprehensively assess the capabilities of the Gen-2 system, none of these studies has provided a full accounting of statistical and other uncertainties—meaning decision makers have no means of understanding the precision or confidence in what is known about system capabilities. Best practices in risk analysis and cost-benefit analysis require an explicit accounting of uncertainties so that decision makers can grasp the reliability of, and precision in, estimates to be used for decision making. Estimates of the Gen-2 system’s limits of detection, produced by the four key tests described earlier, contain multiple sources of uncertainty, which we describe in the next section of this report. None of the modeling and simulation studies that were provided or described to us incorporated information about the uncertainties associated with estimates of the system’s limits of detection.

We also found that these studies did not account for uncertainty in some model inputs and assumptions, including estimates of how infectious each of the BioWatch threat agents is and how quickly each agent decays after it is released in the air. For example, Sandia researchers and a subject matter expert told us that there is considerable uncertainty in even the best available estimates of the infectious dose of anthrax, as these estimates are based on data from nonhuman primates. In an earlier study, Sandia researchers and others reported that “gaps in our knowledge of the correct dose-response relationship significantly limit our ability to predict the outcome of outdoor anthrax attacks.” For many of the assumptions the Sandia models used, researchers dealt with uncertainty by using not just single estimates but ranges of estimates. However, for the infectious doses of the five BioWatch threat agents, researchers used single estimates that DHS provided. The uncertainty in these estimates is important because DHS officials have characterized the operational objective of the BioWatch system as detecting attacks large enough to cause 10,000 casualties. In order to assess the system’s capability to achieve this objective, DHS must be able to correctly define the types and sizes of attack that fall into this category. If anthrax were less infectious than the models assumed, then DHS would be underestimating the system’s ability to detect catastrophic anthrax attacks. Conversely, if anthrax were more infectious than the models assumed, then DHS would be overestimating the system’s ability to detect such attacks. We recognize that more precise infectious dose estimates may not exist, but this underscores the uncertainty in the ability of the BioWatch system to meet its operational objective—uncertainty that should be articulated to better inform decision makers about the capabilities of the Gen-2 system and inform cost-benefit decisions about any possible enhancements to the system.

Limitations and Uncertainties in Test Results

We found limitations and uncertainties in the four key tests of the Gen-2 system’s performance characteristics—in particular, in the use of test chambers instead of operational environments and the use of simulants in place of live biothreat agents. As noted earlier, it is not possible to test the BioWatch system directly by releasing live biothreat agents into the air in operational environments. Because of this constraint, which is beyond DHS’s control, the agency commissioned tests that involved aerosols in test chambers or were limited to components of the system for which aerosols were not necessary. Further, officials and experts told us there are no test chambers where testing the Gen-2 system with live agent is technically feasible; thus some tests have involved simulants in place of live biothreat agents.

Using laboratory chambers and simulants effectively addressed certain challenges, but both introduced uncertainties into testing results. Chambers often differ from operational environments in ways that can affect a system’s performance. For example, chamber environments are generally not designed to be representative of operational environments in such factors as air temperature, humidity, and, according to an expert, the presence of potential interferents in the air. Similarly, simulants may not mimic the biothreat agents that the system is designed to detect in all of the ways that matter for system performance; therefore, the system might perform differently when presented with the target biothreat agents than when tested with simulants. As a result, chambers and simulants create uncertainty as to whether test results accurately describe how the system would perform in an operational environment against live BioWatch threat agents, and this uncertainty should be clearly articulated for decision makers.

Additionally, while one of the four tests assessed the performance of the whole Gen-2 system, the three other tests were limited by their focus on components of the system, including (1) the aerosol collection component; (2) the filter wash process, in which aerosol particles are transferred from the filter into a liquid solution; (3) the DNA extraction process, in which genetic material is extracted from the particles in liquid solution; and (4) the analytical component, in which PCR assays are used to detect genetic signatures of the BioWatch threat agents. According to a National Research Council (NRC) committee, it is uncertain whether test results from individual components of a biodetection system will accurately reflect the performance of the whole system. An expert told us that components may perform differently when combined than when tested separately, and parts of samples may be lost during transitions from one component to another in ways that affect the end-to-end performance of the system.

DHS took steps to mitigate the limitations associated with not testing the Gen-2 system in an operational environment with live biothreat agents, but these limitations could not be eliminated entirely. For example, to address the fact that killed agents might not perfectly mimic live biothreat agents, the Dugway tests included a direct comparison of live and killed agents, but this could be done only for the analytical component of the system (that is, the PCR assays). The Edgewood Chemical Biological Center test of the aerosol collection component included variations in temperature and humidity. This somewhat mitigated the fact that chambers may not be representative of operational environments; however, only a small number of combinations of temperature and humidity were tested, and an expert told us other characteristics that might differ between chambers and operational environments were not varied. The Los Alamos National Laboratory tests of the PCR assays included testing the assays with a set of environmental organisms and substances. However, this test was limited to the specific organisms and substances used, and results do not generalize to other organisms and substances that might occur in BioWatch operational environments. In sum, although the key tests of the Gen-2 system took steps to mitigate limitations, uncertainties remain, and test results constitute only limited measures of key system performance parameters in an operational environment.

While challenges associated with testing a system like BioWatch make some limitations unavoidable, according to experts and agency officials, we found that some limitations could likely have been mitigated. In 2004, an NRC committee proposed a framework for testing biodetection systems that was designed to minimize the uncertainties associated with laboratory chambers and simulants. In this framework, both the whole system and its components are systematically tested with simulants and, where possible, live biothreat agents. This is done in both laboratory chambers and environments that are more representative of operational environments where the system will be deployed. Importantly, this framework entails an integrated, systematic approach to testing in which some factors (e.g., agents, simulants, and environmental factors) are held constant while others are varied. The committee recommended focusing on a certain category of simulants, known as killed related strains, that have potential to mimic live biothreat agents while also enabling more realistic and frequent testing. The overall goal of the NRC framework is to home in on the true performance of the system when challenged with live agents in an operational environment—and, in so doing, to reduce the risk that the system will perform differently in the real world than it did during testing.

DHS has not systematically tested the Gen-2 system under the most realistic possible conditions. Although DHS officials said they based their approach to testing Gen-2 on the NRC framework, we found that the Dugway test of system sensitivity did not incorporate killed related strains as simulants, as recommended by the NRC. Killed related strains would offer greater flexibility for use in more operationally representative environments. Furthermore, the Dugway test did not attempt to incorporate potential environmental interferents. As noted earlier, DHS also commissioned a demonstration that the Gen-2 technology could detect a simulant in an outdoor environment. However, aerosol concentrations were not varied systematically and measured independently in such a way as to produce statistical estimates of the system’s sensitivity. Furthermore, this open-air demonstration involved a simulant for just one of the five BioWatch threat agents, and, unlike the killed related strains recommended by the NRC, this simulant required that the system be modified to use a different PCR assay than is used for the actual threat agent. A GAO subject matter expert on outdoor testing of biodetection systems with simulants assessed the Dahlgren trials to be deficient in the equipment that was used to characterize the aerosols; if alternative equipment known as slit-to-agar samplers had been used, they could have provided more useful information on aerosol concentrations and exposure times. According to this expert, additional problems were associated with an inadequate dissemination system and inadvisable testing at wind speeds below 2.0 meters/second. In general, DHS’s understanding of the performance characteristics of the current system would benefit from a more systematic approach to testing under more realistic conditions.

In the next year, some Gen-2 equipment will reach the end of its lifecycle, and DHS will need to make decisions about reinvesting in the program. Further, DHS officials told us they are considering potential improvements or upgrades to the Gen-2 system. Based on OMB guidance, cost-benefit decisions about investing in new equipment and in potential system improvements will require information about current operational capabilities. However, because of the limitations we have identified, decision makers are not assured of having sufficient information to ensure future investments are actually addressing a capability gap not met by the current system. As noted earlier, some limitations in testing are unavoidable given the nature and purpose of the BioWatch system. Likewise, some limitations and uncertainties are unavoidable in the modeling and simulation work that DHS has commissioned to link test results to operational capabilities (e.g., the uncertainty in infectious dose estimates for the BioWatch threat agents). These limitations underscore the need for a full accounting of statistical and other uncertainties, without which decision makers lack an understanding of the precision in what is known about the system’s capability to detect attacks of defined types and sizes and cannot make informed decisions about possible upgrades to Gen-2.

Gen-3 Testing Partially Aligned with Best Practices, and Gen-2 Upgrades Could Benefit by Applying Lessons Learned

In 2013, in collaboration with the National Academies, we identified eight best practices for developmental testing of threat detection systems. When comparing DHS’s actions and decisions regarding the planned acquisition and testing of Gen-3, we found that DHS’s actions partially aligned with these best practices. We also identified several lessons DHS could learn by applying these practices more systematically to improve future testing and acquisition efforts—for example, testing of possible upgrades or enhancements to Gen-2. We also highlight the testing of other DHS acquisitions that faced challenges.

Best Practices and Lessons Learned from Gen-3 Phase I Testing

Role of Developmental Testing Developmental testing is intended to assist in identifying system performance, capabilities, limitations, and safety issues to help reduce design and programmatic risks. According to experts recruited in coordination with the National Academies, the best practices they identified apply to the process of developmental testing of binary threat detection systems; they also apply if the tested system is commercial-off-the-shelf (COTS), modified COTS, or newly developed for a specific threat detection purpose by a vendor or the government.

In previous work, in collaboration with the National Academies, we recruited experts to develop best practices for developmental testing of binary threat detection systems. According to the experts, the best practices apply if the system is commercial-off-the-shelf (COTS), modified COTS, or newly developed for a specific threat detection purpose by a vendor or the government. In discussing the role of testing early in an acquisition, officials from S&T’s Office of Operational Test and Evaluation (OT&E) said programs face a significant challenge in acquiring COTS products, like the BioWatch Gen-3 acquisition, because these products or systems are not designed to operate or initially tested with the same requirements needed by the federal government. Typically, OT&E officials said, program officials underestimate the testing needed to acquire a COTS product for their mission, but said testing is needed to ensure the COTS solution is reliable, scalable, and secure. We reported in 2012 that according to BioWatch program officials, developing autonomous biological detection had proved challenging, in part because some of the technology required was novel, but also because even the existing technologies—for example, the aerosol collection unit and the apparatus that reads the PCR results—had not been combined for this specific application in an operational environment before. S&T OT&E officials reflecting on the acquisition said Gen-3 Phase I testing became developmental in nature, with additional steps built into the test design to ensure the technology could progress to the next level, particularly to ensure assay detection met the program’s requirements.

We consider the best practices for developmental testing applicable to Phase I of the Gen-3 acquisition because they could have helped identify challenges early in the acquisition process. Additionally, applying the best practices and lessons learned during the Gen-3 testing could help mitigate the risk that DHS acquires immature technology as part of its effort to make enhancements to Gen-2. Appendix I has more information on our methodology for developing the best practices, and appendix III has a description of each best practice. Our analysis of DHS’s alignment with the eight best practices for developmental testing of Gen-3 follows below and is summarized in table 2.

DHS’s Actions Partially Aligned with Best Practice 1 DHS took some risk-mitigating steps during Gen-3 Phase I, but did not conduct a full risk assessment at the outset of the acquisition. According to program officials, Phase I of the overall Gen-3 acquisition was itself a risk mitigation activity, designed to assess the capability of industry to provide mature autonomous detection solutions before committing to a rapid and extensive program to procure and field operational systems. For example, according to BioWatch program officials, DHS conducted market research to assess whether solutions potentially capable of meeting DHS performance and maturity (and therefore schedule) requirements existed. In an effort to ensure accountability, OHA held a “Demonstration Day” early in the Phase I source selection process, where vendors participated in tests designed to confirm their technical maturity claims. This reduced (but did not eliminate) the risk of awarding Phase I contracts to a vendor not capable of completing the tests planned under Phase I. Additionally, officials said OHA initially planned to conduct all testing in parallel. However, because of schedule slips associated with contract issues, program officials ended up scheduling the tests incrementally to allow for the insertion of decision points on whether testing should continue. This allowed program officials to engage in the testing and evaluate test results to help reduce technical and financial risk. As a result of this multi-stage Phase I testing approach, DHS identified limitations to one vendor’s detection system that could not be overcome before proceeding with the next stages of testing.

DHS Could Apply Lessons Learned An earlier evaluation of risk may have eliminated difficulties with cost and schedule estimates of the Gen-3 acquisition that in part led to its cancellation in April 2014. According to this best practice, risk needs to be assessed when the system is COTS, modified COTS, or a newly developed system, because even with commercial items, significant modifications may be needed. DHS ultimately performed a formal risk assessment of the Gen-3 acquisition but not until after Phase I testing ended. The absence of a risk assessment at the start of the acquisition led to challenges that the acquisition could not overcome because of its inflexibility. In 2012, we reported that DHS did not fully engage in the early phases of its acquisition framework to ensure that the acquisition pursued an optimal solution in the context of its costs, benefits, and risks. Our prior work has found that stable parameters for performance, cost, and schedule are among the factors that are important for successfully delivering capabilities within cost and schedule expectations. Our work has also found that without the development, review, and approval of key acquisition documents, agencies are at risk of having poorly defined requirements that can negatively affect program performance and contribute to increased costs. For example, despite having limited assurance that the acquisition would successfully deliver the intended capability within cost and on schedule, the Deputy Secretary approved the initial stages of the acquisition. DHS’s Post Implementation Review Report, which lays out lessons learned on the Gen-3 acquisition, states that the Phase I testing demonstrated that schedule risk analyses should be used to set realistic test and evaluation schedule expectations. According to program officials, they set the acquisition schedule estimate aggressively because there was pressure to quickly deploy an autonomous detection capability. However, schedule revisions were needed because of significant changes in performance, deployment schedule, and cost expectations as a result of the Phase I testing.

By not engaging in the initial steps of the acquisition framework to effectively account for risks early in the acquisition, DHS did not demonstrate full accountability and exceeded its cost and timeframe estimates. As a result of the Phase I testing, and on the basis of outside reviews of the Gen-3 acquisition, DHS directed OHA to conduct a more robust analysis of alternatives that included assessing risk. This led to the cancellation of the Gen-3 acquisition in April 2014. The identification of potential risks, and strategies to overcome these risks, helps ensure accountability on the part of the agency and may alleviate problems with the acquisition of threat detection systems if they are part of the early planning for testing. As DHS considers upgrades to the currently deployed Gen-2 BioWatch system or considers the acquisition of new detection technologies, early identification of risks may help better guide DHS by identifying areas for enhanced engagement that may be needed during the testing and to help ensure proper accountability for decision making during the acquisition.

Practice 2: Include Representatives from the User Community in Design and Developmental Testing Teams to Ensure Acceptance of the System by the User Community

Acquisition Decision Event (ADE) 2A An acquisition decision event, where the Acquisition Review Board—a cross- component board of senior DHS officials— determines whether a proposed acquisition has met the requirements of the relevant Acquisition Life-Cycle Framework phase and is able to proceed. ADE-2A is the culminating event for the Analyze/Select phase of the DHS acquisition framework, where DHS determines whether to authorize the acquisition to proceed to the Obtain phase, where testing and evaluation occur.

DHS did not sufficiently involve the end user community in the development of Gen-3 system requirements or parts of Phase I testing; rather it relied on internal subject matter experts to develop requirements. As we reported in 2012, the process used to set the sensitivity requirement did not reflect stakeholder consensus about how to balance mission needs with technological capabilities. Specifically, the BioWatch program did not prepare a concept of operations (CONOPS) before ADE-2A. According to DHS acquisitions guidance, in developing a concept of operations, stakeholders engage in a consensus-building process regarding how to balance technological capabilities with mission needs in order to gain consensus on the use, capabilities, and benefits of a system. For BioWatch, this could include specifying the level of population protection the system should provide and then specifying the sensitivity levels needed to provide that level of protection. According to OHA officials, the high-level capability gaps documented in the mission need statement (timeliness, population coverage, time resolution, and cost-effectiveness) were representative of feedback from this user community with respect to improvements needed on the Gen-2 system, particularly for indoor deployment. Therefore, officials said they did not directly involve jurisdictional public health stakeholders in establishing the technical requirements, including sensitivity requirements, for Gen-3.

Role of Subject Matter Experts (SMEs) in Developmental Testing Distinct from the user community, SMEs provide independent technical advice and monitor the status of developmental testing to help ensure tests are conducted and analyzed properly—a practice the National Academies has supported for BioWatch. SMEs may include test designers, engineers, and statisticians. For a program like BioWatch, SMEs may have expertise in epidemiology, environmental health, public health laboratory systems, infectious diseases, genetics, and detection technology, among other disciplines. Department of Homeland Security (DHS) acquisitions policy includes guidance on using independent technical advisors as part of the test and evaluation process. For example, an operational test agent (OTA) is an independent entity that supports development of the test and evaluation master plan (TEMP) and monitors developmental testing in order to understand system performance early and determine how to execute integrated developmental and operational testing. The OTA presents objective and unbiased conclusions in reports and test readiness reviews. DHS guidance also describes the role of the Integrated Process Team, which is composed of representatives from program leadership, stakeholders, and SMEs involved in testing activities.

OHA relied on in-house experts and other department officials who they said had the expertise to convert the high-level mission needs into detailed technical performance requirements. These included requirements for system sensitivity, time needed to detect an attack, and the probability of false positives, among other things. According to OHA officials, jurisdictions (especially the four that were selected for OT&E) were kept informed through meetings with the program office, independent test agencies, and the two competing vendors in the Phase I testing. OHA officials also said they conducted numerous updates at events such as the National BioWatch Stakeholders Workshop, webinars, and invitations to all jurisdictions to submit questions to the Gen-3 or BioWatch program managers. According to OHA officials, the end user community was also invited to observe testing at two special testing events during Phase I. However, informing end users of the status of testing is not the same as including them in the development of requirements and testing. If DHS had involved end users as part of the testing process, it could have shed additional light on the potential challenges for end users to operate the technology being considered. For example, an official who tested the NG-ADS system described months- long training that was required to understand the systems being tested and evaluated. More closely involving the end users in the testing may have revealed additional end user views on their ability and willingness to use the equipment, given its complexity. Additionally, in December 2010, the Undersecretary for Management issued an ADM that, among other things, highlighted the continued need to develop a CONOPS, citing significant risk to the program because of the high-level coordination required for acting upon detection information produced by the autonomous detection system and that there was insufficient detail in the program documentation describing the necessary coordination process among end users and other stakeholders.

DHS Could Apply Lessons Learned DHS recognized in its Post Implementation Review that stakeholders and end users need to be involved earlier in the acquisition process, including validating advanced detection systems and methods before they are fielded. To better understand the needs, concerns, and capabilities of the user community, in the future, DHS could take steps to engage with stakeholders early on in the development process. As DHS considers upgrades to the current Gen-2 system, DHS should, in accordance with DHS acquisition guidance, prepare a concept of operations and ensure end users are engaged throughout the testing of upgrades or enhancements to the Gen-2 system or new acquisitions for BioWatch.

Practice 3: Take a Systems Engineering View of the System prior to Entering into Any Developmental Test

DHS’s Actions Partially Aligned with Best Practice 3 The systems engineering approach was outlined in the Gen-3 test and evaluation master plan (TEMP), which was generally clear in defining the boundaries of the system to be tested. The tests were for the most part appropriately scoped given the systems engineering view that was taken. However, testing revealed that evaluation of these detection systems may benefit from more robust testing methods, particularly to test performance against environmental contaminants. The primary purpose of the Phase I testing was characterizing the systems’ performance through a series of tests that included an aerosol collection subsystem test, evaluation of assays, an analytical subsystem test, a system chamber test, and a field test. The TEMP clearly identified these test boundaries and stated that all technologies must meet the five key performance parameters (KPP) during Phase I testing before selection would be made and Phase II testing (operational testing) was initiated. KPPs included detection of the biological agents, system sensitivity, the time to detect, achieved availability (for example, the probability the detector will operate under normal conditions), and the probability of a false positive.

The TEMP recognized inherent limitations to the systems engineering testing for the Gen-3 system. For example, whereas the environmental conditions under which the Gen-3 system must operate were outlined in the TEMP, no chamber yet exists in which these requirements can be fully tested. In addition, according to DHS officials, legal constraints, public safety, and public perception limit the type of material that can be aerosolized in a realistic setting for test purposes. The TEMP outlined the systems engineering approach for Gen-3 testing by articulating the major issues that needed to be addressed in testing the system, including the key performance parameters, and accounted for the limitations given constraints on the type of testing that can be done.

DHS Could Apply Lessons Learned Although DHS clearly articulated the boundaries of the testing for Phase I and took steps to test the autonomous detectors against likely environmental contaminants that might interfere with detection, the systems engineering approach could have benefited from more robust and comprehensive testing methods. For example, tests at Dugway also attempted to account for the possibility that environmental pollutants might interfere with the performance of the PCR assays by placing samples in liquid solutions taken from actual BioWatch filters retrieved from operational environments. However, researchers told us they were unaware of which operational environments the filter washes had come from, and there was no sub-analysis by type of environment (e.g., outdoor versus indoor versus subway), which raises the possibility that pollutants from certain environments may not have been represented in the test or that pollutants from certain environments may have been diluted with filter wash from other environments. The pooled filter wash was used to test the Gen-3 analytical process in order to look for possible inhibition (e.g., interference) of the PCR assays. Officials said that during the Gen-3 testing, the pooled filter wash used to test the analytical process never showed interference in acquiring a result to an extent that would have required additional testing steps.

However, in the final Phase I test, DHS fielded detectors in Chicago to demonstrate the performance of the candidate technology’s full system in a representative environment. Results of the testing showed that the candidate system’s performance was inconsistent in different operational environments. Specifically, detectors located on underground subway platforms had higher incidences of malfunction than detectors in other locations. These malfunctions may be associated with the presence of metallic brake dust, which demonstrates that different operational environments pose different challenges. Additional rigor in the testing design could have identified limitations earlier and perhaps mitigated them prior to the field testing in Chicago. In fact, the final report by the National Assessment Group on the operational assessment of the Gen-3 Phase I testing concluded that in retrospect, and based on the outcome of testing in the Chicago field test, levels of some possible environmental inhibitors, such as metallic brake dust, represented in the Dugway testing were significantly diluted or did not represent a concentration that compared with concentrations in some of the more problematic operating environments. Therefore, a more robust systems engineering approach to testing contaminants may have helped identify challenges like this earlier. The BioWatch program office agreed that the limited environmental contaminant testing was insufficient to draw any conclusions about system performance on this issue. According to program office officials, the program planned to implement robust interferent tests during Phase II to characterize performance against all of the operational requirements and said Phase I was only meant to characterize the state of the market for autonomous detection systems. However, the TEMP for Gen-3 states that the goal of Phase I testing was to evaluate the ability of the candidate systems to meet performance requirements specified in an operational requirements document (ORD). The ORD for Gen-3 specifically outlines the indoor and outdoor environmental conditions under which the system is expected to operate, including, but not limited to, exposure to dust, metallic dust, diesel exhaust, pollen, rain, snow, ice, wind, salt spray, as well as ranges in temperature, humidity, and altitude. DHS’s plan to defer more robust testing of conditions representative of Gen-3’s intended operational environment does not align with best practices, as performance problems uncovered in later stages of testing can be more costly and require additional testing.

While a basic approach to account for environmental contaminants was included in the systems engineering approach, the program office recognized that the results of the field testing highlighted additional limitations to the test approach for environmental contaminants. The BioWatch program may benefit by incorporating this lesson learned when designing future testing approaches for upgrades or enhancements to the Gen-2 program.

Practice 4: Use Statistical Experimental Design Methodology to Establish a Solid Foundation for Developmental Testing

DHS Partially Aligned with Best Practice 4 DHS included statistical experimental design in its Gen-3 testing plans in order to test performance and characterize uncertainty in the test results. However, the statistical experimental design constructed by DHS was not sufficient to estimate system performance in a realistic environment and did not link KPPs in the ORD to overall program objectives via creation and use of an appropriate model of system performance. In the analytical subsystem test, conducted at Dugway, the candidate Gen-3 system was challenged with aerosols of different concentrations in order to estimate its probability of detection for four of five BioWatch threat agents. Concentrations were systematically varied and were selected using a statistical method that was designed to yield reliable estimates of the system’s probability of detection as efficiently as possible (i.e., reducing the experimental effort required to obtain sufficiently reliable information). Statistical uncertainties were calculated for the resulting estimates, and statistical modeling was used to characterize the relationship between aerosol concentration and the system’s probability of correctly detecting the presence of each biothreat agent. Statistical experimental designs were also used in the tests of the aerosol collection component, conducted at Edgewood Chemical Biological Center, and the PCR assays, conducted at Los Alamos National Laboratory. Additionally, the Gen-3 tests included experimental conditions designed to account for factors seen in operational environments, though we have identified limitations in some of these tests, as discussed earlier in the report.

DHS Could Apply Lessons Learned According to this best practice, test design should be based on a clear understanding of goals and incorporate users’ needs. This could be achieved, for example, by linking KPPs in the ORD to overall program objectives and user needs via creation and use of an appropriate model of system performance, but this was not done prior to the Gen-3 testing. Operational requirements, from which the KPPs were derived, were not clearly linked to an overall mission need or program goal. The absence of these linkages among mission need, requirements, and parameters for measuring system performance means results from the Gen-3 testing cannot speak to whether the system would address an established mission need or users’ needs. According to program officials, the operational requirements outlined in the ORD were directly linked to the approved Mission Needs Statement for Gen-3. However, in 2012, we reported that officials were aware that the Mission Needs Statement did not reflect a systematic effort to justify a capability need and we reported that the program wrote the Mission Needs Statement later to justify a predetermined solution of acquiring an autonomous detection capability.

Additionally, these tests did not cover all of the system requirements specified by the ORD. For example, one of the system requirements was a maximum false positive rate. DHS noted in the TEMP that the probability of false positives would be estimated from test results using statistical techniques; however, the TEMP did not explain how this would be done. In testing, DHS did not address this. Instead DHS tested 20 times (per agent) in the assay tests and 15 times (per agent) in the Dugway tests, and drew conclusions despite not having designed experiments that would produce estimates of the system’s false positive rate with defined levels of statistical precision.

By using statistical experimental designs in the testing of the Gen-3 technologies, DHS was able to characterize some uncertainty in the test results. However, DHS was not able to determine with a defined level of statistical certainty the false positive rate, and thus was not able to conclude that any tested system or system component satisfies its stated requirement. As DHS considers upgrades to the Gen-2 system or future technology switches for the BioWatch program, DHS could apply lessons learned from the Gen-3 testing to help develop meaningful requirements that are linked to a mission need and where operational objectives (e.g., lives saved) are linked to measurable KPPs, such as system sensitivity. DHS could also use statistical experimental design in future testing of upgrades or enhancements to the Gen-2 system to help characterize uncertainty in results and ensure a representative range of operating conditions are sufficiently used to test system performance.

Practice 5: Measure and Characterize System Performance with Established Procedures, Methods, and Metrics

DHS’s Actions Partially Aligned with Best Practice 5 DHS created and executed a well-articulated, but incomplete, plan for measuring and characterizing certain aspects of system performance using established procedures, methods, and metrics. For example, measuring system performance included ranking and scoring by agent the system’s ability to detect known strains of an agent, including near- neighbor strains of an agent, and ability to detect agents with possible environmental contaminants present. System sensitivity was characterized using an adaptive methodology to determine, for a range of concentrations of agents, the probability of detection at each concentration. These probabilities were associated with confidence intervals to allow assessment of the range of performance one may expect. Both live and killed agents were used to assess detection sensitivity so that results from component testing could be extrapolated to whole system performance, because currently no facilities exist to perform whole-system live agent testing. Additionally, DHS evaluated suitability requirements by testing variables such as human factors; reliability, availability, maintainability (RAM); supportability; and survivability in an operational environment.

DHS Could Apply Lessons Learned Whereas the TEMP and other test plans list the five KPPs for the Gen-3 BioWatch candidate systems, there is no specific link between these metrics and the mission objectives for the Gen-3 system. Further, it is not clear how the results of testing in non-representative environments would be used to determine system suitability for Gen-3 purposes. In addition to these best practices, developmental testing guidance indicates that developmental testing is intended to vet systems early to determine the suitability of the system for meeting performance requirements. By testing in a variety of modes intended to replicate an operating environment, results of the tests can be used to characterize and evaluate relevant system performance. However, the tests that were intended to account for factors seen in operational environments were of limited range and scope. For example, the test plan for the Chicago field test suggested that the system would be exposed to dust, metallic dust, smoke, diesel exhaust, and pollen, under extended temperature, altitude, and relative humidity ranges. However, there was no design of statistical tests intended to address these variations in Phase I. The analytical sub- system testing did not allow for sub-analysis of pollutants from different environments (e.g., outdoor, indoor, subway), and so it was not possible to identify effects of any specific pollutant, such as subway brake dust. Additionally, the Edgewood test of the aerosol collection component included tests at different temperatures and humidities; however, relatively few combinations of temperature and humidity were tested, a fact that limited the range of environmental conditions represented and limited the utility of their results in determining how the system would perform under various operational environments.

Because DHS performed only limited testing in this regard, it was not able to draw conclusions early in the testing process to determine whether the systems would meet performance requirements. In the future, DHS could improve BioWatch testing efforts by incorporating clearly articulated measures and use of established procedures, methods, and metrics to characterize system performance. This will help ensure DHS collects the information it needs to evaluate operational suitability of upgrades or enhancements to the Gen-2 system.

Practice 6: Test to Build in Resilience, Especially in the Development Stages

DHS’s Actions Partially Aligned with Best Practice 6 DHS performed some testing that could help build resilience into the system during Phase I testing, but it could improve resilience testing with more rigorous methods. According to OHA officials, the Phase I acquisition and test and evaluation (T&E) strategies were specifically designed to address the concern of identifying potential vulnerabilities in the systems under test. As part of this strategy, for example, Edgewood tested the collection efficiency of the filters under varying temperature and humidity conditions. The system was also tested in an operational environment in Chicago to assess the effects of environmental interferents, among other things, to help identify vulnerabilities in operating the system. Further, DHS recognized that developing autonomous detection had proved challenging because in addition to some of the technology required being novel, even the existing technologies—for example, the aerosol collection unit and the apparatus that reads the PCR results—had not been combined for the specific application of autonomous detection in an operational environment before. By executing assay evaluation and subsystem- and system-level characterization tests incrementally, this allowed for the insertion of decision points to reduce technical and financial risk. As a result of its multi-stage approach to testing the systems, DHS identified limitations to one vendor’s detection system that could not be overcome before proceeding with the next stages of Phase I testing, so that vendor did not complete the entire Phase I testing plan. DHS also included provisions in the Phase I contracts for engineering change proposals to allow the vendors, at DHS discretion, to address deficiencies found during testing to inform DHS decisions on proceeding to Phase II.

DHS Could Apply Lessons Learned While DHS took some steps to build resilience into the Phase I testing, additional rigor could have been built into the testing design to reveal potential vulnerabilities in the performance of the systems tested. DHS’s final test in Phase I, designed to test resilience, involved fielding the detectors in Chicago to demonstrate the performance of the candidate technology’s full system in a representative environment. DHS was able to identify some limitations—such as environmental contaminants—to the detection systems being evaluated. However, this occurred late in Phase I testing, and represented only a limited cross-section of possible challenges, including temperature, humidity, and vibrations. For example, while the test plan for the Chicago test lists a range of temperatures and humidities the system is expected to operate under, the field testing did not reflect the entire operational range.

Additionally, other tests on temperature and humidity conditions were performed in limited combinations, such as high temperature and low humidity, but not others, such as high temperature and high humidity. Further, these temperature and humidity tests were done only on the aerosol collection unit, and not the entire Gen-3 system. Because the Gen-3 system as a whole was expected to operate continuously under such conditions, just as Gen-2 currently operates, simply testing one component, the aerosol collection unit, under a limited combination of temperatures and humidities does not adequately test the system for purposes of building in resilience. Some of these conditions may have provided earlier indications of vulnerabilities that did not arise until near the end of Phase I testing. For example, to assess the effect of interferents, Dugway testing used pooled filter washes provided by DHS. The environments from which the filters came were not provided to Dugway and interferents from different filters were combined. In doing so, the effect of individual interferents was diluted. When the systems were tested in Chicago near the end of Phase I, DHS found that system problems occurred, attributed to petrochemicals near highways and to brake dust in subway stations. If specific interferents, such as subway brake dust, were tested prior to the Chicago testing at representative concentrations, they may have revealed issues earlier in Phase I.

Other conditions, such as network communication performance, were not tested or were tested in a limited fashion. According to the final report on the Gen-3 testing, tests of network communication performance were modified based on user needs in the Chicago area where this capability would have been demonstrated. Therefore, network performance was left undetermined at the end of Phase I testing.

As a result of not including more rigorous testing methods to test resilience, information about system failures in different environments was not revealed until late in the Phase I testing. By following this best practice in future testing of Gen-2 upgrades or enhancements, DHS could help mitigate risks the agency may likely face in acquiring these types of threat detection technologies.

Practice 7: Use Developmental Tests to Refine Performance Requirements

DHS’s Actions Partially Aligned with Best Practice 7 DHS used Phase I test results to determine the likelihood Gen-3 candidate systems could meet performance requirements, but revisions to performance requirements were based on a modeling study, rather than the outcome of the Phase I testing. According to OHA officials, at the time OHA began Phase I planning and execution, there was not a robust analytical capability to determine mission-based requirements for key technical parameters (such as system sensitivity). Officials said absent that, a consensus on the technical parameters was made based on the collective best judgment of OHA BioWatch Program Office and S&T Chem Bio Defense subject matter experts. According to DHS officials, the result was more of a “technology push” requirement than a mission outcome driven requirement, and was based upon what was believed to be the state of the art for PCR-based systems. However, as we reported in 2012, when the technologies were unable to meet the technology push requirement as determined by some Phase I testing, DHS encountered delays and uncertainty about how to move forward. In response to these concerns, the BioWatch Program Office conducted a detailed requirements analysis through a national lab consortium led by Sandia National Laboratory. The study assessed the utility of biodetection systems with varying levels of sensitivity in terms of detection timeliness, population coverage, and lives saved in a bioterror attack. As we reported in 2012, the study, which was completed in January 2012, contained findings that, according to BioWatch Program officials, confirm that the sensitivity requirement could be relaxed without significantly affecting the program’s public health mission. As a result, DHS set a new sensitivity requirement based on the modeling studies.

DHS Could Apply Lessons Learned According to experts, developmental testing should be viewed as a tool in helping to refine performance requirements, and a meaningful performance requirement is one that is not only achievable but also strives to maximize the fulfillment of a mission need. As we reported in 2012, according to BioWatch program officials, the original sensitivity requirement was set based on interest in pushing the limits of potential technological achievement rather than in response to a desired public health protection outcome. They said that this led to a requirement that may have been too stringent, resulting in higher costs and schedule delays without demonstrated mission imperative.

Further, having requirements based on operational objectives would allow the results from Phase I testing to inform the users regarding the anticipated capabilities of the system and the likelihood that a tested system would be able to meet mission needs. According to experts, developmental testing may help to identify the performance envelope of the system and inform decisions about refined performance requirements. Failure of the Gen-3 candidate system to meet the initial performance requirements resulted in delays and uncertainty regarding the Gen-3 acquisition. DHS took steps to refine the sensitivity requirement for Gen- 3, but it was the modeling study, rather than the developmental testing protocol, that led to the change. However, we found that even the relaxed sensitivity requirement did not link system performance to a clear operational objective (e.g., to detect attacks of certain types and sizes), as discussed earlier in this report. Instead, the relaxed requirement was based on ideas about the performance characteristics of the current (Gen-2) system—the idea being that Gen-3 could be less sensitive than Gen-2 but still achieve comparable public health outcomes because of its greater speed. According to DHS, the change in sensitivity requirement was linked to casualty reduction, and the agency does not agree with our assessment. However, as noted earlier in this report, the performance of Gen-2 has not been linked to a clear operational objective; therefore, because the revised sensitivity requirement for Gen-3 was based on Gen- 2, it was not grounded in an operational objective, either. In any future acquisition, upgrade, or enhancement to Gen-2, having initial requirements based more closely on mission need and operational objectives may prevent delays and uncertainty. In its Post Implementation Review, DHS also recognized the need to better communicate with stakeholders about using a flexible testing approach to refine requirements to avoid any misperception that the requirements would be adjusted to accommodate the vendor’s capabilities.

Practice 8: Engage in a Continuous Cycle of Improvement by (1) Conducting Developmental Testing, (2) Conducting Operational Testing, and (3) Incorporating Lessons Learned

DHS’s Actions Aligned with Best Practice 8, as Applicable DHS took steps to engage in a continuous cycle of improvement during the Gen-3 acquisition, but not all components of this practice apply, as DHS never reached the stage of operational testing. In the Gen-3 TEMP, DHS describes an integrated test approach, designed as a continuum from developmental testing through operational testing, with the previous test events acting as the foundation for the follow-on events. However, given the expectation of rapid acquisition and deployment, DHS missed opportunities to fully engage in a broader testing approach needed for a novel system approach to biodetection. By not engaging in a more rigid testing approach, when the autonomous detection systems tested did not yield favorable outcomes, it raised too much uncertainty about cost and performance for the program. As a result, DHS canceled the Gen-3 acquisition and prepared lessons-learned documents for the Gen-3 acquisition that are intended to inform the BioWatch program’s actions in the future. This could include applying the lessons learned to improvements or upgrades to the existing Gen-2 system. OHA officials reported that following the Phase I testing, the BioWatch program facilitated lessons-learned conferences that included relevant stakeholders. For example, OHA officials said results from the Phase I testing indicated that the basic approach used to assess the technology worked well and appeared to answer the relevant questions as to the readiness of a technology considered for deployment. Specifically, they said testing of major subsystems independently was useful as was the chamber testing conducted using a killed (nonviable) agent. Other positive aspects included the usefulness of an interagency test and evaluation Working Integrated Product Team, and the value of independent test agencies. According to OHA, lessons learned include the need for widely accepted PCR Assay Performance standards (recently reviewed by a National Academies consensus committee) and the establishment and enforcement of operational performance criteria during testing to avoid repeated adjustments such as those that were made by one of the vendors to its agent identification algorithm and assay chemistry. According to OHA officials, these adjustments delayed the testing and increased costs.

DHS Identified Key Lessons Learned from the BioWatch Gen-3 Acquisition 1. Any policy decision to accelerate a future acquisition of detection technology should be fully documented, capturing the justification for urgency, an understanding of the limitations of current technology capabilities, and the minimum acceptable (non-technical) requirements needed to achieve the improvement envisioned. 2. Deliberations regarding biodetection research and development during an acquisition process should have pre-designated forums that are capable of resolving issues (scientific or otherwise) as they arise. An inability to reach resolution or consensus should be documented and openly acknowledged, allowing for the development of an adjusted path forward for the acquisition process, if necessary. 3. Communication efforts should be increased to Department stakeholders and leadership, both to improve efficiency of the acquisition process and to fully document an acquisition’s updates to any requirements, timelines and/or procedures. 4. State and local government stakeholders, especially in locally-operated programs (such as BioWatch), should be integrated into the requirements generation process as early as possible, recognizing their ultimate role as the end-user.

In April 2014, after assessing the results of the Phase I testing outcomes, DHS canceled the Gen-3 acquisition when unfavorable testing outcomes raised too much uncertainty about cost and performance of the autonomous detection systems tested. While some might consider the cancellation of the Gen-3 acquisition a failure, by carefully weighing the pros and cons of moving forward with an acquisition that had not produced favorable results, DHS incorporated parts of the best practice of continuous improvement and avoided greater expense for a system that had not met program requirements. By applying the lessons learned that DHS identified as a result of evaluating the Gen-3 acquisition, as well as those identified through application of these best practices for testing, DHS can continue to engage in a continuous cycle of improvement for its testing and acquisition of detection technologies as it considers upgrades or enhancements to the Gen-2 system.

Lessons Learned from Other DHS Acquisitions of Threat Detection Technologies

BioWatch Gen-3 is one of several DHS technical system acquisitions that have faced challenges because of system immaturity or unreliable performance in an operational environment. In some cases, DHS canceled the acquisitions after testing did not yield favorable results. For example, we previously found the Secure Border Initiative Network (SBInet) testing did not appropriately account for risk or provide sufficient information to ensure system performance. Aiming to enhance border security and reduce illegal immigration, DHS launched SBInet to create a “virtual fence” along the border using surveillance technologies. However, as with our 2012 findings on the Gen-3 acquisition, we found the SBInet Program Office had not effectively performed key requirements of development and management practices. For example, some operational requirements for SBInet, which are the basis for all lower-level requirements, were found to be unverifiable, and we concluded that the risk of SBInet not meeting mission needs and performing as intended was increased. As noted above, the Gen-3 acquisition also did not have operational requirements directly tied to mission need. Additionally, we found none of the SBInet plans for tests of system components addressed testing risks and mitigation strategies. As noted above, assessing technical and performance risk in an acquisition can help mitigate cost and schedule problems. Although we made several recommendations aimed at improving the rigor and discipline of SBInet testing, DHS canceled the SBInet acquisition in January 2011, in response to internal and external assessments that identified concerns regarding the performance, cost, and schedule for implementing the systems.

Similarly, we previously reported that a primary lesson learned regarding testing of the Domestic Nuclear Detection Office’s (DNDO) advanced spectroscopic portal (ASP) was that the push to replace existing equipment with new technology led to a testing program that lacked the necessary rigor. The ASP was a type of portal monitor designed to both detect radiation and identify the source to reduce both the risk of missed threats and the rate of innocent alarms. DNDO considered these to be key limitations of radiation detection equipment used by Customs and Border Protection (CBP) at U.S. ports of entry. Based on our body of work on ASP testing, one of the primary lessons to be learned is to avoid the pitfalls in testing that stem from a rush to procure new technologies. We have previously reported on the negative consequences of pressures imposed by closely linking testing and development programs with decisions to procure and deploy new technologies. In the case of ASPs, as well as the Gen-3 acquisition, the push to replace existing equipment with the new portal monitors led to a testing program that initially lacked the necessary rigor. DNDO’s schedule consistently underestimated the time required to conduct tests, resolve problems uncovered during testing, and complete key documents, including final test reports. We also found that testing of the ASPs lacked sufficient rigor to demonstrate performance of the detectors in an operational environment. For example, ASP testing did not include a sufficient amount of the type of materials that would mask or hide dangerous sources that ASPs would likely encounter at ports of entry, which is fundamental to the performance of radiation detectors in the field. As noted above, Gen-3 testing of possible environmental contaminants also lacked sufficient rigor to identify potential vulnerabilities in the system’s detection capabilities prior to placing them in the field. Despite several recommendations we made to DHS on ways to improve the testing and management of the ASP acquisition, because of unsatisfactory test results, ASP did not pass field validation testing, which led DHS to cancel the program in October 2011.

DHS has taken several steps in recent years to improve acquisition management in response to our previous recommendations. By establishing a policy that largely reflects key program management practices, dedicating additional resources to acquisition oversight, and improving documentation of major acquisition decisions in a more transparent and consistent manner, DHS has improved its ability to manage acquisition programs. The decision to cancel the Gen-3 BioWatch acquisition is an example of improved oversight, and DHS could continue to improve by implementing some lessons learned from the Gen-3 acquisition. In April 2015, we reported that DHS’s Director of Operational Test and Evaluation has expressed interest in becoming more involved in testing earlier in the development process to increase influence over program execution. The Director told us that determining how test activities should inform key decisions would help mitigate risk for all types of programs. As DHS considers its options regarding the currently deployed Gen-2 BioWatch system, including possible technology enhancements or even future technology switches, future BioWatch acquisition and testing efforts may benefit by incorporating the lessons learned from the Gen-3 Phase I testing and other recent DHS acquisitions that have faced testing challenges.

PCR Technology Is Most Mature for an Autonomous Detection System to Upgrade Gen-2, but Such Systems Face Uncertain Benefits and Several Likely Challenges

PCR is the most mature and sensitive technology for an autonomous detection system, and DHS is considering autonomous detection as an upgrade to Gen-2. While autonomous detection may provide benefits that include reduction in casualties and clean-up costs and greater cost- efficiency, the potential benefits of an autonomous system for BioWatch depend on specific assumptions, some of which are uncertain. For example, reductions in casualties would depend on a rapid, coordinated response from multiple entities at the federal, state, and local levels; whether such a response would materialize is uncertain and partially outside DHS’s control. Further, an autonomous detection system would have to address several likely challenges, including minimizing possible false positives, securing a networked detection and communication system, and operating under various environmental conditions.

PCR Is the Most Mature Technology for an Autonomous Detection System

The most mature analysis technology for an autonomous detection system is currently PCR, according to a National Academies report on promising technologies for autonomous detection for BioWatch and interviews with stakeholder officials, including CDC, the DHS BioWatch program manager, and other experts. As mentioned earlier, while DHS canceled the Gen-3 acquisition of an autonomous detection system for BioWatch, OHA and S&T are collaborating to address the capability gap that Gen-3 intended to fill by evaluating upgrades or enhancements to the current Gen-2 system. According to DHS officials, autonomous detection is among the technologies being considered. The National Academies report presented perspectives from local officials and technological assessments, gathered from a workshop that DHS requested, to explore alternative cost-effective systems that would meet the needs for a next- generation BioWatch system. This proposed next-generation system was intended to operate autonomously to detect BioWatch threat agents from aerosol samples. The National Academies report described the state of autonomous detection technology in 2013 and evaluated four broad classes of technologies (see table 3). Those technologies were PCR, immunoassays and protein signatures, genomic sequencing, and mass spectrometry.

Three Key Terms for Understanding Autonomous Detection Technologies Genes: Genes are sections of nucleic acids that determine how proteins are made. Nucleic acids, such as deoxyribonucleic acid (DNA), are long chains of molecules made up of bases, of which there are four possible kinds. The order of the bases represents the sequence of the DNA. Because proteins determine much of the function of an organism, sequencing the genes can provide information about its identity. The set of all genes of an organism is known as its genome. Protein signatures: Proteins are long chains of building blocks called amino acids. Because there are many types of amino acids, the way each protein interacts with different stimuli, such as light, can be unique. For example, shining one color of light on a protein can result in its emitting different colors. The set of colors the protein gives off can be considered an “optical fingerprint” of the protein, commonly referred to as a type of protein signature. Antibodies: Antibodies are proteins created by the immune system that attach to specific chemicals on the surface of disease-causing organisms, resulting in the organisms’ being rendered harmless. Antibodies can be designed to attach to a given target and modified to be detectable under certain conditions. Thus it is possible to use antibodies to see whether a target is present by soaking a sample with the antibodies, washing off unattached antibodies, and then measuring those that remain.

PCR, which is used to detect nucleic acid signatures, is used to amplify and detect genetic material, or nucleic acids, of organisms. By amplifying (i.e., repeatedly duplicating) those sections of genes associated with certain biological agents, it is possible to distinguish the agents among various organisms. Because of the amplifying capability of PCR, small amounts of genetic material are sufficient for detection, resulting in high sensitivity for this technology. Specificity can be high if the sections of the genes being amplified are unique to the agent. However, related organisms, called genetic near-neighbors, may contain similar gene sections and lead to a PCR detection when the agent itself is not present. PCR is the method used in the current (Gen-2) BioWatch system.

Immunoassays and protein signatures use antibodies or light to identify organisms. Immunoassays use antibodies that attach to chemicals that primarily appear on certain biological agents; thus immunoassays can be tailored for high specificity. Protein signatures analyze how light interacts with different chemicals (such as proteins) on target agents, using light “signatures” emitted by specific proteins to be identified. However, neither of these methods is as specific as PCR. Also, because there is no amplification, the sensitivity of these methods is not as high as that of PCR.

Genomic sequencing provides a genetic sequence for all or part of a detected organism’s genes. Because each agent contains unique genetic sequences, this method is very specific and could eventually provide information regarding antibiotic susceptibility. However, the method is not considered standalone since it depends on another method, such as PCR, to work. If used with PCR, then this method is also very sensitive. Of the four broad technologies examined by the National Academies, this method is also the least mature because of issues with systems integration—for example, developing the software to perform the analysis locally (within the device itself).

Mass spectrometry breaks apart a sample (for example, by directing a laser onto the sample) and analyzes the resulting fragments. Different chemicals yield different types and amounts of fragments, so it is possible to reconstruct the chemical composition of the original sample. Because the chemical make-up of agents is unique, it is possible to identify their presence in the sample. Mass spectrometry is not as sensitive or as specific as PCR.

Benefits of an Autonomous Detection System Depend on Several Assumptions

We identified key potential benefits of an autonomous detection system from discussion with agency officials, a review of agency and national laboratory documentation, and a literature review. Most of these potential benefits were owing to faster detection; however, we determined that the extent to which faster detection confers benefits depends on specific assumptions, some of which are uncertain and some of which are outside of DHS’s control. Additionally, from our review of literature, we identified potential benefits that included decreasing user errors, such as dropped collection filters. However, since these benefits depend on the actual design and implementation of the system, it is difficult to predict the extent to which they would be realized. The benefits and challenges discussed in this report apply broadly to autonomous detection; that is, they do not depend on which of the four broad classes of technology is deployed.

According to a 2011 National Academies report, an autonomous detection system could detect agents more quickly than the Gen-2 system because of a shorter sample collection period, elimination of sample transport, and completion of the detection step within the system itself (see fig. 3). In particular, the report stated that while the current Gen-2 BioWatch system could detect agents in 10-36 hours, an autonomous detection system could detect agents in as little as 4-6 hours. Further, DHS officials and Sandia modeling studies state that faster detection enabled by automation can provide: 1. reduction in casualties and/or fatalities because of faster detection and faster situational assessment; 2. lowering costs, including clean-up costs by halting the entrance of transport vehicles, such as trains or airplanes, into contaminated areas; and 3. reduction in the total annual cost of the detection system, per detection cycle.

DHS officials told us that an autonomous detection system would offer many of these benefits but did not provide evidence to support them, saying that their assertion is “common sense.” DHS officials also referred to Sandia’s modeling studies. However, we determined that these benefits and the conclusions of the Sandia modeling studies depend on specific assumptions, some of which are uncertain and some of which are outside of DHS’s control, although DHS officials stated that they believe the modeling assumptions are reasonable for the intended purpose. However, a CDC official cautioned against relying on models to determine program effectiveness.

Life-Saving Benefits of Early Detection Are Uncertain and Depend on Factors Outside of DHS’s Control

The number of lives saved from faster detection could not be determined, because some key factors affecting response time are uncertain. For example, the time it takes decision makers to determine that a detection represents a threat to public health and warrants dissemination of medical countermeasures is variable. The time between a BAR and dissemination of medical countermeasures may include the time needed to characterize the incident, determine who was exposed, make decisions regarding evacuation of contaminated regions and relocation of individuals, determine where to set up medication “points of dispensing” and to actually mobilize the medication stockpile, and distribute medication to potentially exposed people and keep track of who received medication. According to the National Academies report and current DHS guidelines, steps taken after detection, to instill confidence for requesting medication, include assessing known threats, conducting additional local lab work, and initiating a national conference call (see fig. 4). There may be additional tasks such as culturing of the agents to determine their viability and antibiotic resistance. If local stakeholders follow the guidelines, there could be considerable delay prior to mobilization of stockpiled medication. For example, an official at the National Academies workshop reported that he takes an additional hour to perform an assessment prior to any national conference call. According to another National Academies report, the BioWatch national conference call usually occurs 1-2 hours after a local call of the BioWatch Advisory Committee. Thus, the time between an attack and when medication is fully distributed—and the number of lives potentially saved by minimizing this time—could vary from jurisdiction to jurisdiction. DHS officials agreed that the jurisdictional response can vary and conducts exercises and training to help plan for a response. However, it is not clear what effect such exercises have on response time variability.

In addition, faster detection may not be the most effective way to save lives. For example, a modeling study showed results indicating that an attack detected in 2 days, but requiring 10 days to distribute medication, would result in more deaths than an attack detected in 5 days, but requiring only 2 days to distribute medication. Thus, according to this model, a jurisdiction that shifts resources from medical distribution capacity to faster detection may end up with more deaths. Decisions of resource prioritization are not under BioWatch program control, neither is the part of the response involving medication or other intervention. However, the benefits from early detection depend on such resource prioritization and effective overall responses. DHS officials agree, noting that biosurveillance is a coordinated, holistic endeavor.

Sandia ran a response model to estimate, among other things, the number of casualties and fatalities given the time that passes between the attack and detection (time to detection). Sandia’s modeling studies showed that a faster detection system would reduce the number of casualties and fatalities, but that the extent of these reductions would depend on assumptions in the model. One such assumption was the probability that BioWatch correctly detected the attack. As discussed earlier in this report, the probability that BioWatch correctly detects an attack depends on many factors, including the performance characteristics of the technology and the characteristics of the attack itself. Many of Sandia’s estimates of the life-saving benefits of a detection system—automated or not—are downstream analyses presuming that an attack was detected. If the attack was not detected, the faster response enabled by a detection system would not occur, and there would be no life-saving benefits from operating such a detection system. Therefore, if those results are read out of context of this presumption, the expected reductions in casualties and fatalities may be overestimated. This limitation applies to autonomous detection as well as the Gen-2 system.

Estimated reductions in casualties and fatalities from faster detection also depend on assumptions about the infectivity of the BioWatch threat agents. In the Sandia modeling studies, infectivity was represented by infectious dose estimates—that is, estimates of the doses that would lead to illness; however, we found uncertainty in these estimates. As described in our earlier discussion of the current (Gen-2) system, Sandia researchers and other experts told us there is considerable uncertainty in even the best available infectious dose estimates for anthrax, as these estimates are based on data from nonhuman primates.

Finally, the life-saving benefits of faster detection that Sandia reported varied significantly, depending on the properties of the illness that the agent caused. These properties included how long it takes for symptoms to exhibit in a patient after exposure and how effectively medication can prevent death in ill people. According to Sandia, some agents act very slowly—that is, they have long incubation periods—which diminishes the effect of faster detection. For example, an agent that takes over 7 days to cause symptoms will be detected by a 36-hour and a 4-hour detection system with similar outcomes. Another factor is how effectively a developed illness can be treated. According to Sandia, this factor is also variable, so that for some agents, the numbers of lives saved in shifting from 36-hour to 4-hour detection change little. Thus, reducing fatalities by faster detection depends largely on the agent used in an attack.

The Lowering of Cleanup Costs Is Uncertain because of Traffic Variability

Sandia reported that faster detection improves the ability to divert transport vehicles—such as trains and airplanes— from contaminated areas so that they do not have to be subsequently cleaned up. The benefit’s extent is uncertain because it depends on the amount of traffic entering a given location. For example, according to Sandia, the number of subway cars entering New York City’s Grand Central Terminal over a period of 5 hours can range from as few as 250 to as many as 1,750, depending on the time of day and the day of the year. Clean-up effort reduction is thus uncertain because of an attack’s unpredictability. A similar analysis can be made for people entering a contaminated area— while early detection could lead to exposure prevention and mitigating the need for additional medication, the actual number of people affected is similarly variable.

We recognize that much of the uncertainty described regarding lives saved or reduction in clean-up costs is out of the control of the BioWatch program. However, when describing benefits of faster detection, particularly concerning the number of lives possibly saved by an autonomous system or any early-warning system, it is important to understand the uncertainties in these assumptions, such as response time or infectious dose of the agent. Without a comprehensive enumeration of these assumptions and their effects on the modeling results, assertions regarding the value of autonomous detection systems are questionable.

The Projected Cost Savings for an Autonomous Detection System May Not Materialize

In 2012, we found that DHS performed a limited cost trade-off assessment of switching to an autonomous detection system that focused on cost per detection cycle—that is, the cost each time an autonomous detector tests the air for agents versus the cost each time a Gen-2 filter is manually collected and tested in a laboratory. We reported in 2012 that cost per detection cycle was lower with an autonomous detection system, but that overall annual program costs would increase from the current Gen-2 system program costs. From figures DHS provided in 2015 regarding the cost of switching to an autonomous detection system with coverage comparable to that of the current Gen-2 system, our analysis yielded results similar to our 2012 findings.

To determine potential cost savings between the Gen-2 system and an autonomous system, DHS compared an autonomous system with a modified Gen-2 BioWatch system. Gen-2 generally runs one detection cycle daily, but for DHS’s analysis the agency compared an autonomous system with a modified Gen-2 system which would run three daily detection cycles. We determined that total annual program costs would increase if current operations for Gen-2, which involve one detection cycle per day, were replaced with an automated detection system. Only by comparing the total annual program costs of operating Gen-2 with three detection cycles per day with the total program costs of operating an automated system were cost savings realized (see table 4). As we reported in 2012, conducting a more complete analysis of costs and benefits would help DHS develop the kind of information that would inform trade-off decisions regarding changes to BioWatch technology.

Automation may lead to additional potential benefits including fewer user errors and greater efficiency—for example, using fewer resources to accomplish the same amount of work—and greater worker safety by facilitating the handling of dangerous materials, according to literature on automation. However, because automated detection systems have not been deployed, assessing these benefits is difficult. Additionally, uncertainty about how the technology will work means that its benefits might be countered by new problems. For example, according to a 2007 DHS Inspector General report, transferring BioWatch system filters was done improperly several times in 2004. By eliminating the need for transporting filters, automation could avoid this problem, but new problems could arise, such as system crashes. For example, repeated system crashes occurred when the BioWatch Program Office conducted a trial deployment of an autonomous detection system, in New York in 2008. Thus, it is not clear that an autonomous system would realize these benefits.

Autonomous Detection Systems Must Overcome Several Likely Challenges

The challenges an autonomous detection system must overcome include ensuring its detection sensitivity and protecting against threats to networked communications. From a National Academies workshop and interviews with agency officials, we identified five likely challenges (shown in fig. 5).

According to DHS, ensuring that the autonomous detection system meets BioWatch sensitivity requirements represents a major technical hurdle. As discussed earlier, the original sensitivity requirement for the Gen-3 system was based on a technology push because DHS lacked the analytical tools needed to generate a mission-based sensitivity requirement. DHS later revised the sensitivity requirement based on a Sandia-led modeling study. As we described earlier, the Sandia model is subject to limitations and assumptions, and how the sensitivity requirement may be linked to mission outcomes, such as detecting attacks that lead to 10,000 casualties, remains uncertain. Additionally, challenges may be associated with designing a technology to meet a given sensitivity requirement.

One way to manage sensitivity requirements is to assess whether the technology in a detection system conforms to performance standards. The standards may be subject to validation by independent groups or agencies, and constitute guidelines for the technology. For example, the number of times a test should be run and the verification of reagents are standardized so that results can be interpreted meaningfully. With the development of newer technologies for detection, a method known as multiplexing is being increasingly used. However, validating the use of multiplexing in detection systems has no performance standards. DHS commissioned the National Academies to examine performance standards for PCR, including multiplexed PCR. However, the report recently released by the National Academies does not provide clear standards for multiplexing, instead noting that combining Food and Drug Administration (FDA) multiplexing guidance with certain standards, such as SPADA, which discuss multiplexing, should form a starting point for validation testing. The report also notes that changing to multiplexing (from singleplexing) reduces the sensitivity of the assay, although the effect of this reduction is unclear. Given that PCR is the most mature technology for autonomous detection systems, implementing other technologies may require similar, or greater, effort in establishing performance standards.

The Challenge of False Positives

A challenge for deploying autonomous detection systems identified by participants at the National Academies workshop is the avoidance of false positive readings—readings that indicate the presence of an agent that is not present. False positive readings can lead to major disruption from shutting down crucial transportation and economic facilities (such as airports and shopping centers—referred to as high-consequence actions) and to the unnecessary medication of an uninfected population—which can lead to adverse effects and medical stockpile waste. Local public health officials stated that false positives are likely to be a bigger issue with autonomous detection systems, because operating more detection cycles could increase their frequency.

According to the National Academies report, another common concern among public health officials is their credibility when making high- consequence decisions. At the workshop, officials stated that the integrity of public health is critically important, and thus they needed complete confidence in an autonomous system, which is intended to provide results without human interaction or interpretation. Similarly, according to a Sandia workshop in 2009, public health officials largely felt that wrongly taken high-consequence actions would result in loss of credibility. Finally, an LLNL scientist stated that debugging a complex system to determine whether a potential false positive occurred may be an issue with some autonomous systems. He noted that false positives from naturally occurring genetic near-neighbors of the BioWatch threat agents might be a particular challenge and that DHS has made limited investments in determining background DNA signatures to address this issue. DHS officials stated they use data gathered from current operations to assess such background signatures, but it is unclear whether this approach would be effective for an autonomous detection system.

Ensuring Reliable and Secure Information Technology Networks Is Challenging

According to DHS and CDC officials, another challenge autonomous detection systems face is securing the networked communication system against interference, such as from hackers. DHS officials stated that the security of network communications for transmitting results to the local officials was an important issue for autonomous detection systems. For example, during the Gen-3 effort, DHS officials specifically planned for testing of network security as described in the TEMP. DHS officials stated that an unsecure system would be vulnerable to hackers’ planting results or shutting systems down. In 2012, we reported that the 2011 Operational Assessment stated that failure to demonstrate network security may seriously inhibit user confidence in the system. A CDC official also noted that network communication is an area of concern, citing previous issues with the deployment of related technologies.

Data Management and Interpreting Results Are Challenging

DHS identified data management challenges for autonomous systems, including reviewing the reported data and interpreting the data to determine appropriate follow-up actions. A participant at the National Academies workshop expressed concern over a system that would provide data every few hours, leading to strain on limited and diminishing public health resources. DHS describes system data as containing information on how the detector was functioning as well as laboratory analysis data. According to DHS, data from an autonomous detection system would need to be reviewed by local or state staff across the 24/7 reporting period. Further, those staff would need to be trained in appropriate data interpretation. According to DHS officials, the cost for these local public health resources is not included in their cost projections of the autonomous BioWatch calculations.

Maintaining Autonomous Detection System Operation Is Challenging

Finally, DHS officials stated that keeping the autonomous detection system continuously functioning in a dirty environment is challenging. Additionally, an LLNL official stated that a dirty environment can contain chemicals that interfere with the technology used to detect the agent. As we reported in 2012, and according to the 2011 Operational Assessment on Gen-3, autonomous detection systems during the Gen-3 acquisition experienced malfunctions, exhibited issues with the positive controls, and required unscheduled maintenance, attributed to either traffic emissions due to proximity to an interstate, or to metallic dust generated by train brakes. This underscored the challenge of an autonomous detection system needing to operate in different operational environments. In addition, according to the Analysis of Alternatives conducted by the Institute for Defense Analyses in 2013, detection systems are vulnerable to vandalism and accidents.

Conclusions

BioWatch’s rapid deployment in 2003—to provide early detection of potentially catastrophic aerosolized biological attacks—did not allow for sufficient testing and evaluation against defined performance requirements to understand the system’s capabilities. Since that time, DHS has commissioned tests of the system, but has not defined technical performance requirements that would link test results to conclusions about the types and sizes of attack that the Gen-2 system could reliably detect. DHS has also commissioned modeling and simulation studies, but none of these studies was designed to directly and comprehensively assess what is known about the capabilities of the currently deployed system, using specific test results and accounting for statistical and other uncertainties. Finally, while DHS has addressed certain limitations in testing, it has not systematically tested the system against realistic conditions, and there remains potential to reduce risk and uncertainty in what is known about the system’s capabilities when deployed in a real- world environment. As a result of these gaps and limitations, considerable uncertainty remains as to the types and sizes of attack that the Gen-2 system could reliably detect. DHS officials have stated that the system’s operational objective is to detect attacks large enough to cause 10,000 casualties, but DHS cannot conclude with any defined level of statistical certainty that the system can reliably achieve this objective. In the wake of the cancellation of the Gen-3 acquisition, DHS is planning for technology upgrades or improvements to the Gen-2 system, and some Gen-2 equipment is nearing the end of its life-cycle and will need to be replaced if the program is to continue. However, effective and cost- efficient decisions cannot be made regarding upgrades and reinvestments if the operational capabilities of the Gen-2 system are uncertain. Assessing the operational capabilities of the Gen-2 system against technical performance requirements directly linked to an operational objective, incorporating specific test results, and explicitly accounting for statistical and other uncertainties would help ensure that decisions about future investments are actually addressing a capability gap not met by the current system and address a clear mission need.

In recent years, DHS has canceled major acquisitions that we previously found could have been more rigorous in their test design or execution, including Gen-3. The nation’s ability to detect threats against its security requires judicious use of resources directed toward systems whose capabilities can be demonstrated. Applying lessons learned from the Phase I testing of Gen-3 candidate technologies, as well as incorporating the best practices we identified, may help enable DHS to mitigate risk in future acquisitions for these types of threat detection technologies. Specifically, DHS could apply them to the BioWatch program once informed decisions have been made regarding upgrades or enhancements to Gen-2.

Furthermore, DHS officials have continued to express interest in an autonomous detection capability as a possible upgrade or enhancement to Gen-2. If DHS were to pursue an autonomous detection system in the near future, PCR would be the most mature technology available. However, the extent to which the potential benefits of such a system would materialize is uncertain, because of uncertainty in the assumptions upon which these benefits depend. Additionally, pursuit of such a system faces several likely challenges. Understanding the inherent challenges to faster detection and contextualizing the benefits of autonomous detection technologies will help decision makers make informed decisions regarding use of limited resources. BioWatch is just one biosurveillance activity used to detect potential biological threats to our national security, and as we have previously reported, because the nation does not have unlimited resources to protect the nation from every conceivable threat, it must make risk-informed decisions regarding its homeland security approaches and strategies. In July 2012, the White House released the National Strategy for Biosurveillance to describe the U.S. government’s approach to strengthening biosurveillance, but it is too soon to tell what effect the strategy and corresponding implementation plan may have on determining resource allocation priorities across the interagency. As some Gen-2 equipment reaches the end of its lifecycle, DHS will need to make decisions about investing in the future of the BioWatch program. DHS initiated the BioWatch program in 2003 to address the perceived threat at the time. Since then, numerous Bioterrorism Risk Assessments have been issued, but these have been criticized for the methodology, and none has been issued in the last 5 years. Consequently, as the National Academies has noted, there is considerable uncertainty about the likelihood and magnitude of a biological attack. Investment decisions about the future of BioWatch should be guided by the agreed-upon priorities of the various stakeholders within the biosurveillance community to help ensure investments address the current threats posed by biological hazards.

Recommendations for Executive Action

To help ensure that biosurveillance-related funding is directed to programs that can demonstrate their intended capabilities, and to help ensure sufficient information is known about the current Gen-2 system to make informed cost-benefit decisions about possible upgrades and enhancements to the system, the Secretary of Homeland Security should direct the Assistant Secretary for Health Affairs and other relevant officials within the Department to not pursue upgrades or enhancements to the current BioWatch system until OHA: establishes technical performance requirements, including limits of detection, necessary for a biodetection system to meet a clearly defined operational objective for the BioWatch program by detecting attacks of defined types and sizes with specified probabilities; assesses the Gen-2 system against these performance requirements to reliably establish its capabilities; and produces a full accounting of statistical and other uncertainties and limitations in what is known about the system’s capability to meet its operational objectives.

To help reduce the risk of acquiring immature detection technologies, we recommend that the Secretary of Homeland Security direct the Assistant Secretary for Health Affairs, in coordination with the Under Secretary for Science and Technology, to use the best practices outlined in this report to inform test and evaluation actions for any future upgrades or changes to technology for BioWatch.

Agency Comments and Our Evaluation

In written comments provided in response to our draft report, DHS concurred with our recommendations and described actions the agency is taking to address them. DHS also provided technical comments, which we incorporated as appropriate. DOE provided technical comments, which we incorporated as appropriate. CDC and DOD reviewed the draft report and provided no comments. DHS’s written comments are reproduced in full in appendix IV of this report.

DHS concurred with our recommendation to establish technical performance requirements, including limits of detection, necessary for a biodetection system to meet a clearly defined operational objective for the BioWatch program by detecting attacks of defined types and sizes with specified probabilities. DHS stated the BioWatch program has already completed a series of tests that establish the performance and capabilities of currently deployed technologies and provide baseline performance requirements for any future technological improvements. DHS also stated the BioWatch program will consider including additional measures of system performance, such as probability of detection, to augment and validate the system’s ability to detect attacks, pending available resources and at a time yet to be determined.

However, using existing test results as a baseline for future technological improvements provides no information about the current—or any future— system’s ability to meet a clearly defined operational objective. DHS should first establish requirements for the current system, which will enable DHS to assess its system performance measures, such as sensitivity, against its stated mission goal: to detect attacks causing 10,000 casualties. Without establishing such performance requirements, the agency does not know what the existing test results mean for the system’s ability to detect attacks, and thus cannot establish the benefits of any future improvements. Further, DHS mentioned using additional system performance measures to augment and validate the system’s capability. We emphasize that the program’s current preferred measure of system performance, fraction of population protected (Fp), does not have a clear linkage to the system’s operational objective. What is needed is a measure that directly supports conclusions about the system’s ability to meet its objective by detecting attacks of defined types and sizes.

DHS concurred with our recommendation to assess the Gen-2 system against the performance requirements described above to reliably establish its capabilities, and stated that the results of the testing and evaluation events referred to above have already been incorporated into existing modeling and simulation studies. DHS also stated, however, that should a significant difference between two performance measures—Fp and probability of detection (Pd)—be observed, the BioWatch program would consider additional modeling and simulation studies to determine the performance capabilities of the deployed Gen-2 BioWatch system.

While DHS’s response suggests that it has largely met this recommendation already by having tested the system and incorporating the test results into modeling and simulation studies, this conflicts with what we have been told by BioWatch program officials. These officials told us that they have not commissioned or produced an analysis in which the best available test results have been combined with modeling and simulation to reach specific conclusions about the system’s ability to detect attacks of defined types and sizes. As we detailed in the report, modeling and simulation studies commissioned by DHS either considered ranges of hypothetical values for the system’s sensitivity or else involved old test results, based on an earlier version of the system, for just one of the BioWatch threat agents. Furthermore, we identified important limitations in the tests DHS has conducted that could be addressed through a more systematic approach to reducing risk and uncertainty in what is known about the system’s capabilities when deployed in a real- world environment. While it is true that the system cannot be tested directly by releasing live biothreat agents in the environments where BioWatch is deployed, both the National Research Council and subject matter experts with whom we spoke identified methods by which the system can be tested and its performance characteristics estimated in more realistic environments.

DHS concurred with our recommendation to produce a full accounting of statistical and other uncertainties and limitations in what is known about the system’s capability to meet its operational objectives. DHS stated it already has sufficient understanding of the statistical uncertainties and limitations associated with testing and modeling of the BioWatch system. However, DHS also agreed that there is value in consolidating this information into a single, comprehensive document and plans to do so by April 30, 2016.

As described in the report, DHS does not have a sufficiently comprehensive accounting of the uncertainties and limitations in what is known about the system’s capabilities. A comprehensive analysis of uncertainties and limitations should account for how such uncertainties and limitations affect the key outcome: the system’s ability to meet its operational objective by detecting attacks of defined types and sizes. Statistical uncertainties should be represented with clearly defined confidence intervals; for uncertainties that are difficult to quantify, such as uncertainties associated with testing in chambers rather than operational environments, the judgments of subject matter experts may be useful. This full accounting of uncertainties and limitations should be provided to administration and congressional decision makers so they better understand the precision in what is known about the system’s capabilities in an operational context. Decision makers should be able to use this information not only when comparing the costs of the current system to the benefits it may provide, but also when weighing decisions about proposed upgrades or enhancements to the system.

DHS concurred with our recommendation to use the best practices we outline in the report to inform test and evaluation actions for any future upgrades or changes to technology for BioWatch. DHS stated that changes to BioWatch will adhere to new DHS acquisition guidance that incorporates the best practices outlined in our report.

DHS’s reference to new acquisition guidance is to DHS-wide guidance that was issued in 2010, after DHS began testing the Gen-3 technology. This guidance includes additional detail on factors to consider when planning and testing new acquisitions and addresses many of the practices described within this report. However, when it comes to ensuring the acquisition will not only meet technical requirements but also perform as intended in an operational environment, we believe more robust testing earlier in the acquisition to test resilience can help reduce the risk of acquiring immature technologies. This is especially important for a system like BioWatch, which cannot be fully tested in an operational environment. While we see proper implementation of DHS’s updated acquisition guidance as a positive step towards addressing our recommendation, as we reported in April 2015, DHS’s Director of Operational Test and Evaluation expressed interest in becoming more involved in testing earlier in the development process. Therefore, we believe the lessons learned on Gen-3 testing and full adoption of testing practices aimed at establishing the operational performance of a system earlier in the acquisition should also be considered to help inform future DHS decisions.

While DHS concurred with the three parts of our first recommendation, the agency did not agree with key findings that led to these recommendations; therefore, it is important to address parts of their response for clarification.

DHS took exception to our conclusion that it has not defined technical performance requirements that would link test results to conclusions about the types and sizes of attack that the Gen-2 system could reliably detect. DHS stated it uses the metric called fraction of population covered (Fp) to make this linkage. However, when asked about this directly, agency officials declined to explain how specific values of Fp would enable DHS to conclude what types and sizes of attack the system can detect. Furthermore, officials said they have not commissioned or produced an analysis in which the best available test results are used to calculate Fp values and draw conclusions about the system’s ability to detect attacks of defined types and sizes. How a given value of Fp would provide information about the types and sizes of attacks BioWatch Gen-2 can detect remains uncertain, and how Fp relates to the probability of detecting attacks large enough to cause 10,000 casualties—DHS’s stated objective for the BioWatch program—remains unclear. As we note in this report, we recognize that Fp is a useful metric for certain purposes, but it does not directly support conclusions that align with the BioWatch operational objective.

DHS stated that it disagreed with the conclusion that the BioWatch Program does not incorporate empirical data gathered on the current Gen-2 system to inform modeling and simulation studies. However, DHS incorrectly attributed this conclusion to us. We did not state that DHS did not use any empirical data to inform their modeling and simulation studies. We stated that (1) the modeling and simulation studies did not incorporate specific, best available test results (for example, particular estimates of the system’s limits of detection) to draw specific conclusions about the BioWatch Gen-2 system’s capability to detect attacks of defined types and sizes, and (2) the modeling and simulation studies did not incorporate uncertainties in the empirical test results that are important for understanding the precision or confidence in the modeling and simulation results.

Finally, DHS acknowledged the evolving threat of bioterrorism and its continued commitment to following DHS-wide acquisition policy for any future upgrade or enhancement to the current BioWatch system. Analogous to what we reported in 2012 regarding the Gen-3 acquisition, by ensuring any future upgrades or enhancements to the BioWatch system align with the earliest steps in DHS’s acquisition process, such as being grounded in a justified mission need, and reflect a systematic analysis of costs, benefits, and risks, DHS can gain assurance that it is pursuing an optimal solution. Because, as DHS stated, the threat of bioterrorism continues to evolve, and because the last full Bioterrorism Risk Assessment (BTRA) was issued in 2010, it will be important for DHS to demonstrate that any proposed upgrades or enhancement address the threat posed by the intentional release of select aerosolized biological agents at the time upgrades are considered.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretaries of Homeland Security, Health Human and Services, Defense, and Energy; and interested congressional committees. The report is also available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact Tim Persons at (202) 512-6412 or personst@gao.gov or Chris Currie at (404) 679-1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

The objectives of this report were to discuss: (1) the extent to which the Department of Homeland Security (DHS) has assessed the technical capability of the currently deployed system (Gen-2) to detect a biological attack; (2) the extent to which DHS adhered to best practices for developmental testing during Gen-3 Phase I, and what lessons can be learned; and (3) the most mature technology for an autonomous detection system, as well as what the potential benefits and likely challenges would be if DHS were to pursue an autonomous detection system for the BioWatch program in the near future.

To determine the extent to which DHS has assessed the technical capability of the Gen-2 system to detect an attack, we reviewed and analyzed test reports and other agency and agency-commissioned documents containing information on the design, development, deployment, and technical performance characteristics of the system. We also reviewed reports of modeling and simulation studies, conducted by Department of Energy (DOE) national laboratories for DHS, that analyzed the performance and capabilities of the system. We interviewed DHS officials from the BioWatch Program Office and from the Science and Technology Directorate (S&T) who had knowledge of the history of the program, the Gen-2 technology and changes that had been made to the technology over time, and the tests and studies that had been conducted on the Gen-2 system’s technical capabilities. We also interviewed officials and researchers who conducted or were familiar with the tests and the modeling and simulation studies; these included officials and researchers at Dugway Proving Ground, Sandia National Laboratories, Lawrence Livermore National Laboratory, and Los Alamos National Laboratory. In interviews with researchers who had conducted tests and studies, we questioned them about the scope and purposes of their work; the methods they had used; conclusions drawn, as well as any caveats on those conclusions; and the strengths and limitations of the tests and studies. We conducted a site visit to Dugway Proving Ground and saw facilities and equipment that had been used to test the Gen-2 system, as well as facilities under construction that could potentially be used for future testing of the BioWatch system. To assess the strengths and limitations of tests and studies of the Gen-2 system, we used (1) a framework for testing and evaluation of biodetection systems developed by the National Research Council, (2) leading practices in risk analysis and cost-benefit analysis, and (3) judgment of internal (GAO) and selected external experts in the fields of engineering, aerobiology, microbiology, and testing and evaluation of biodetection systems. To gather information on the field of biodetection and the strengths and limitations of alternative technologies, we attended two conferences on biodetection technologies.

To determine whether DHS’s actions during Gen-3 Phase I adhered to best practices for developmental testing and to identify lessons that could be learned, we reviewed the best practices previously developed in conjunction with the National Academes to assess their appropriateness to our review. We consulted with GAO specialists familiar with the best practices for developmental testing to discuss their proper application. We determined the practices that could be applied to Gen-3 Phase I testing, as the testing was developmental in nature and presented opportunities for DHS to de-risk the Gen-3 acquisition, which is the intent of the practices—to de-risk acquisitions of binary threat detection technologies by the government. We analyzed Gen-3 Phase I acquisition and testing documents, such as the test and evaluation master plans, individual test plans and results, and the operational requirements documents. We analyzed other DHS documentation on lessons learned, including the Post Implementation Review assessment, in which DHS identified its own lessons learned on the Gen-3 acquisition. We reviewed the acquisition decision memorandum on the cancellation of the Gen-3 acquisition.

We interviewed DHS officials in the BioWatch Program Office, the Office of the Director of Test and Evaluation at the Science and Technology Directorate, and officials at the national laboratories and Department of Defense (DOD) test agencies who were familiar with the testing performed during Gen-3 Phase I. We collected information from these officials on DHS’s actions and decisions during Phase I testing and compared that with the recommended actions outlined in the best practices for developmental testing. We also compared the steps outlined in the test planning documents with the recommended steps described in the best practices. We consulted with internal and external experts on our assessment of DHS’s actions and decisions compared with the best practices and acquisitions more broadly. We reviewed prior GAO reports on the Gen-3 acquisition and the biosurveillance enterprise. We also reviewed prior GAO work on other DHS acquisitions that met challenges during early phases of testing to draw comparisons with other DHS acquisitions that may have benefited from more robust testing guidance.

To develop the best practices for developmental testing of binary threat detection systems, we conducted a 1-day meeting on June 4, 2013, with 12 experts we selected with assistance from the National Academies. These experts were from academia, industry, and the federal government and had experience in developmental testing methodologies, binary threat detection systems, automatic target recognition, and advanced imaging technologies, from fields that included homeland security, defense, and standards development. To identify the experts, the National Academies considered experts with previous experience on appropriate National Academy studies, requested suggestions from the members of the National Academies’ National Materials and Manufacturing Board and the Computer Sciences and Telecommunications Board, searched internal databases and the Web, and contacted other relevant individuals for recommendations. We facilitated the experts’ identification of best practices with pre-meeting interviews, structured questioning during the meeting, and post-meeting expert voting and ranking procedures. According to the experts, the best practices apply to the process of developmental testing of binary threat detection systems; they also apply if the system is commercial-off-the- shelf (COTS), modified COTS, or newly developed for a specific threat detection purpose being created by a vendor or the government.

To identify the most mature technology for autonomous detection, we reviewed a report of a 2013 workshop conducted by the National Academies that assessed the state of technologies that are potentially suitable for autonomous detection for the BioWatch program. We also interviewed officials at the Centers for Disease Control and Prevention, the Department of Homeland Security’s Office of Health Affairs, Lawrence Livermore National Laboratory, and the Department of Defense who were familiar with BioWatch and biodetection technologies to gather their views on the state of autonomous detection technology. A conclusion of the National Academies workshop held in 2013 was that the polymerase chain reaction (PCR) was the most mature technology suitable for autonomous detection for BioWatch. As a check for any more recent developments that might affect this conclusion, we performed a literature review of journals and conference proceedings published since 2012 to identify any technologies potentially more mature than PCR based on the following criteria: 1. whether the detection technology is specified, meaning the technology is defined and not just referred to as biodetection or detection technology; 2. capacity to detect at least bacteria and virus; 6. having both indoor and outdoor performance capabilities in realistic 7. ability to detect independently (standalone); 8. technology readiness level (TRL) of 6 or higher, if reported; 9. sampled from aerosols/air; 10. whether the technology is used for disease surveillance or modeling instead of pathogen detection; and 11. whether the technology depends on, or is a variant of, PCR.

We excluded press releases and news articles, studies that did not include sufficient detail for evaluating technological detection capability, technologies that were intended for non-aerosol detection (such as for food or clinical specimen testing), or technologies that were intended to be used alongside other technologies for detection (for example, used to supplement or verify a finding, or used as a trigger warning system). Our literature review was not intended to be a comprehensive examination of all technologies that might possibly be applied to BioWatch, but rather a supplement to the National Academies workshop report and a check to help ensure that the conclusions of that workshop were not affected by more recent developments in the field.

To assess the potential benefits and likely challenges of autonomous detection, we analyzed reports published by the Sandia National Laboratories, as well as our prior work on the Gen-3 BioWatch system. We performed a literature review for models of how response timing to a positive detection of agent release may affect response effectiveness, in terms of lives saved. We searched for models published in the last 12 years, a range that was designed to cover work done following the anthrax attacks of 2001. We interviewed officials at the Centers for Disease Control and Prevention, the Department of Homeland Security’s Office of Health Affairs, and Lawrence Livermore National Laboratory to gather their views on the potential benefits and likely challenges of autonomous detection in the near future, which we defined as the next 5 years. Additionally, we reviewed Gen-3 BioWatch testing reports to identify likely challenges to autonomous detection systems.

To determine the potential cost saving benefits of an autonomous detection system, we analyzed cost data provided by DHS. The agency provided annual operation and maintenance costs and total annual program costs for the current BioWatch system under the assumption that detection cycles would be increased to three per day (up from once per day, which is the current practice in most jurisdictions), as well as total annual costs for running a hypothetical autonomous detection system with six to eight detection cycles per day, with comparable coverage. The total annual cost for operating the current BioWatch system was calculated by dividing the annual operation and maintenance costs by three but keeping the remaining costs constant under the assumption that non-operation and maintenance costs remain the same.

Our analysis of potential benefits and likely challenges represent key ones that were identified by the sources listed above, and is not intended to be comprehensive. In particular we did not assess or mention characteristics that were difficult or impossible to meaningfully discuss within the context of this report (for example, deterrent effects of a biodetection system, or finding qualified personnel to hire). For benefits, we focused primarily on reports published by Sandia National Laboratories because they focused most directly on the BioWatch program.

To help collect and analyze information for all three of our research objectives—and to help ensure the technical accuracy of our work—we consulted with subject matter experts under contract with GAO in the fields of aerobiology, microbiology, and biodetection.

We conducted this performance audit from December 2013 to October 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: The Department of Homeland Security Has Made Adjustments to the Gen-2 System Designed to Reduce False Positives

In the Gen-2 system, if the polymerase chain reaction (PCR) assays used in both the screening step and the verification step yield positive results, suggesting the presence of a BioWatch threat agent, then a BioWatch Actionable Result (BAR) is declared. From the program’s inception in 2003 through 2014, there were 149 BARs. None was found to be associated with the release of a biothreat agent, and these BARs have been termed false positives by Centers for Disease Control and Prevention (CDC) officials and others.

We found that all of the BARs from 2003 through 2014 were associated with PCR assays for two biothreat agents: Brucella and Francisella tularensis. The majority were associated with the assays for Francisella tularensis, and these have been attributed to detections of a non-disease- causing relative, or near-neighbor, of the Francisella tularensis bacterium that occurs naturally in the environment. Expert stakeholders told us that, before BioWatch, scientists had no reason or occasion to assess the presence or prevalence of these naturally occurring, non-disease-causing near-neighbors.

Department of Homeland Security (DHS) officials and other stakeholders said several adjustments were made to the Gen-2 system in 2011 and 2012 to reduce the number of false positives. In August of 2011, a stricter criterion for deciding that PCR assays revealed the presence of biothreat agents was adopted. Between November of 2011 and December of 2012, BioWatch adopted new PCR assays for the screening step of analysis. Previously, assays developed by CDC for use in its Laboratory Response Network (LRN) had been used for both screening and verification. The new assays were from the Department of Defense’s (DOD) Critical Reagents Program (CRP) and were designed to look for different genetic signatures of the BioWatch threat agents. In general, assays designed to detect greater numbers of unique genetic signatures will provide greater specificity—that is, greater ability to distinguish between the agents of interest and other, genetically similar agents. In December of 2012, BioWatch adopted new PCR assays specifically intended to distinguish between disease-causing and non-disease- causing species of Francisella; under the new analysis protocol, these new assays are run in the verification step if the screening step returned a positive result for Francisella.

Another adjustment to the system was not made in order to reduce the number of false positives but likely had this effect. In March, 2008, the PCR assays for Brucella were discontinued. According to BioWatch officials, this was because CDC had reclassified Brucella into a lower- threat category. Some of the BARs prior to March of 2008 were associated with the PCR assays for Brucella, and such BARs were no longer possible after this agent was discontinued as a BioWatch threat agent.

The annual number of BARs decreased during the years when the adjustments designed to reduce false positives were made, and after the final adjustment (the adoption of the new Francisella assays) there were no BARs through 2014 (see fig. 6). This decrease is consistent with the possibility that the adjustments have provided greater specificity, as intended; however, there was large, unexplained variability in the annual numbers of BARs in earlier years, and we did not conduct an independent analysis to assess the extent to which the decrease since 2010 might be associated with the specific adjustments DHS made to the system.

According to a recent report by the National Academies, there is no once- and-for-all solution to the problem of false positives for a biodetection system based on PCR assays. This is because biological agents continue to evolve, and new strains and near-neighbors continue to arise. Consequently, a biodetection system based on PCR assays, such as BioWatch, will likely require ongoing adjustments to manage or prevent false positives, and the National Academies recommended that the BioWatch program continue to test its assays against panels of near- neighbors as these panels are reviewed and updated over time.

Appendix III: Best Practices for Developmental Testing

In 2013, in collaboration with the National Academies, we identified eight best practices for developmental testing of threat detection systems. According to the experts who deliberated on the best practices, the practices apply to the process of developmental testing of binary threat detection systems; they also apply if the system is commercial-off-the- shelf (COTS), modified COTS, or newly developed for a specific threat detection purpose being created by a vendor or the government. For additional information on the methods used to determine the best practices, see appendix I.

The identified eight best practices for developmental testing of binary threat detection systems are described below, often using the context of the BioWatch program.

Practice 1: Ensure that accountability and engagement in developmental testing are commensurate with the amount of risk accepted.

According to experts, the level of government involvement in the development of a given system should be commensurate with the level of risk it is accepting. Risk needs to be assessed when the system is COTS, modified COTS, or a newly developed system, because even with commercial items, significant modifications may be needed. Experts also told us that relying solely on the vendor and holding the vendor responsible for any problem that arises is not consistent with the accountability and engagement required for acquisitions where the government is accepting significant risk. Further, an understanding of the technical risk associated with the development of a given system is important, or in the case of the purchase of a commercial item, understanding the modifications needed to an existing system to accommodate government-specific needs, is important.

Practice 2: Include representatives from the user community in design and developmental testing teams to ensure acceptance of the system by the user community.

Design and developmental testing teams need to understand the needs, concerns, and capabilities of the user community or they run the risk of designing and testing a system that, in the case of autonomous biological detection systems, (1) operators may have difficulty operating, or (2) decision makers may have difficulty with when interpreting results. According to experts, the user community may have suggestions that could improve the system or make the developmental tests more realistic. Distinct from subject matter experts that monitor developmental testing, these representatives are integral parts of the design and developmental testing teams. The role of these team members is to make sure that the needs, concerns, and capabilities of the user community are considered throughout design and developmental testing efforts.

Practice 3: Take a systems engineering view of the system prior to entering into any developmental test.

According to experts, to take a systems engineering view of a system, the tester must understand the boundaries of what it is being tested prior to developmental testing. For example, would DHS plan to test just the assays or analytical components of the Gen-3 candidate systems or would it plan to test the whole end-to-end system (i.e., collection, extraction, analysis, and communication of result), and was that plan communicated in the test and evaluation master plan (TEMP)? This is critical, since different system boundaries impose different testing methods and constraints.

Practice 4: Use statistical experimental design methodology to establish a solid foundation for developmental testing.

According to experts, use of statistical experimental design methodology ensures that a test has been designed with a clear understanding of goals and acceptable limitations, that the test is clearly documented, and that the test results are rigorously analyzed. Experts said statistical experimental design is the tool used to define the test goals, limitations, and procedures, and establishes a detailed plan for conducting the experiment. Further, experts stated that the creation and use of an appropriate model against which system performance can be evaluated is fundamentally important when establishing the statistical experimental design. According to experts, the system’s operational objectives and user’s needs should be identified before designing the experiment, and uncertainties should be characterized and reported with all system performance estimates. Experts told us that well-chosen statistical experimental designs maximize the amount of information that can be obtained for a given amount of experimental effort.

Practice 5: Measure and characterize system performance with established procedures, methods, and metrics.

According to experts, binary threat detection systems have an established body of statistically based methods and procedures used to evaluate and characterize them. Further, experts stated it is important to use certain objective metrics to characterize system performance.

Practice 6: Test to build in resilience, especially in the development stages.

According to experts, one way to improve resilience is to uncover vulnerabilities as early as possible through rigorous and comprehensive testing of the system against various scenarios. For example, the BioWatch system operates in a number of different environmental settings with varying contaminants. Settings vary from warm to cold climates, dry to wet climates, and indoor and outdoor settings. The BioWatch sensors might be exposed to dust, metallic dust, smoke, and diesel exhaust in indoor environments, as well as to rain, fog, snow, ice, wind, salt spray, sand, and pollen in outdoor environments. To the extent possible, these potential contaminants should be part of the testing of a detection system to help identify vulnerabilities to performance in these environments.

According to experts, the further the system moves down the development path, the more fixed the design becomes. Thus, when the developmental testing team uncovers an error (i.e., the system failed a test), it is increasingly expensive to fix. Any time there is a change in the design, everything that worked before needs to be re-tested to make sure the change did not undo something that already has been shown to work. Therefore, according to experts, agencies should focus on building in resilience during early and intermediate developmental testing so as to minimize the number of hidden failures found in the later stages of testing.

Practice 7: Use developmental tests to refine performance requirements.

According to experts, developmental testing should be viewed as a critical tool in helping to refine performance requirements. Experts told us that a meaningful performance requirement is one that not only is achievable but also strives to maximize the fulfillment of a mission need—which in the case of BioWatch might be the number of lives saved. While the minimum required performance thresholds may be achievable, they may fall short of the maximum achievable performance. Experts informed us that the maximum achievable performance can be uncovered only by understanding what the system is actually capable of doing through comprehensive developmental testing that unrestrainedly explores the performance boundaries of the system.

Practice 8: Engage in a continuous cycle of improvement by (1) conducting developmental testing, (2) conducting operational testing, and (3) incorporating lessons learned.

Experts told us that it is important to consider developmental testing and operational testing as a continuum by defining developmental testing broadly to cover, for example, operational test activities that traditionally have been viewed as post-development, rather than artificially limiting the development of a system to a fixed stage. Experts also said it is important to use lessons learned on preceding tests to improve the probability of success (proper system performance) on following tests and to use lessons learned from test failures as feedback into the design process to continuously improve system performance.

Appendix IV: Comments from the U.S. Department of Homeland Security

Appendix V: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to the individuals named above, Edward George (Assistant Director), Sushil Sharma (Assistant Director), Russ Burnett, Kendall Childers, Eric Hauswirth, Hayden Huang, Susanna Kuebler, Jack Melling, Jeff Mohr, Rebecca Shea, and Katherine Trimble made key contributions to this report.