Biosurveillance: DHS Should Reevaluate Mission Need and Alternatives before Proceeding with BioWatch Generation-3 Acquisition
Released: 2012-09-12T00:00:00

Letter

Background

BioWatch and the Biodefense Enterprise

Biological threats that could result in catastrophic consequences exist in many forms and arise from multiple sources. For example, several known biological agents could be made into aerosolized weapons and intentionally released in a transportation hub or other populated urban setting, introduced into the agricultural infrastructure and food supply, or used to contaminate the water supply. Concerned with the threat of bioterrorism, in 2004, the White House released HSPD-10, which outlines four pillars of the biodefense enterprise and discusses various federal efforts and responsibilities that help to support it. The biodefense enterprise is the whole combination of systems at every level of government and the private sector that can contribute to protecting the nation and its citizens from potentially catastrophic effects of a biological event. It is composed of a complex collection of federal, state, local, tribal, territorial, and private resources, programs, and initiatives, designed for different purposes and dedicated to mitigating various risks, both natural and intentional.

The four pillars of biodefense outlined in HSPD-10 are (1) threat awareness, (2) prevention and protection, (3) surveillance and detection, and (4) response and recovery. Protecting humans, animals, plants, air, soil, water, and critical infrastructure from potentially catastrophic effects of intentional or natural biological events entails numerous activities carried out within and between multiple federal agencies and their nonfederal partners. Figure 1 shows the four pillars of biodefense, examples of some federal efforts that can support them, and federal agencies responsible for those efforts. The BioWatch program falls under the surveillance and detection pillar. It is an example of an environmental monitoring activity.

DHS, in cooperation with other federal agencies, created the BioWatch program in 2003. The goal of BioWatch is to provide early warning, detection, or recognition of a biological attack. When DHS was established in 2002, a perceived urgency to deploy useful—even if imperfect—technologies in the face of potentially catastrophic consequences catalyzed the rapid deployment of many technologies, including the earlier generations of BioWatch collectors. In the initial deployment of BioWatch collectors—known as Generation 1—DHS deployed detectors to 20 major metropolitan areas, known as BioWatch DHS completed this jurisdictions, to monitor primarily outdoor spaces.initial deployment quickly—within 80 days of the President’s announcement of the BioWatch program during his 2003 State of the Union Address. To accomplish this quick deployment, DHS adapted an existing technology that was already used to accomplish other air monitoring missions. In 2005, DHS expanded BioWatch to an additional 10 jurisdictions, for a total of 30. This expanded deployment—referred to as Generation 2 (Gen-2)—also included the addition of indoor monitoring capabilities in three high-threat jurisdictions and provided additional capacity for events of national significance, such as major sporting events and political conventions.

Currently, the BioWatch program collaborates with 30 BioWatch jurisdictions throughout the nation to operate approximately 600 Gen-2 collectors. These detectors rely on a vacuum-based collection system that draws air samples through a filter. These filters must be manually collected and transported to state and local public health laboratories for analysis using a process called Polymerase Chain Reaction (PCR).

During this process, the sample is evaluated for the presence of genetic material from five different biological agents. If genetic material is detected, a BioWatch Actionable Result (BAR) is declared. Using this manual process, the determination of a BAR can occur from 12 to 36 hours after an agent is initially captured by the air filter. This 36-hour timeline consists of up to 24 hours for air sampling, up to 4 hours for sample recovery, and up to 8 hours for laboratory testing.

BioWatch in Action

Each BioWatch jurisdiction has either a BioWatch Advisory Committee or equivalent decision making group in place, composed of public health officials, first responders, and other relevant stakeholders. The BioWatch Advisory Committee is responsible for the day-to-day BioWatch operations, including routine filter collection and laboratory analysis of filter samples. In the event of a BAR, the BioWatch Advisory Committee is also responsible for determining whether that BAR poses a public health risk and deciding how to respond. The declaration of a BAR does not necessarily signal that a biological attack has occurred, as the Gen-2 detection process is highly sensitive and can detect minute amounts of pathogens that naturally occur in the environment. For example, at least two of the agents the program monitors occur naturally and have been detected in numerous areas of the United States. Since 2003, more than 100 BARs have been declared according to BioWatch program officials, but none were determined to be a potential risk to public health. Figure 2 shows the process that local BioWatch jurisdictions are to follow when deciding how to respond to a BAR.

To reduce the time required to detect biological pathogens, DHS has been pursuing an autonomous detection capability for the BioWatch program. Envisioned as a laboratory-in-a-box, the autonomous detection system that DHS seeks would automatically collect air samples, produce and read PCR results every 4 to 6 hours, and communicate the results to public health officials without manual intervention. By automating the analysis, DHS anticipates that detection time could be reduced to 6 hours or less, making the technology more appropriate for monitoring indoor high-throughput facilities such as transportation nodes. DHS also anticipates that operational costs will be reduced through the elimination of the daily manual collection and laboratory analysis process.

Developing autonomous detection has proved challenging according to BioWatch program officials, in part because some of the technology required was novel, but also because even the existing technologies—for example, the air collection system and the apparatus that reads the PCR results—had not been combined for this specific application in an operational environment before. As shown in figure 3, DHS began to develop autonomous detection technology in 2003. Initially, development of technologies to support autonomous detection was led by DHS’s Science and Technology Directorate (S&T), which partnered with industry. Since fiscal year 2007, DHS’s OHA has been responsible for overseeing the acquisition of this technology. In its 2011 report, the National Academies reported that the proposed enhancements to the BioWatch system will be possible only if significant scientific and technical hurdles are overcome. Similarly, as recently as March 2012, DHS’s Assistant Secretary for Health Affairs testified that the Gen-3 technology has been challenging to develop.

The overall policy and structure for acquisition management outlined in DHS’s AMD 102-01 includes the department’s Acquisition Life-cycle Framework—a template for planning and executing acquisitions. According to the directive, DHS adopted the Acquisition Life-cycle Framework to ensure consistent and efficient acquisition management, support, review, and approval throughout the department. As we have previously reported, without the development, review, and approval of key acquisition documents, agencies are at risk of having poorly defined requirements that can negatively affect program performance and contribute to increased costs.

As shown in figure 4, DHS’s Acquisition Life-cycle Framework includes four acquisition phases through which DHS determines whether it is sensible to proceed with a proposed acquisition: (1) identify a capability need; (2) analyze and select the optimal solution to meet that need; (3) obtain the solution; and (4) produce, deploy, and support the solution. During the first three phases, the DHS component pursuing the acquisition is required to produce key documents in order to justify, plan, and execute the acquisition. These phases each culminate in an Acquisition Decision Event (ADE), where the Acquisition Review Board— a cross-component board of senior DHS officials—determines whether a proposed acquisition has met the requirements of the relevant Acquisition Life-cycle Framework phase and is able to proceed. The Acquisition Review Board is chaired by the Acquisition Decision Authority—the official responsible for ensuring compliance with AMD 102-01. For the Gen-3 acquisition, DHS’s Deputy Secretary serves as the Acquisition Decision Authority.

DHS held an Acquisition Review Board related to ADE-2B on August 16, 2012, during which the BioWatch program was seeking approval to initiate the next phase of the acquisition.authority did not make a final ADE-2B decision, but did authorize the program to issue a solicitation for performance testing under the next testing phase. The Acquisition Decision Authority also required that the program office return to the Acquisition Review Board for approval prior to issuing a performance testing contract—which would allow the program to acquire a small number of test units. Furthermore, before undertaking the remaining steps in the acquisition, the program office must return to the Acquisition Review Board for ADE-2B with updated information, including an Analysis of Alternatives and Concept of Operations. DHS has not specified a time frame for completing these actions, but according to DHS officials, completing the Analysis of Alternatives may take up to 1 year.

DHS Did Not Fully Develop Critical Information for Initial Investment Decision Making

DHS approved the Gen-3 acquisition in October 2009, but it did not fully engage in the early phases of its acquisition framework to ensure that the acquisition was grounded in a justified mission need and that it pursued an optimal solution—for example, DHS did not fully develop a Mission Needs Statement or an Analysis of Alternatives with a cost-benefit analysis, as called for in its Acquisition Life-cycle Framework.

DHS Did Not Use Acquisition Life-cycle Framework Processes to Identify a Justified Mission Need

DHS skipped Phase 1 of the Acquisition Life-cycle Framework for the Gen-3 acquisition.Mission Needs Statement later to justify a predetermined solution. According to DHS’s acquisition lifecycle framework, in Phase 1, the program office is to develop a Mission Needs Statement to make a case to decision makers that the acquisition represents a justified need that Specifically, it did not hold ADE-1 and prepared a warrants the allocation of limited resources. At the end of Phase 1, the Acquisition Decision Authority is to review the Mission Needs Statement and other information during ADE-1 and decide whether the need is of sufficiently high priority to continue with the acquisition. However, according to BioWatch program officials, the Gen-3 acquisition began at ADE-2A, which is intended to be the decision gate at the end of Phase 2 of the Acquisition Lifecycle Framework. The Mission Needs Statement was finalized on October 6, 2009, just weeks before ADE-2A.

As shown in figure 5, DHS began to pursue a specific autonomous detection solution well before completing a Mission Needs Statement. Specifically, DHS’s Integrated Planning Guidance (IPG) for fiscal year 2010-2014, which was finalized in March 2008, included very specific goals for the next generation of BioWatch—to deploy in all major cities an autonomous BioWatch detection device reducing the operating cost per site by more than 50 percent and warning time to less than 6 hours. The purpose of DHS’s IPG is to communicate the Secretary’s policy and planning goals to component-level decision makers to inform their programming, budgeting, and execution activities. As such, this specific set of goals for BioWatch Gen-3 demonstrates that DHS leadership had established a course for the acquisition by March 2008, in advance of any efforts to define the mission need through the Mission Needs Statement process, which was finalized more than a year and a half later.

BioWatch program officials said they were directed by DHS to prepare the Mission Needs Statement—along with other required documentation for the ADE-1 and ADE-2A decision gates—on an accelerated time frame of about 6 weeks to prepare for the ADE-2A decision, efforts that they said would typically require at least 8 months. According to these officials, they were aware that the Mission Needs Statement prepared for ADE-2A did not reflect a systematic effort to justify a capability need. Although such an effort would provide a platform to help make trade-off decisions in terms of costs, risks, and benefits throughout the remainder of the acquisition process, officials said the time they were given would not have allowed for such an effort. They said that the department directed them to proceed because there was already departmental consensus around the solution.

Moreover, in its fiscal year 2009 budget request, submitted in February 2008, DHS requested funding to procure BioWatch automated detection sensors and initiate deployment activities of the automated sensor system. These funds, requested more than 18 months prior to the acquisition’s formal approval at ADE-2A—were intended to fund operational testing activities for Gen-3 BioWatch prototypes as well as the procurement of 150 automated detection sensors that DHS sought to deploy as an interim solution until the full Gen-3 acquisition could be completed. A prototype version of this interim solution was first fielded in 2007, shortly after DHS’s OHA assumed responsibility for the program.

Limited documentation is available to reflect the decision making process that occurred before the October 2009 Mission Needs Statement was finalized, including decisions related to the very specific IPG goals, the pursuit of funding for Gen-3, and the deployment of an interim solution before undertaking an effort to establish a justified mission need. We interviewed multiple officials in various DHS offices who had knowledge of Gen-3 in this early decision making period and the process that DHS used to justify the need to acquire Gen-3. However, none of these officials could describe what processes, if any, the department followed to determine that Gen-3 was a justified need. On the other hand, these officials all described a climate, in the wake of the September 11, 2001, terrorist attacks and the subsequent Amerithrax attacks, in which the highest levels of the administration expressed interest in quickly deploying the early generation BioWatch detectors and subsequently improving the functionality of these detectors—as quickly as possible—to allow for faster detection and an indoor capability. On the basis of this interest, officials from the multiple DHS offices said it was their understanding that the administration and departmental leadership had already determined that the existing BioWatch technology would need to be expanded and entirely replaced with an autonomous solution well before the acquisition was approved at ADE-2A.

DHS guidance states that the Mission Needs Statement should consider the IPG, but it also directs the program to focus on the capability need without specifying a specific technical solution. The Mission Needs Statement is designed to serve as the foundational document upon which subsequent Acquisition Life-cycle Framework efforts are based. As such, a Mission Needs Statement that focuses on the capability need can help articulate and build consensus around the goals and objectives for a program in a way that provides a touchstone throughout the rest of the acquisition processes as the program endeavors to identify optimal solutions and contends with technology, budget, schedule, and risk realities. The Gen-3 Mission Needs Statement prepared for ADE-2A, in response to the very specific solution set prescribed by DHS leadership in the IPG, asserted a specific technological solution—total replacement of the existing vacuum-based, manual technology with autonomous detectors—as the only viable solution. Because the Mission Needs Statement was completed after DHS had prescribed specific goals for Gen-3 in the IPG and requested funding to field an interim solution, it appears to be justification for a predetermined solution, rather than the deliberate and systematic consideration of capability needs that would serve as the foundation for the remaining acquisition processes. As such, its utility as a foundation for subsequent acquisition efforts—for example identifying an optimal solution and balancing mission requirements with budget, schedule, and risk considerations—was limited.

DHS Did Not Use Acquisition Life-cycle Processes to Analyze Alternatives Based on Cost, Schedule, and Risk to Identify an Optimal Solution

DHS did not use the processes in Phase 2 of the Acquisition Life-cycle Framework to systematically identify the optimal solution based on cost- benefit and risk information. We have long advised that DHS make risk- informed investments of its limited resources. For example, in February 2005, we reported that because the nation cannot afford to protect everything against all threats, choices must be made about protection priorities given the risk and how to best allocate available resources. More recently, we reported in September 2011 that because DHS does not have unlimited resources and cannot protect the nation from every conceivable threat, it must make risk-informed decisions regarding its homeland security approaches and strategies. Phase 2 of the DHS Acquisition Life-cycle Framework is intended to support these kinds of trade-off decisions by requiring DHS components to complete an Analysis of Alternatives that systematically identifies possible alternative solutions that could satisfy the identified need, considers cost-benefit and risk information for each alternative, and finally selects the best option from among the alternatives.

The Analysis of Alternatives is intended to provide assurance to DHS at the ADE-2A decision gate that the component has chosen the most cost- effective solution to mitigate the capability gap identified in the Mission Needs Statement. To provide this assurance and allow DHS to make trade-off decisions, the guidance states that developing the Analysis of Alternatives should be a systematic analytic and decision making process to identify and document an optimal solution that includes an understanding of the costs and benefits of at least three viable alternatives. The guidance directs the program to compare the alternatives based on cost, risk, and ability to respond to identified capability gaps. Finally, the guidance calls for an independent entity to complete this analysis to ensure that it is done objectively and without bias or vested interest in the study’s outcome.

The Gen-3 Analysis of Alternatives, completed in conjunction with the Mission Needs Statement by the BioWatch Program Office, does not reflect a systematic analytic and decision making process. Instead, the Analysis of Alternatives, like the Mission Needs Statement, was designed to support the decision the department had already made to pursue autonomous detection. The Analysis of Alternatives maintained that no modifications to the existing system would satisfy the goals in the IPG, and as such, it concluded that replacing the deployed technology entirely with autonomous detectors was the only viable solution. Along these lines, the Analysis of Alternatives included two alternatives: 1. expanding program coverage within current BioWatch cities and additional cities by replacing all currently deployed detectors with autonomous detection technology, and; 2. undertaking the same expansion using the currently deployed detectors but modifying the procedures to allow for filter collections every 8 hours instead of every 24 hours (a time frame that by definition would not meet the specific goals of the IPG).

The analysis did provide some cost information for each alternative, but it did not fully explore costs or consider benefit and risk information. As with the Mission Needs Statement, program officials told us that they were advised that a comprehensive Analysis of Alternatives would not be necessary because departmental consensus already existed that autonomous detection was the optimal solution.

In discussing the cost trade-offs between the autonomous detection solution and more frequent filter collection using the currently deployed technology in the Analysis of Alternatives, DHS focused on cost per detection cycle—that is, the cost each time an autonomous detector tests the air for pathogens or the cost each time a Gen-2 filter is manually collected and tested in a laboratory. According to our analysis of the June 2011 Life-Cycle Cost Estimate, cost per detection cycle is estimated to be seven times lower with the Gen-3 technology than the cost per detection cycle based on historical data for Gen-2 detectors. However, by only considering the cost per detection cycle of the two alternatives, the analysis does not help ensure the pursuit of an optimal solution based on cost, risk, and capability—as called for in the guidance.

To help ensure the optimal solution, DHS could benefit from a more complex and nuanced cost analysis that considers a number of factors in addition to the cost per detection cycle. For example, although not a cost- effectiveness analysis, table 1 shows that deploying and operating Gen-3 detectors is not necessarily more affordable than the existing Gen-2 deployment. According to our analysis, the total annual cost to operate Gen-3 is estimated to be about four times more than the cost of the existing Gen-2 deployment. The higher cost reflects both higher annual operating costs per detector and an increase in the number of detectors and jurisdictions covered.

In addition to a limited cost analysis, the Gen-3 Analysis of Alternatives contained no analysis of benefits. In fact, it did not identify any benefits of investment beyond the assumption—inherent in its focus on increasing the number of detection cycles per day—that earlier detection has the potential to save lives and limit economic loss, a basic and accepted principle for all enhanced surveillance efforts. In selecting the optimal solution, other costs and benefit factors like the examples shown in table 2 could have been helpful.

Identifying benefits and conducting a more complete analysis of cost and benefits would help DHS develop the kind of information that would inform tradeoff decisions and the selection of an optimal solution. For example, DHS plans to deploy about four times as many Gen-3 detectors than Gen-2 detectors, and each Gen-3 detector will test the air more frequently, so the Gen-3 deployment plan will increase the depth and range of the coverage to be provided. Specifically, Gen-3 is expected to cover 90 percent of the population in the jurisdictions where it is deployed, as opposed to 65 percent with the currently deployed technology. On the basis of the estimated annual program cost and the percentage of U.S. population covered, we calculated that the estimated Gen-3 deployment will cost about $4 million annually, on average, for each 1 percent of population covered within BioWatch jurisdictions (about $2.7 million more for each 1 percent covered than Gen-2). Moreover, according to Biowatch program officials, this kind of calculation may actually underestimate the cost of increasing population coverage by deploying more detectors, because the relationship between number of detectors and population covered is not linear. However, because the Gen-3 Analysis of Alternatives does not include a discussion of benefits, or a cost-benefit analysis, it does not consider the extent to which expanding the population covered under the proposed Gen-3 deployment would contribute to a reduction of risk and at what cost.

In its evaluation of BioWatch and public health surveillance, the National Academies stated that the BioWatch program should not expand its coverage of biological agents or jurisdictions without a clear understanding of the change’s contribution to reducing mortality or morbidity in conjunction with clinical case finding and public health. However, without a discussion of benefits, the Analysis of Alternatives could not help DHS develop this understanding before approving the acquisition at ADE-2A. In June 2011, the program office commissioned a study from Sandia National Laboratories that began to develop a basis for this kind of understanding of benefits related to public health outcomes. However, critical information related to both the costs and the benefits of the planned Gen-3 approach remains to be explored. DHS has commissioned the Homeland Security Institute (HSI) to conduct an independent study that has as its overarching objective the characterization of the state of Gen-3 technology—that is, whether it is mature enough to continue as an acquisition or whether it needs additional development work. As part of that study, DHS has asked HSI to consider, among other things, (1) whether the threat is adequately described, (2) whether it is possible to determine costs and benefits, and (3) to what extent prior studies like the Sandia study have been validated and used to inform plans for Gen-3 deployment. DHS told us that the study would be completed by September 1, 2012. As of early-September 2012, DHS has not provided us with a copy of the study or responded to requests to provide an updated timeline for the study.

Beyond the uncertainty related to the costs and benefits of the planned Gen-3 approach, there is additional uncertainty about the benefit of this kind of environmental monitoring because as a risk mitigation activity, it has a relatively limited scope. As the study committee for the National Academies evaluation of BioWatch noted, there is considerable uncertainty about the likelihood and magnitude of a biological attack, and how the risk of a release of an aerosolized pathogen compares with risks from other potential forms of terrorism or from natural diseases. The report also notes that while the BioWatch program is designed to detect certain biological agents (currently five agents) that could be intentionally released in aerosolized form, detecting a bioterrorism event involving other pathogens or routes of exposure requires other approaches. Given the higher total estimated operating cost for the Gen-3 program, it is important, especially in an increasingly resource-constrained environment, to also consider the benefit—in terms of its ability to mitigate the consequences of a potentially catastrophic biological attack—that the extra investment provides. These scope limitations provide context in both the consideration of mission need and in analyzing cost effectiveness.

Because the Gen-3 Analysis of Alternatives focuses on justifying total replacement of Gen-2 technology with an autonomous detection technology, it did not explore whether another solution might be more effective. For example, according to BioWatch program officials, it is possible that other options—including but not limited to deployment of some combination of both technologies, based on risk and logistical considerations—may be more cost-effective. Along these lines, program officials told us that in 2011, to help them manage various budget contingencies, they prepared a summary of available deployment options for Gen-3 that includes a mixed deployment of Gen-3 and Gen-2 units. However, a more comprehensive solution set was not available to be considered at ADE-2A; nor has such an effort since been undertaken to inform investment and trade-off decisions at the departmental level. Given the uncertainty related to the costs, benefits, and risk mitigation potential of Gen-3, DHS does not have reasonable assurance that the strategy of expanding and completely replacing the existing Gen-2 technology with an autonomous detection technology provides the most cost-effective solution.

DHS Did Not Develop Complete and Reliable Performance, Schedule, and Cost Information before Approving the Gen-3 Acquisition

In October 2009, DHS approved the Gen-3 acquisition at ADE-2A— based on the information contained in acquisition documents provided by the BioWatch program—authorizing the BioWatch program to proceed with characterization testing of Gen-3 candidate technologies. One critical purpose of the ADE-2A documentation set required by DHS’s acquisition guidance is to describe the expected performance, cost, and schedule parameters for an acquisition. We reported in June 2010 that stable parameters for performance, cost, and schedule are among the factors that are important for successfully delivering capabilities within cost and schedule expectations. We also reported in May 2012 that without the development, review, and approval of key acquisition documents, agencies are at risk of having poorly defined requirements that can negatively affect program performance and contribute to increased costs. However, the ADE-2A Acquisition Decision Memorandum stated that significant data necessary for the proper adjudication of an ADE-2A decision were missing. Specifically, it noted that the ADE-2A documentation set did not contain three required documents, including: (1) a Concept of Operations—intended to provide critical information on how an acquisition will function in the operational environment, (2) an Integrated Logistics Support Plan—intended to document how an acquisition will be supported and sustained through its life-cycle, and (3) a Life-Cycle Cost Estimate—intended to provide a credible estimate of the life-cycle cost of the acquisition. Additionally, we found that certain information contained in the ADE-2A documentation set on operational requirements, schedule projections, and cost were not developed using reliable methods as discussed later in this report. For more information on the limitations of Gen-3 acquisition documents and processes at ADE-2A, see appendix I.

As was the case for the Mission Needs Statement and the Analysis of Alternatives, BioWatch program officials stated that they had to prepare ADE-2A documentation quickly because ADE-2A had been accelerated by 14 months. However, in the absence of complete and reliable information, DHS had limited assurance that the acquisition would successfully deliver the intended capability within cost and on schedule. Nevertheless, the Deputy Secretary approved the acquisition, but she required the program office to provide quarterly progress updates.

On the basis of the Gen-3 documentation submitted at ADE-2A, DHS expected to acquire a system that would cost $2.1 billion, be fully deployed by fiscal year 2016, and meet certain performance requirements. As shown in table 3, as of July 2012, the performance, schedule, and cost parameters for the Gen-3 acquisition are significantly different from the parameters DHS expected when it approved the acquisition at ADE-2A.

Regarding performance expectations, the BioWatch program submitted a revised Operational Requirements Document to DHS for approval that includes a proposed revision to the key performance parameter for system sensitivity— the amount of a pathogen that would have to be present in the air for the system to detect its presence. DHS acquisitions guidance requires components to develop key performance requirements that an acquisition must meet in order to fulfill the program’s fundamental purpose and close the capability gap(s) identified in the Mission Needs Statement, and to document these requirements in an Operational Requirements Document. However, BioWatch program officials told us that the original sensitivity requirement was based on what DHS thought the technology could theoretically achieve, and was not informed by a scientific and risk-informed assessment of what level of sensitivity would be needed—from an operational perspective—to fulfill the Gen-3 purpose of mitigating consequences in the event of a biological attack. Additionally, the process used to set the sensitivity requirement did not reflect stakeholder consensus about how to balance mission needs with technological capabilities. Specifically, the BioWatch program did not prepare a Concept of Operations before ADE-2A. According to DHS acquisitions guidance, in developing a Concept of Operations, stakeholders engage in a consensus-building process regarding how to balance technological capabilities with mission needs in order to gain consensus on the use, capabilities, and benefits of a system. Because DHS did not prepare a Concept of Operations before establishing operational requirements, the sensitivity requirement did not reflect broad stakeholder engagement in balancing schedule, cost, and risk realities with achieving a specified mission outcome—for example, a specific level of population protection.

During characterization testing, the candidate technology tested was unable to meet the original sensitivity requirement. According to the September 2011 Operational Assessment, the system sensitivity demonstrated during characterization testing was orders of magnitude lower than the original requirement, meaning that a significantly greater concentration of a pathogen than specified in the requirement would have to be present in the air to trigger detection. According to BioWatch program officials, the original sensitivity requirement was set based on interest in pushing the limits of potential technological achievement rather than in response to a desired public health protection outcome. They said that this led to a requirement that may have been too stringent, resulting in higher costs and schedule delays without demonstrated mission imperative. Because DHS did not ground the sensitivity requirement in Gen-3 program goals, when the candidate technologies were unable to meet the requirement, DHS encountered delays and uncertainty about how to move forward. In response to these concerns, the BioWatch program directed Sandia National Laboratories to evaluate the level of system sensitivity that would be necessary for the Gen-3 program to fulfill its fundamental purpose. The study, which was completed in January 2012, contained findings that, according to BioWatch Program officials, confirm that the sensitivity requirement could be relaxed without significantly affecting the program’s public health mission. In response to this study, the BioWatch program submitted an updated Operational Requirements Document with a revised sensitivity requirement to DHS in March 2012 for approval in preparation for ADE-2B, as shown in figure 6.

The need to reevaluate the sensitivity requirement for the Gen-3 acquisition has contributed to delays in the acquisition schedule. For example, in August 2011, the BioWatch program requested to postpone the ADE-2B, scheduled for September 2011, until December 2011 to give the program time to address the testing issues associated with the sensitivity requirement. Given that the Sandia study was not available until January 2012, the program office again requested that ADE-2B be delayed until March or April 2012. As of September 2012, DHS has not approved the revised sensitivity requirement and plans to revisit that decision at the next Acquisition Review Board for ADE-2B. DHS acquisition guidance states that the accurate definition of requirements is imperative if an acquisition is to be completed within schedule constraints and still meet the component and department’s mission performance needs. It follows that these schedule delays could have been mitigated if the original sensitivity requirement had been more realistically set using scientific and risk information to ensure that it aligned with the mission need of the program and balanced mission goals with technological feasibility.

In addition to the impact that changing the sensitivity requirement had on the acquisition schedule, the change in schedule expectations since October 2009 can also be explained by DHS not employing reliable schedule estimation methods to produce the schedule estimate that was submitted with the Acquisition Program Baseline in the ADE-2A documentation set. Our prior work has found that realistic acquisition program baselines with stable requirements for cost and schedule are among the factors that are important to acquisitions successfully delivering capabilities within cost and schedule constraints. However, BioWatch program officials told us that they set the ADE-2A schedule estimate aggressively because there was pressure to respond quickly to the call to deploy autonomous detection. Additionally, they reported that they did not account for risk in the schedule estimates that were included in the Acquisition Program Baseline for ADE-2A. The BioWatch program office has revised the acquisition schedule since ADE-2A was held in 2009. The most recent update—completed in January 2012—estimated full deployment of the Gen-3 system in fiscal year 2022, 6 years later than anticipated. While the acquisition is currently on track with the January 2012 schedule, the schedule remains subject to uncertainty, in part because of a pending decision about the acquisition strategy.

GAO-09-3SP. to engage in a full effort to develop a Life-Cycle Cost Estimate in accordance with the GAO Cost Estimating Guide ahead of ADE-2A, but were directed by the department to proceed with the best point estimate they could derive. Additionally, both BioWatch program and PARM officials described a climate before ADE-2A in which the department’s business processes—including acquisition practices—were maturing and thus were less rigorous in their adherence to best practices for cost and schedule estimating.

The BioWatch program has revised the cost estimate using more reliable methods since the ADE-2A estimate was prepared in 2009. The most recent update—completed in June 2011—shows the estimated life-cycle cost for the Gen-3 acquisition to be $5.8 billion (80 percent confidence), much higher than the $2.1 billion point estimate presented at ADE-2A. The 2011 Life-Cycle Cost Estimate was aligned with GAO’s Cost Estimating Guide, which recommends that agencies calculate a range of possible cost estimates based on different risk levels in order to account for uncertainty. According to the guide, experts agree that program cost estimates should be budgeted to at least the 50 percent confidence level, but budgeting to a higher level (for example, 70 percent to 80 percent) is now a common practice. Moreover, a higher confidence level in cost estimating may be more prudent, as experts stress that contingency reserves are necessary to cover increased costs resulting from unexpected design complexity, incomplete requirements, technology uncertainty, and other uncertainties that can affect programs, according to the GAO Cost Estimating Guide. Acknowledging the benefit of a higher confidence level for cost estimates, the BioWatch program recommended that the 80 percent confidence level estimate be used for planning purposes. As such, the $5.8 billion figure presented in the 2011 cost estimate was calculated at the 80 percent confidence level—meaning that there is an 80 percent chance that the actual life-cycle cost will be this amount or less, according to BioWatch officials.

BioWatch program officials told us that the large difference between the ADE-2A cost estimate and the June 2011 cost estimate is primarily driven by the inclusion of risk in the June 2011 estimate, rather than by changes to the program. However, these officials also noted other factors that contributed to the difference. For example, the 2009 estimate was not as robust as the 2011 estimate because it was not based on the work breakdown structure for the program. Additionally, because of changes in the schedule estimates, the June 2011 estimate considers costs through fiscal year 2028, whereas the 2009 estimate considered costs through fiscal year 2020. These changes in performance, schedule, and cost, along with maturation in the department’s acquisition management process, create an opportunity for DHS to reevaluate the mission need and alternatives in a more comprehensive and systematic fashion, and in accordance with DHS acquisitions guidance, to help ensure that it invests its limited resources in the most cost-effective solution possible. In addition, using comprehensive and systematically developed information, in conjunction with good practices for cost and schedule estimating like those described in the GAO Cost Estimating Guide, could help ensure that the department and policymakers have the most reliable performance, schedule, and cost information available for decision making.

BioWatch Must Demonstrate System Performance and Receive Approval before Full Deployment, Estimated for 2022

According to DHS officials the remaining steps in the Gen-3 acquisition include performance testing, operational testing and evaluation, production, deployment, and sustainment. Figure 7 shows the timeline, based on the January 2012 Acquisition Program Baseline and discussions with BioWatch program officials, for the remaining steps to deploy and operate Gen-3.

First, DHS plans to issue a solicitation for performance testing in the next testing phase, but the Acquisition Review Board must provide approval before the program awards a contract. In addition, final ADE-2B approval will be required for the remaining acquisition steps, including operational testing and evaluation. In preparation for ADE-2B, the BioWatch program has updated key acquisition documents—including the Life- Cycle Cost Estimate and Acquisition Program Baseline— as required by the Acquisition Decision Authority in a February 2012 memo. In order to inform the ADE-2B decision, these documents must accurately reflect changes to Gen-3 performance requirements and updated cost and schedule estimates for the acquisition and therefore may require further revisions.

For more information on characterization test events and results, see appendix II. environment remain unverified, and failure to demonstrate this capability may seriously inhibit user confidence in the system.

Results from operational test and evaluation will be used to inform ADE-3, which, if approved would authorize full-rate production and deployment of Gen-3. DHS’s Acquisition Life-cycle Framework requires that the BioWatch program provide proof that the technology satisfies the operational requirements. To ensure that the full system satisfies the operational requirements, the BioWatch program intends to design a testing plan that demonstrates that the full system—including the information technology network when it is developed—can operate as intended, while complying with legal restrictions on testing for pathogens in BioWatch jurisdictions. DHS has not yet finalized a testing strategy, and the final test plan will depend on the candidate technologies chosen for testing following ADE-2B. Whatever the strategy, DHS officials from the BioWatch program and the Science and Technology Directorate office that oversees testing said that operational test and evaluation will include a number of subsystem and full system test events from which performance in an operational environment can be modeled and extrapolated. Table 4 provides examples of possible test events to demonstrate Gen-3 performance.

Collectively, the BioWatch program estimates that this testing will take approximately 3 years and cost approximately $89 million. During operational testing and evaluation, the BioWatch program must prepare for and mitigate several limitations. These limitations include the following: Inability to fully test Gen-3’s detection capability: BioWatch officials told us that legal restrictions on the aerosolized release of all five BioWatch agents in U.S. cities limit the BioWatch program’s ability to demonstrate full and subsystem performance in an operational environment. Without releasing the agents in BioWatch jurisdictions, the BioWatch program is unable to test the system’s ability to detect them in the operational environment. According to BioWatch program officials and DHS S&T officials who assist with test design, designing laboratory and field tests that can compensate for these limitations on pathogen use is a goal that is guiding the development of the testing plan for Operational Testing and Evaluation.

Inconsistent performance in different operational environments: The candidate system tested during the characterization field test performed better at some sites than others. Specifically, detectors located on underground subway platforms had higher incidences of malfunction than detectors in other locations. These malfunctions may be associated with the presence of metallic brake dust; however, this failure demonstrates different operational environments pose different challenges, and the BioWatch program plans to conduct laboratory testing as well as modeling to further assess detector performance under different operational conditions.

Difficulty verifying false positive rate: In order to build user confidence in the system, the BioWatch program has established a stringent threshold of 1 in 10 million for the false positive rate—that is, the rate at which the system is allowed to indicate a pathogen is present when one is not. However, according to BioWatch documentation, 33.5 years of operational testing would be required to fully demonstrate that the system meets the established false positive rate. Therefore, the BioWatch program plans to use data from laboratory testing to model and extrapolate the probability of a false positive. According to program documentation, the amount of time planned for operational testing will be sufficient to reveal any issues with false positive performance of the candidate technologies tested.

The goal of the next phase of testing is to demonstrate that Gen-3 candidate technologies can operate as intended in the operational environment. To achieve that goal, which is required for ADE-3, the BioWatch program must successfully mitigate these testing limitations. For example, to address the inconsistent performance in testing environments, the program must determine whether and how to adjust laboratory conditions to better reflect the operational environment by exposing the detectors to contaminants such as dust and pollen.

To be ready to produce and deploy Gen-3, DHS must demonstrate technological readiness for the full system based on both individual component readiness and the maturity of the integration of those components. In August 2011, on the basis of results of characterization testing, the Institute for Defense Analysis conducted a Technology Readiness Assessment—a formal independent review that assesses the maturity of critical hardware and software technologies to be used in systems—for Gen-3. Using the Department of Defense’s (DOD) Technology Readiness Level (TRL) scale, which defines levels of technological maturity on a scale of 1 to 9, the assessment assigned TRL scores to the Gen-3 candidate technology’s individual critical technology elements, that is, those subsystems that are vital to the functioning of the system and are either new or novel applications or pose major technical risk. This assessment rated all but one of the critical technology elements it assessed as TRL 7—indicating a relatively high level of maturity for each technology element assessed. However, the assessment does not provide an overall TRL for the full Gen-3 system. It notes that doing so could obscure the strengths and weaknesses of individual system components, and says that the DOD’s Technology Readiness Assessment Deskbook, which provides guidance for assigning TRLs, does not describe how to aggregate TRLs. However, other DOD guidance specific to chemical and biological defense says that a TRL evaluation is generally undertaken to establish a system’s level of maturity relative to a specific purpose, which suggests that the next phase of testing should result in a technology readiness assessment that provides an indication of how well these components perform together in order to meet the mission need of autonomous detection. Furthermore, we have previously reported that underestimating the complexity of systems integration can be a cause of significant cost and schedule growth. DHS also has not assessed the technology readiness of the data network, a major component of the Gen-3 system, or its integration into the system because it has not yet been developed. The data network and its integration will therefore require demonstration prior to production and deployment of Gen-3.

If the BioWatch program can demonstrate that the candidate technology meets requirements and DHS approves the Gen-3 acquisition at ADE-3, the DHS June 2011 life-cycle cost estimate indicates that Gen-3 is expected to cost $5.8 billion (80 percent confidence) through June 2028. As shown in figure 8, approximately $5.7 billion of this total has not yet been spent and is expected to primarily fund operations once the system is deployed.

To prepare for the deployment of Gen-3, the BioWatch program must work with Gen-3 jurisdictions to prepare sites for detector placement and to develop location-specific Concepts of Operations to provide key information and considerations—such as specifying roles and responsibilities and developing public information and risk communication messages—that are integral to response operations in the event that Gen-3 detects a pathogen. Like the Gen-2 system, the Gen-3 system is to be operated by BioWatch jurisdictions, and therefore the system’s usefulness in improving response time is expected to be determined, in part, by each jurisdiction’s willingness to respond to a positive test result, which, if incorrect could have large monetary costs and public and political repercussions. According to BioWatch program officials, they want the jurisdictions to have enough confidence in the system that they are willing to take action based on positive results from a Gen-3 detector without confirmatory laboratory testing. Therefore, according to BioWatch program officials, they have taken steps to increase jurisdictions’ confidence in the Gen-3 system. For example, they provide guidance to jurisdictions and are in the process of developing a quality assurance process to track system performance. Furthermore, these officials anticipate running Gen-2 and Gen-3 concurrently for up to 6 months in BioWatch jurisdictions, and requiring all candidate technologies to archive positive samples so that the jurisdictions can run confirmatory laboratory analysis on the samples.

Conclusions

Despite Gen-3’s potential to save lives under specific conditions, uncertainty remains about its general risk mitigation value. DHS established the strategy to quadruple the number of deployed detectors and replace all Gen-2 technology with an autonomous solution while expanding to 20 additional cities without engaging in a robust mission needs effort to serve as a foundation for subsequent acquisition efforts. As we have previously reported, because DHS does not have unlimited resources and cannot protect the nation from every conceivable threat, it must make risk-informed decisions regarding its homeland security approaches and strategies. In addition, we have previously reported that programs that conduct a limited assessment of alternatives before the start of system development tend to experience poorer outcomes than programs that conduct more robust analyses. Without a justified mission need to ground acquisition decision making or a systematic analysis of the cost-benefits and risk, DHS has pursued goals (such as the time threshold of 6 hours) and specific technological requirements (such as the sensitivity threshold) that may or may not support optimal solutions. Reevaluating the mission need and systematically analyzing alternatives based on cost-benefit and risk information could help DHS gain assurance that it is pursuing an optimal solution.

Furthermore, difficulty attaining the original goals has contributed to challenges in meeting milestones and deadlines for deployment. In 2009, when the Acquisition Decision Authority approved the Gen-3 acquisition, it was anticipated that Gen-3 technologies would be in initial deployment by 2013 and fully deployed by the first quarter of 2016. In 2011, DHS’s most recent estimate, which contains significant uncertainty because of testing limitations, among other reasons, was that full deployment would be 6 years later, in 2022. Similarly, the $2.1 billion cost estimate presented to DHS decision makers and Congress for planning purposes at the start of the acquisition is now $5.8 billion (for the first 13 years of deployment; only 6 of which are for full deployment) and may still rise because of lingering uncertainty about the acquisition strategy. These changes in cost, schedule, and performance, along with maturation in the department’s business processes—including acquisitions and risk management—reinforce the importance and provide an opportunity for DHS to reevaluate the mission need and alternatives in a more robust, considered, and systemic fashion, as called for in the Acquisition Life- cycle Framework, to help ensure that it makes the most sound investments possible. In addition, comprehensive and systematically developed information, developed using good practices for cost and schedule estimating like those described in the GAO Cost Estimating Guide, could help ensure that the department and policymakers have the most reliable performance, schedule, and cost information available for decision making.

Recommendations for Executive Action

To help ensure that Gen-3’s public health and risk mitigation benefits justify the costs, the program pursues an optimal solution, and DHS bases its acquisition decisions on reliable performance, cost, and schedule information developed in accordance with guidance and good practices, we recommend that before continuing the Gen-3 acquisition, the Secretary of Homeland Security ensure that program and acquisition decision makers take the following two actions: 1. reevaluate the mission need and systematically analyze alternatives based on cost-benefit and risk information, using information from studies like those conducted by the Homeland Security Institute and Sandia National Laboratories, along with any other risk and cost information that may need to be developed, and 2. update other acquisition documents, such as the Acquisition Program Baseline and the Operational Requirements Document, to reflect any changes to performance, cost, and schedule information that result from the reevaluation of mission needs and alternatives.

Agency Comments and Our Evaluation

We provided a draft of this report to DHS for comment, and DHS provided written comments on the draft report, which are reproduced in full in appendix III. DHS also provided technical comments, which we incorporated as appropriate. DHS concurred with both recommendations, but did not concur that these actions need to be completed before continuing with the acquisition.

With respect to the first recommendation to reevaluate the mission need and alternatives, DHS agreed that further evaluation of the mission need and alternatives is necessary. DHS stated that, on August 16, 2012, it directed the BioWatch program to complete an updated Analysis of Alternatives and Concept of Operations, which, according to a DHS official must be completed before ADE-2B, but DHS did not specify how it plans to reevaluate the mission need. With respect to the second recommendation to update other acquisition documents to reflect any performance, cost, and schedule information that might result from reevaluation, DHS acknowledged that it may be necessary and appropriate to do so. However, DHS did not agree that it should implement these two recommendations before continuing the acquisition.

In its response, DHS stated its intent to issue a solicitation for performance testing concurrent with the efforts to implement the recommendations. DHS stated that BioWatch will be required to return to the Acquisition Review Board prior to issuing a contract stemming from this solicitation.

We are pleased that DHS plans to reevaluate the mission need and alternatives and that the department believes this action would be beneficial as it seeks to reduce programmatic risk and demonstrate sound fiscal stewardship in an increasingly constrained fiscal environment. Additionally, we commend DHS’s stated commitment to use Acquisition Management Directive 102-01 to ensure consistent and efficient acquisition management, support, review, and approval. The directive’s acquisition life-cycle framework is designed to establish a foundation based on critical examination of the capability gap an acquisition would fill and to build sequentially on that foundation to support solid, knowledge- based acquisition decision making. To satisfy the larger purpose of the framework—providing assurance that DHS makes judicious decisions about how to invest limited resources and implements them effectively—it is vital that it be used consistently, that each acquisition adheres to the framework throughout its entire life-cycle, and that specified steps are completed in a sequential manner to support key acquisition decisions.

Accordingly, we are concerned by DHS’s intention to continue the acquisition efforts before ensuring that it has fully developed the critical knowledge a comprehensive acquisition life-cycle framework effort is designed to provide. Our work showed that DHS does not have reasonable assurance that the solution it has been pursuing warrants investment of limited resources and that it represents an optimal solution. We believe it is possible that an earnest effort to reconsider the Gen-3 mission needs and alternatives would result in a different plan and course of action than the current effort.

DHS stated in its response that it has directed the BioWatch program to complete an updated Analysis of Alternatives, but it remained silent on what actions, if any, it will take to reevaluate the mission need. As such, it is not clear from DHS’s response to what extent it intends to engage in a fresh reevaluation of the mission need in the broader context of DHS’s biodefense and related responsibilities before it undertakes efforts to update its Analysis of Alternatives. During discussions with program officials about the recommendation to reevaluate the mission need, the officials told us that they had resubmitted the original Mission Needs Statement to DHS for review. If DHS were to approve the original Mission Needs Statement and use it to guide the reevaluation of alternatives, it would overlook the intent of the recommendation. The intent is that DHS reevaluate existing capability gaps through the mission needs process to provide a foundation for future acquisition decision making—including the Analysis of Alternatives—that is grounded in better understanding and consensus about how filling these gaps will contribute to larger biodefense needs. Moreover, DHS’s plans to pursue testing of the Gen-3 solution—a solution which has driven DHS’s efforts for a number of years, including prior efforts to define mission need and analyze alternatives— even while agreeing to reconsider whether it is an appropriate course of action. This plan raises questions about whether the department plans to systematically and objectively reevaluate the mission need and alternatives for fulfilling that need.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 3 days from the report date. At that time, we will send copies to the Secretary of Homeland Security and interested congressional committees. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-8757 or jenkinswo@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Limitations of Gen-3 Acquisition Documents and Processes at ADE-2A

Department of Homeland Security (DHS) Acquisition Management Directive (AMD) 102-01—intended to ensure consistent and efficient acquisition management, support, review, and approval throughout the department—outlines the overall policy and structure for acquisition management at DHS. Specifically, this directive includes document and process requirements for each of the four phases of the department’s Acquisition Life-cycle Framework through which DHS determines whether it is sensible to proceed with a proposed acquisition. DHS formally approved the Gen-3 acquisition at Acquisition Decision Event (ADE) 2A in October 2009 without fully using and completing the processes and analyses in Phases 1 and 2 of the Acquisition Life-cycle Framework: (1) identify a capability need and (2) analyze and select the means to provide that capability. However, as shown in table 5, the documentation set DHS had available to inform the ADE-2A decision was incomplete and certain required information was not developed using the most reliable methods available.

Appendix II: BioWatch Gen-3 Characterization Test Events and Candidate Technology Performance against Key Performance Parameters

From May 2010 to June 2011, the BioWatch program completed a series of characterization tests on a Gen-3 candidate technology. The goals of this testing included characterizing the state of the market and evaluating the candidate systems’ abilities to meet performance requirements developed by the BioWatch program. As described in table 6, DHS completed four independent laboratory tests and a field test as part of characterization testing.

Based on the results of the test events described above, an independent assessment was completed to evaluate the performance of the candidate Gen-3 system tested against the requirements developed by the BioWatch program. These requirements were listed in the Gen-3 operational requirements document, approved at ADE-2A in 2009, and included five key performance parameters (KPP)—the most important and non-negotiable requirements that must be met in order for the program to fulfill its purpose. As shown in table 7, the summary report found that of the five KPPs, the candidate system that completed testing met or partially met three, did not meet one, and that performance against the final KPP remained unresolved.

Appendix III: Comments from the Department of Homeland Security

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Edward George, Assistant Director; Kathryn Godfrey; Allyson Goldstein; and Katy Trenholme made significant contributions to the work. Harold Brumm, Nirmal Chaudhary, Michelle Cooper, Marcia Crosse, Katherine Davis, Amanda Gill, Eric Hauswirth, Tracey King, Susanna Kuebler, David Lysy, Amanda Miller, Jan Montgomery, Jessica Orr, Katherine Trimble, and Teresa Tucker also provided support.