Biosurveillance: Observations on the Cancellation of BioWatch Gen-3 and Future Considerations for the Program
Released: 2014-06-10T00:00:00

Letter

Background

DHS Acquisitions and the Cancellation of Gen-3

We have highlighted DHS acquisition management issues in our high-risk list since 2005. Over the past several years, our work has identified significant shortcomings in the department’s ability to manage an expanding portfolio of major acquisitions. We have also reported that while DHS acquisition policy reflects many key program management practices intended to mitigate the risks of cost growth and schedule slips, the department did not implement the policy consistently. In 2011, expressing concerns about whether DHS had undertaken a rigorous effort to help guide its Gen-3 decision making, members of Congress asked us to examine issues related to the Gen-3 acquisition. We released a report that evaluated the acquisition decision-making process for Gen-3 in September 2012. As discussed later in the statement, we recommended that before continuing the Gen-3 acquisition, DHS should carry out key acquisition steps, including reevaluating the mission need and systematically analyzing alternatives based on cost-benefit and risk information.

On April 24, 2014, DHS issued an Acquisition Decision Memo (ADM) announcing the cancellation of the acquisition of Gen-3. The ADM also announced that S&T will explore development and maturation of an effective and affordable automated aerosol biodetection capability, or other operational enhancements, that meet the operational requirements of the BioWatch system. DHS’s S&T conducts research, development, testing, and evaluation of new technologies that are intended to strengthen the United States’ ability to prevent and respond to nuclear, biological, explosive, and other types of attacks within the United States. S&T has six technical divisions responsible for managing S&T’s research R&D portfolio and coordinating with other DHS components to identify R&D priorities and needs.applied research and development projects for its DHS customers.

BioWatch in Action

The BioWatch program collaborates with 30 BioWatch jurisdictions throughout the nation to operate approximately 600 Gen-2 collectors. These detectors rely on a vacuum-based collection system that draws air samples through a filter. These filters must be manually collected and transported to state and local public health laboratories for analysis using a process called Polymerase Chain Reaction (PCR). During this process, the sample is evaluated for the presence of genetic material from five different biological agents. If genetic material is detected, a BioWatch Actionable Result (BAR) is declared. Figure 1 shows the process that local BioWatch jurisdictions are to follow when deciding how to respond to a BAR.

Our Prior Work on the Gen-3 Acquisition Identified Challenges and DHS Has Since Cancelled the Program

Our prior findings and recommendations related to the Gen-3 acquisition provide DHS with lessons learned for future decision making. In September 2012, we found that DHS approved the Gen-3 acquisition in October 2009 without fully developing critical knowledge that would help ensure sound investment decision making, pursuit of optimal solutions, and reliable performance, cost, and schedule information. Specifically, we found that DHS did not engage the initial phase of its Acquisition Life- cycle Framework, which is designed to help ensure that the mission need driving the acquisition warrants investment of limited resources. BioWatch officials stated that they were aware that the Mission Needs Statement prepared in October 2009 did not reflect a systematic effort to justify a capability need, but stated that the department directed them to proceed because there was already departmental consensus around the solution. Accordingly, we concluded that the utility of the Mission Needs Statement as a foundation for subsequent acquisition efforts was limited.

Additionally, in September 2012, we found that DHS did not use the processes established by its Acquisition Life-cycle Framework to systematically ensure that it was pursuing the optimal solution—based on cost, benefit, and risk—to mitigate the capability gap identified in the Mission Needs Statement. The DHS Acquisition Life-cycle Framework calls for the program office to develop an analysis of alternatives (AoA) that systematically identifies possible alternative solutions that could satisfy the identified need, considers cost-benefit and risk information for each alternative, and finally selects the best option from among the alternatives. However, we found that the AoA prepared for the Gen-3 acquisition did not reflect a systematic decision-making process. For example, in addition to—or perhaps reflecting—its origin in a predetermined solution from the Mission Needs Statement, the AoA did not fully explore costs or consider benefits and risk information as part of the analysis. Instead, the AoA focused on just one cost metric that justified the decision to pursue autonomous detection—cost per detection cycle—to the exclusion of other cost and benefit considerations that might have further informed decision makers. Additionally, we found that the AoA examined only two alternatives, though the guidance calls for at least three. The first alternative was the currently deployed Gen-2 technology with a modified operational model (which by definition was unable to meet the established goals). The second alternative was the complete replacement of the deployed Gen-2 program with an autonomous detection technology and expanded deployment.

As we reported in September 2012, BioWatch program officials acknowledged that other options—including but not limited to deploying some combination of both technologies (the currently deployed system and an autonomous detection system), based on risk and logistical considerations—may be more cost-effective. As with the Mission Needs Statement, program officials told us that they were advised that a comprehensive AoA would not be necessary because there was already departmental consensus that autonomous detection was the optimal solution. Because the Gen-3 AoA did not: evaluate a complete solution set; consider complete information on cost and benefits; and include a cost-benefit analysis, we concluded that it did not provide information on which to base trade-off decisions.

To help ensure DHS based its acquisition decisions on reliable performance, cost, and schedule information developed in accordance with guidance and good practices, in our September 2012 report, we recommended that before continuing the Gen-3 acquisition, DHS reevaluate the mission need and possible alternatives based on cost- benefit and risk information. DHS concurred with the recommendation and in 2012, DHS directed the BioWatch program to complete an updated AoA.

DHS contracted with the Institute for Defense Analyses (IDA) to conduct the updated AoA, which they issued in December 2013. In January 2014, as part of recommendation follow-up, we reviewed the completed analysis. IDA cited the DHS Acquisition Management Instruction/Guidebook and its appendix on conducting an AoA as the criteria for their study. The management directive lays out a sample framework that details the specific steps to take in evaluating acquisition alternatives, which the contractor used for completing its study. On the basis of our review, we concluded that the IDA-conducted AoA followed the DHS guidance and resulted in a more robust exploration of alternatives than the previous effort. The AoA was not intended to identify a specific solution to address DHS’s requirements for earlier warning and detection capabilities. According to IDA, the AoA does not claim to select a solution, but rather to present alternatives and the information required to select an alternative based on cost and effectiveness trade-offs.

On April 24, 2014, the DHS Acquisition Review Board reviewed the BioWatch Gen-3 acquisition with OHA and issued an ADM announcing the cancellation of the acquisition of Gen-3. According to the DHS ADM, the AoA “did not confirm an overwhelming benefit to justify the cost of a full technology switch” to Gen-3. The ADM also announced that S&T will explore development and maturation of an effective and affordable automated aerosol biodetection capability, or other operational enhancements, that meet the operational requirements of the BioWatch system.

In April 2014, BioWatch Program officials said multiple factors influenced the decision to end the Gen-3 acquisition, including budget considerations, considerations regarding the readiness level of the technology, and the cost to field and maintain the technology. BioWatch Program officials said that the Homeland Security Studies and Analysis Institute’s and our recommendations to complete a robust AoA, which resulted in not identifying a clear path forward for a single technology type for the Gen-3 acquisition, was also a contributing factor. According to BioWatch Program officials, DHS has not ruled out the possibility of pursuing autonomous detection for the BioWatch program, but officials said the technology would have to cost less to develop and maintain than was estimated for the Gen-3 system.

Earlier this year, we reported that when programs have been canceled, cost, schedule, and performance problems have often been cited as reasons for this decision, and cancellation can be perceived as failure. However, in some circumstances, program cancellation may be the best choice. In an April 2014 interview, BioWatch Program officials said the Gen-3 acquisitions process yielded many benefits, despite its cancellation. BioWatch Program officials said the program office has learned and gained much from this experience, including engaging state and local stakeholders to help ensure confidence in the system and BioWatch program; finding better ways to test technologies and refine the Testing and Evaluation guidance; and developing robust acquisition documentation for the department. BioWatch program officials said the decision to cancel the Gen-3 acquisition was a cost-effectiveness measure, because the system was going to be too costly to develop and maintain in its current form. We reported in 2012 that while the DHS June 2011 life-cycle cost estimate reported $104 million in actual and estimated costs from fiscal year 2008 through fiscal year 2011, it also indicated that Gen-3 was expected to cost $5.8 billion (80 percent confidence) from fiscal year 2012 through June 2028. However, the original life-cycle cost estimate for the 2009 decision—a point estimate unadjusted for risk—was $2.1 billion.

DHS R&D Efforts Also Face Challenges that Could Impact the BioWatch Program

DHS has taken positive steps as we recommended to complete a robust assessment of the available biodetection technology alternatives and has taken into consideration the cost and readiness level of the current technology. However, our prior work reviewing DHS research and development efforts highlights challenges DHS may face in transitioning the future biodetection development efforts S&T is now charged with exploring back to the program office, OHA. For example, S&T works with DHS components to ensure that it meets their R&D needs by signing technology transition agreements (TTA) to ensure that components use the technologies S&T develops. However, we previously reported in September 2012 that while S&T had 42 TTAs with DHS components, none of these TTAs has yet resulted in a technology being transitioned from S&T to a component. In that review we also found that other DHS component officials we interviewed did not view S&T’s coordination practices positively. Specifically, we interviewed officials in six components to discuss the extent to which they coordinated with S&T on R&D activities. Officials in four components stated that S&T did not have an established process that detailed how S&T would work with its customers or for coordinating all activities at DHS. For example, officials in one component stated that S&T has conducted R&D that it thought would address the component’s operational need but, when work was completed, the R&D project did not fit into the operational environment to meet the component’s needs.

We also reported in 2012 that OHA, which oversees operation of the BioWatch program, and S&T already had a history of working together on advancing the technology used by the BioWatch program. differences of opinion on key performance measures had created a challenge for these two offices related to future biodetection technologies. For example, during our 2012 review of the Gen-3 acquisition, officials from OHA said both OHA and S&T commissioned the Sandia National Laboratory to conduct similar studies on the performance characteristics of the Gen-3 autonomous detection system, but the two offices requested the use of different performance metrics to evaluate Gen-3’s detection capability. OHA officials said they supported using the fraction of the population covered as the metric because it is directly related to public health outcomes, while S&T preferred to use the probability of detection. While we recognize there are advantages and disadvantages for choosing different performance metrics, technology transition of the R&D project developed by S&T could prove challenging in the future if fundamental differences like this are not resolved early to help ensure the technology meets the operational needs of the program office. processes for coordinating R&D. As a result, we recommended that DHS develop and implement policies and guidance for defining and overseeing R&D at the department-level that includes a well-understood definition of R&D that provides reasonable assurance that reliable accounting and reporting of R&D resources and activities for internal and external use are achieved. DHS agreed with our recommendation, and in April 2014, updated its guidance to include a definition of R&D, but efforts to develop a specific policy outlining R&D roles and responsibilities and a process for coordinating R&D with other offices remain ongoing and have not yet been completed.

Future Considerations for the Currently Deployed Gen-2 system

With the cancellation of the Gen-3 acquisition, DHS will continue to rely on its currently deployed Gen-2 system as an early indicator of an aerosolized biological attack. Cancellation of the Gen-3 system also raises questions that need to be answered about the future maintenance of the Gen-2 system, since it will no longer be replaced, as planned. According to program officials that we recently contacted, DHS is considering multiple options to upgrade the current technology to improve detection capabilities in the wake of the Gen-3 acquisition cancellation. In April 2014, program officials described some of the options they are considering to upgrade the currently deployed system, including:

The addition of a trigger to the current system to enhance performance indoors. These are generally systems that provide very fast but nonspecific warnings of a potential agent release, because they do not identify the type of biological material detected. However, DHS is exploring how to use a trigger to indicate when an air sample should be collected and taken to the laboratory for analysis.

Use of a wet or liquid filter system rather than the current dry filter system. Collecting samples directly into a liquid could also increase the odds that any microorganisms would remain alive for subsequent testing. Increased frequency of manual filter collection and testing, which would likely increase costs.

Other options for hand-held or portable detection devices.

While OHA officials determine the next steps with S&T for the BioWatch program to try and address the capability gap that Gen-3 intended to fill, there are other considerations for the currently deployed system, such as maintainability of the current technology and equipment and the costs associated with any upgrades to extend the life of the existing system. For example, BioWatch program officials indicated they will need to replace the laboratory equipment for the currently deployed system, as early as 2015, and readjust life cycle costs.

Further, while Gen-2 has been used in the field for over a decade, information about the technical capabilities for the Gen-2 system, including the limits of detection, is limited. In 2011, the National Academy of Sciences stated that the rapid initial deployment of BioWatch did not allow for sufficient testing, validation, and evaluation of the system and its components. there is considerable uncertainty about the likelihood and magnitude of a biological attack, and how the risk of a release of an aerosolized pathogen compares with risks from other potential forms of terrorism or from natural diseases. Further, the report also stated that to achieve its health protection goals, the BioWatch system should be better linked to a broader and more effective national biosurveillance framework that will help provide state and local public health authorities, in collaboration with the health care system, with the information they need to determine the appropriate response to a possible or confirmed attack or disease outbreak.

See Institute of Medicine and National Research Council, BioWatch and Public Health Surveillance, 2011. prioritizing resources to help ensure a coherent effort across a vast and dispersed interagency, intergovernmental, and intersectoral network. Therefore, we called for a strategy that would, among other things, (1) define the scope and purpose of a national capability; (2) provide goals, objectives and activities, priorities, milestones, and performance measures; and (3) assess the costs and benefits and identify resource and investment needs, including investment priorities. In July 2012, the White House released the National Strategy for Biosurveillance to describe the U.S. government’s approach to strengthening biosurveillance, but it does not fully meet the intent of our prior recommendations, because it does not yet offer a mechanism to identify resource and investment needs, including investment priorities among various biosurveillance efforts. We remain hopeful that the forthcoming strategic implementation plan which was supposed to be issued in October 2012 and promised to include specific actions and activity scope, designated roles and responsibilities, and a mechanism for evaluating progress will help to address the ongoing need for mechanisms that will help prioritize resource allocation. However, as of March 14, 2014 the implementation plan had not been released.

Chairman Brooks, Ranking Member Payne, and members of the subcommittee, this concludes my prepared statement. I would be happy to respond to any questions you may have.

GAO Contacts and Staff Acknowledgements

If you or your staff members have any questions about this testimony, please contact me at (404) 679-1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Other contributors include; Edward George, Kathryn Godfrey, Eric Hauswirth, Susanna Kuebler, and Linda Miller.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.