Biosurveillance: Observations on the Cancellation of BioWatch Gen-3 and Future Considerations for the Program
Published: 2014-06-10T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: DHS's BioWatch program aims to detect the presence of biological agents considered to be at a high risk for weaponized attack in major U.S. cities. Initially, development of a next generation technology (Gen-3) was led by DHS S&T, with the goal of improving upon currently deployed technology (Gen-2). Gen-3 would have potentially enabled collection and analysis of air samples in less than 6 hours, unlike Gen-2 which can take up to 36 hours to detect and confirm the presence of biological pathogens. Since fiscal year 2007, OHA has been responsible for overseeing the acquisition of this technology. GAO has published a series of reports on biosurveillance efforts, including a report on DHS's Gen-3 acquisition.
In April 2014, DHS cancelled the acquisition of Gen-3 and plans to move development efforts of an affordable automated aerosol biodetection capability, or other enhancements to the BioWatch system to DHS S&T. This statement addresses (1) observations from GAO's prior work on the acquisition processes for Gen 3, and the current status of the program; (2) observations from GAO's prior work related to DHS S&T and the impact it could have on the BioWatch program; and (3) future considerations for the currently deployed Gen-2 system.
This testimony is based on previous GAO reports issued from 2010 through 2014 related to biosurveillance and research and development, and selected updates obtained from January to June 2014. For these updates, GAO reviewed studies and documents and interviewed officials from DHS and the national labs, which have performed studies for DHS.
section_title: What GAO Found
paragraphs: In September 2012, GAO reported that the Department of Homeland Security (DHS) approved the Office of Health Affairs (OHA) acquisition of a next generation biosurveillance technology (Gen-3) in October 2009 without fully following its acquisition processes. For example, the analysis of alternatives (AoA) prepared for the Gen-3 acquisition did not fully explore costs or consider benefits and risk information in accordance with DHS's Acquisition Life-cycle Framework. To help ensure DHS based its acquisition decisions on reliable performance, cost, and schedule information, GAO recommended that before continuing the Gen-3 acquisition, DHS reevaluate the mission need and alternatives. DHS concurred with the recommendation and in 2012 decided to reassess mission needs and conduct a more robust AoA. Following the issuance of the AoA in December 2013, DHS decided in April 2014 to cancel Gen-3 acquisition and move the technology development back to the Science and Technology Directorate (S&T). According to DHS's acquisition decisions memorandum, the AoA did not confirm an overwhelming benefit to justify the cost of a full technology switch to Gen-3. Moreover, DHS officials said the decision to cancel the Gen-3 acquisition was a cost-effectiveness measure, because the system was going to be too costly to develop and maintain in its current form.
GAO's prior work on DHS research and development (R&D) highlights challenges DHS may face in shifting efforts back to S&T and acquiring another biodetection technology. In September 2012, GAO reported that while S&T had dozens of technology transition agreements with DHS components, none of these had yet resulted in a technology developed by S&T being used by a component. At the same time, other DHS component officials GAO interviewed did not view S&T's coordination practices positively. GAO recommended that DHS develop and implement policies and guidance for defining and overseeing R&D at the department that includes a well-understood definition of R&D that provides reasonable assurance that reliable accounting and reporting of R&D resources and activities for internal and external use are achieved. S&T agreed with GAO's recommendations and efforts to address them are ongoing. Addressing these coordination challenges could help to ensure that S&T's technology development efforts meet the operational needs of OHA.
Cancellation of the Gen-3 acquisition also raises potential challenges that the currently deployed Gen-2 system could face going forward. According to DHS officials, DHS will continue to rely on its Gen-2 system as an early indicator of an aerosolized biological attack. However, in 2011, National Academy of Sciences raised questions about the effectiveness of the currently deployed Gen-2 system. While Gen-2 has been used in the field for over a decade, the National Academy of Sciences reported that information about the technical capabilities of the system, including the limits of detection, is limited. In April 2014, DHS officials also indicated that they will soon need to replace laboratory equipment of the currently deployed Gen-2 system and readjust life cycle costs since there will be no Gen-3 technology to replace it.

section_title: Letter
paragraphs: I am pleased to be here today to discuss our observations on the  Department of Homeland Security’s (DHS) BioWatch program, with  particular focus on the cancellation of BioWatch Generation 3 (Gen-3)  and future considerations for the program. In recent years, there has  been an increasing awareness of the potential for biological agents to be  used as weapons of mass destruction. Experts and practitioners, reacting  to an increasing awareness of the speed and intensity with which a  biological weapon of mass destruction could affect the nation, have  sought to augment traditional surveillance activities with biosurveillance  programs and systems. DHS’s BioWatch program is an example of such  an effort. BioWatch aims to reduce the time required to recognize and  characterize potentially catastrophic aerosolized attacks by detecting the  presence of five biological agents—considered to be at a high risk for  weaponized attack—in the air.
DHS’s Office of Health Affairs (OHA) oversees the currently deployed  BioWatch technology—Generation-2 (Gen-2) — which can take 12 to 36  hours to confirm the presence of pathogens. Until recently, DHS had  been pursuing a next generation technology (Gen-3) with the goal of  improving upon existing technology by enabling autonomous collection  and analysis of air samples using the same laboratory science that is  carried out in manual processes to operate the current system (e.g., lab- in-a-box). The new technology would have reduced detection time,  potentially generating a result in under 6 hours, and eliminated certain  labor costs.
This statement includes observations from our prior work (1) on DHS’s  acquisition processes for Gen-3, and the current status of the program;  (2) related to DHS’s  Science and Technology Directorate (S&T) and the  impact it could have on the BioWatch program; and (3) future  considerations for the currently deployed Gen-2 system.
This testimony is based on our previous reports issued from 2010 through  2014 related to biosurveillance, research and development, and  acquisitions. For this work, we reviewed DHS’s acquisition guidance,  including Acquisition Management Directive 102-01. Additionally, we  reviewed acquisition documentation and interviewed agency officials from  the BioWatch program and other DHS offices with development, policy,  and acquisition responsibilities. We then compared the information  developed from our documentation review and interviews against the  guidance. We also interviewed S&T leadership, technical division  directors, and DHS component officials to discuss S&T and DHS’s  research and development (R&D) coordination processes. More detailed  information on our scope and methodology appears in the published  reports. This statement is also based in part on selected updates we  conducted in June 2013 and July 2013 related to DHS’s R&D efforts and  its oversight of R&D efforts across the department and on selected  updates related to the BioWatch program conducted from January to  June 2014.and documents and interviewed knowledgeable officials at DHS and the  national laboratories, which have performed testing and studies for DHS.
For updates on the BioWatch program, we analyzed studies  We conducted the work upon which this statement is based and the  selected updates in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives. Additional details  on our scope and methodology can be found in the individual products  cited throughout this statement.
subsections: section_title: Background
subsections: section_title: DHS Acquisitions and the Cancellation of Gen-3
paragraphs: We have highlighted DHS acquisition management issues in our high-risk  list since 2005. Over the past several years, our work has identified  significant shortcomings in the department’s ability to manage an  expanding portfolio of major acquisitions. We have also reported that  while DHS acquisition policy reflects many key program management  practices intended to mitigate the risks of cost growth and schedule slips,  the department did not implement the policy consistently. In 2011,  expressing concerns about whether DHS had undertaken a rigorous effort  to help guide its Gen-3 decision making, members of Congress asked us  to examine issues related to the Gen-3 acquisition. We released a report  that evaluated the acquisition decision-making process for Gen-3 in  September 2012. As discussed later in the statement, we recommended  that before continuing the Gen-3 acquisition, DHS should carry out key  acquisition steps, including reevaluating the mission need and  systematically analyzing alternatives based on cost-benefit and risk  information.
On April 24, 2014, DHS issued an Acquisition Decision Memo (ADM)  announcing the cancellation of the acquisition of Gen-3. The ADM also  announced that S&T will explore development and maturation of an  effective and affordable automated aerosol biodetection capability, or  other operational enhancements, that meet the operational requirements  of the BioWatch system. DHS’s S&T conducts research, development,  testing, and evaluation of new technologies that are intended to  strengthen the United States’ ability to prevent and respond to nuclear,  biological, explosive, and other types of attacks within the United States.  S&T has six technical divisions responsible for managing S&T’s research  R&D portfolio and coordinating with other DHS components to identify  R&D priorities and needs.applied research and development projects for its DHS customers.
section_title: BioWatch in Action
paragraphs: The BioWatch program collaborates with 30 BioWatch jurisdictions  throughout the nation to operate approximately 600 Gen-2 collectors.  These detectors rely on a vacuum-based collection system that draws air  samples through a filter. These filters must be manually collected and  transported to state and local public health laboratories for analysis using  a process called Polymerase Chain Reaction (PCR). During this process,  the sample is evaluated for the presence of genetic material from five  different biological agents. If genetic material is detected, a BioWatch  Actionable Result (BAR) is declared. Figure 1 shows the process that  local BioWatch jurisdictions are to follow when deciding how to respond to  a BAR.
section_title: Our Prior Work on the Gen-3 Acquisition Identified Challenges and DHS Has Since Cancelled the Program
paragraphs: Our prior findings and recommendations related to the Gen-3 acquisition  provide DHS with lessons learned for future decision making. In  September 2012, we found that DHS approved the Gen-3 acquisition in  October 2009 without fully developing critical knowledge that would help  ensure sound investment decision making, pursuit of optimal solutions,  and reliable performance, cost, and schedule information. Specifically, we  found that DHS did not engage the initial phase of its Acquisition Life- cycle Framework, which is designed to help ensure that the mission need  driving the acquisition warrants investment of limited resources.  BioWatch officials stated that they were aware that the Mission Needs  Statement prepared in October 2009 did not reflect a systematic effort to  justify a capability need, but stated that the department directed them to  proceed because there was already departmental consensus around the  solution. Accordingly, we concluded that the utility of the Mission Needs  Statement as a foundation for subsequent acquisition efforts was limited.
Additionally, in September 2012, we found that DHS did not use the  processes established by its Acquisition Life-cycle Framework to  systematically ensure that it was pursuing the optimal solution—based on  cost, benefit, and risk—to mitigate the capability gap identified in the  Mission Needs Statement. The DHS Acquisition Life-cycle Framework  calls for the program office to develop an analysis of alternatives (AoA)  that systematically identifies possible alternative solutions that could  satisfy the identified need, considers cost-benefit and risk information for  each alternative, and finally selects the best option from among the  alternatives. However, we found that the AoA prepared for the Gen-3  acquisition did not reflect a systematic decision-making process. For  example, in addition to—or perhaps reflecting—its origin in a  predetermined solution from the Mission Needs Statement, the AoA did  not fully explore costs or consider benefits and risk information as part of  the analysis. Instead, the AoA focused on just one cost metric that  justified the decision to pursue autonomous detection—cost per detection  cycle—to the exclusion of other cost and benefit considerations that might  have further informed decision makers. Additionally, we found that the  AoA examined only two alternatives, though the guidance calls for at least  three. The first alternative was the currently deployed Gen-2 technology  with a modified operational model (which by definition was unable to meet  the established goals). The second alternative was the complete  replacement of the deployed Gen-2 program with an autonomous  detection technology and expanded deployment.
As we reported in September 2012, BioWatch program officials  acknowledged that other options—including but not limited to deploying  some combination of both technologies (the currently deployed system  and an autonomous detection system), based on risk and logistical  considerations—may be more cost-effective. As with the Mission Needs  Statement, program officials told us that they were advised that a  comprehensive AoA would not be necessary because there was already  departmental consensus that autonomous detection was the optimal  solution. Because the Gen-3 AoA did not: evaluate a complete solution  set; consider complete information on cost and benefits; and include a  cost-benefit analysis, we concluded that it did not provide information on  which to base trade-off decisions.
To help ensure DHS based its acquisition decisions on reliable  performance, cost, and schedule information developed in accordance  with guidance and good practices, in our September 2012 report, we  recommended that before continuing the Gen-3 acquisition, DHS  reevaluate the mission need and possible alternatives based on cost- benefit and risk information. DHS concurred with the recommendation  and in 2012, DHS directed the BioWatch program to complete an updated  AoA.
DHS contracted with the Institute for Defense Analyses (IDA) to conduct  the updated AoA, which they issued in December 2013. In January 2014,  as part of recommendation follow-up, we reviewed the completed  analysis. IDA cited the DHS Acquisition Management  Instruction/Guidebook and its appendix on conducting an AoA as the  criteria for their study. The management directive lays out a sample  framework that details the specific steps to take in evaluating acquisition  alternatives, which the contractor used for completing its study. On the  basis of our review, we concluded that the IDA-conducted AoA followed  the DHS guidance and resulted in a more robust exploration of  alternatives than the previous effort. The AoA was not intended to identify  a specific solution to address DHS’s requirements for earlier warning and  detection capabilities. According to IDA, the AoA does not claim to select  a solution, but rather to present alternatives and the information required  to select an alternative based on cost and effectiveness trade-offs.
On April 24, 2014, the DHS Acquisition Review Board reviewed the  BioWatch Gen-3 acquisition with OHA and issued an ADM announcing  the cancellation of the acquisition of Gen-3. According to the DHS ADM,  the AoA “did not confirm an overwhelming benefit to justify the cost of a  full technology switch” to Gen-3. The ADM also announced that S&T will  explore development and maturation of an effective and affordable  automated aerosol biodetection capability, or other operational  enhancements, that meet the operational requirements of the BioWatch  system.
In April 2014, BioWatch Program officials said multiple factors influenced  the decision to end the Gen-3 acquisition, including budget  considerations, considerations regarding the readiness level of the  technology, and the cost to field and maintain the technology. BioWatch  Program officials said that the Homeland Security Studies and Analysis  Institute’s and our recommendations to complete a robust AoA, which  resulted in not identifying a clear path forward for a single technology type  for the Gen-3 acquisition, was also a contributing factor. According to  BioWatch Program officials, DHS has not ruled out the possibility of  pursuing autonomous detection for the BioWatch program, but officials  said the technology would have to cost less to develop and maintain than  was estimated for the Gen-3 system.
Earlier this year, we reported that when programs have been canceled,  cost, schedule, and performance problems have often been cited as  reasons for this decision, and cancellation can be perceived as failure.  However, in some circumstances, program cancellation may be the best  choice. In an April 2014 interview, BioWatch Program officials said the  Gen-3 acquisitions process yielded many benefits, despite its  cancellation. BioWatch Program officials said the program office has  learned and gained much from this experience, including engaging state  and local stakeholders to help ensure confidence in the system and  BioWatch program; finding better ways to test technologies and refine the  Testing and Evaluation guidance; and developing robust acquisition  documentation for the department. BioWatch program officials said the  decision to cancel the Gen-3 acquisition was a cost-effectiveness  measure, because the system was going to be too costly to develop and  maintain in its current form. We reported in 2012 that while the DHS June  2011 life-cycle cost estimate reported $104 million in actual and  estimated costs from fiscal year 2008 through fiscal year 2011, it also  indicated that Gen-3 was expected to cost $5.8 billion (80 percent  confidence) from fiscal year 2012 through June 2028. However, the  original life-cycle cost estimate for the 2009 decision—a point estimate  unadjusted for risk—was $2.1 billion.
section_title: DHS R&D Efforts Also Face Challenges that Could Impact the BioWatch Program
paragraphs: DHS has taken positive steps as we recommended to complete a robust  assessment of the available biodetection technology alternatives and has  taken into consideration the cost and readiness level of the current  technology. However, our prior work reviewing DHS research and  development efforts highlights challenges DHS may face in transitioning  the future biodetection development efforts S&T is now charged with  exploring back to the program office, OHA. For example, S&T works with  DHS components to ensure that it meets their R&D needs by signing  technology transition agreements (TTA) to ensure that components use  the technologies S&T develops. However, we previously reported in  September 2012 that while S&T had 42 TTAs with DHS components,  none of these TTAs has yet resulted in a technology being transitioned  from S&T to a component. In that review we also found that other DHS  component officials we interviewed did not view S&T’s coordination  practices positively. Specifically, we interviewed officials in six  components to discuss the extent to which they coordinated with S&T on  R&D activities. Officials in four components stated that S&T did not have  an established process that detailed how S&T would work with its  customers or for coordinating all activities at DHS. For example, officials  in one component stated that S&T has conducted R&D that it thought  would address the component’s operational need but, when work was  completed, the R&D project did not fit into the operational environment to  meet the component’s needs.
We also reported in 2012 that OHA, which oversees operation of the  BioWatch program, and S&T already had a history of working together on  advancing the technology used by the BioWatch program. differences of opinion on key performance measures had created a  challenge for these two offices related to future biodetection technologies.  For example, during our 2012 review of the Gen-3 acquisition, officials  from OHA said both OHA and S&T commissioned the Sandia National  Laboratory to conduct similar studies on the performance characteristics  of the Gen-3 autonomous detection system, but the two offices requested  the use of different performance metrics to evaluate Gen-3’s detection  capability. OHA officials said they supported using the fraction of the  population covered as the metric because it is directly related to public  health outcomes, while S&T preferred to use the probability of detection.  While we recognize there are advantages and disadvantages for  choosing different performance metrics, technology transition of the R&D  project developed by S&T could prove challenging in the future if  fundamental differences like this are not resolved early to help ensure the  technology meets the operational needs of the program office. processes for coordinating R&D. As a result, we recommended that DHS  develop and implement policies and guidance for defining and overseeing  R&D at the department-level that includes a well-understood definition of  R&D that provides reasonable assurance that reliable accounting and  reporting of R&D resources and activities for internal and external use are  achieved. DHS agreed with our recommendation, and in April 2014,  updated its guidance to include a definition of R&D, but efforts to develop  a specific policy outlining R&D roles and responsibilities and a process for  coordinating R&D with other offices remain ongoing and have not yet  been completed.
section_title: Future Considerations for the Currently Deployed Gen-2 system
paragraphs: With the cancellation of the Gen-3 acquisition, DHS will continue to rely  on its currently deployed Gen-2 system as an early indicator of an  aerosolized biological attack. Cancellation of the Gen-3 system also  raises questions that need to be answered about the future maintenance  of the Gen-2 system, since it will no longer be replaced, as planned.  According to program officials that we recently contacted, DHS is  considering multiple options to upgrade the current technology to improve  detection capabilities in the wake of the Gen-3 acquisition cancellation. In  April 2014, program officials described some of the options they are  considering to upgrade the currently deployed system, including:
The addition of a trigger to the current system to enhance  performance indoors. These are generally systems that provide very  fast but nonspecific warnings of a potential agent release, because  they do not identify the type of biological material detected. However,  DHS is exploring how to use a trigger to indicate when an air sample  should be collected and taken to the laboratory for analysis.
Use of a wet or liquid filter system rather than the current dry filter  system. Collecting samples directly into a liquid could also increase  the odds that any microorganisms would remain alive for subsequent  testing.  Increased frequency of manual filter collection and testing, which  would likely increase costs.
Other options for hand-held or portable detection devices.
While OHA officials determine the next steps with S&T for the BioWatch  program to try and address the capability gap that Gen-3 intended to fill,  there are other considerations for the currently deployed system, such as  maintainability of the current technology and equipment and the costs  associated with any upgrades to extend the life of the existing system.  For example, BioWatch program officials indicated they will need to  replace the laboratory equipment for the currently deployed system, as  early as 2015, and readjust life cycle costs.
Further, while Gen-2 has been used in the field for over a decade,  information about the technical capabilities for the Gen-2 system,  including the limits of detection, is limited. In 2011, the National Academy  of Sciences stated that the rapid initial deployment of BioWatch did not  allow for sufficient testing, validation, and evaluation of the system and its  components. there is considerable uncertainty about the likelihood and magnitude of a  biological attack, and how the risk of a release of an aerosolized  pathogen compares with risks from other potential forms of terrorism or  from natural diseases. Further, the report also stated that to achieve its  health protection goals, the BioWatch system should be better linked to a  broader and more effective national biosurveillance framework that will  help provide state and local public health authorities, in collaboration with  the health care system, with the information they need to determine the  appropriate response to a possible or confirmed attack or disease  outbreak.
See Institute of Medicine and National Research Council, BioWatch and Public Health  Surveillance, 2011. prioritizing resources to help ensure a coherent effort across a vast and  dispersed interagency, intergovernmental, and intersectoral network.  Therefore, we called for a strategy that would, among other things, (1)  define the scope and purpose of a national capability; (2) provide goals,  objectives and activities, priorities, milestones, and performance  measures; and (3) assess the costs and benefits and identify resource  and investment needs, including investment priorities. In July 2012, the  White House released the National Strategy for Biosurveillance to  describe the U.S. government’s approach to strengthening  biosurveillance, but it does not fully meet the intent of our prior  recommendations, because it does not yet offer a mechanism to identify  resource and investment needs, including investment priorities among  various biosurveillance efforts. We remain hopeful that the forthcoming  strategic implementation plan which was supposed to be issued in  October 2012 and promised to include specific actions and activity scope,  designated roles and responsibilities, and a mechanism for evaluating  progress will help to address the ongoing need for mechanisms that will  help prioritize resource allocation. However, as of March 14, 2014 the  implementation plan had not been released.
Chairman Brooks, Ranking Member Payne, and members of the  subcommittee, this concludes my prepared statement. I would be happy  to respond to any questions you may have.
section_title: GAO Contacts and Staff Acknowledgements
paragraphs: If you or your staff members have any questions about this testimony,  please contact me at (404) 679-1875 or curriec@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this statement. Other contributors include;  Edward George, Kathryn Godfrey, Eric Hauswirth, Susanna Kuebler, and  Linda Miller.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.