Aviation Security: TSA Has Enhanced Its Explosives Detection Requirements for Checked Baggage, but Additional Screening Actions Are Needed
Published: 2011-07-11T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Explosives represent a continuing threat to aviation security. The Transportation Security Administration (TSA), within the Department of Homeland Security (DHS), seeks to ensure through the Electronic Baggage Screening Program (EBSP) that checked-baggage-screening technology is capable of detecting explosives. Generally, the explosives detection system (EDS) is used in conjunction with explosives trace detection (ETD) machines to identify and resolve threats in checked baggage. As requested, GAO assessed the extent to which: (1) TSA revised explosives detection requirements and deployed technology to meet those requirements, and (2) TSA's approach to the current EDS acquisition meets best practices for schedules and cost estimates and includes plans for potential upgrades of deployed EDSs. GAO analyzed EDS requirements, compared the EDS acquisition schedule against GAO best practices, and interviewed DHS officials. This is a public version of a sensitive report that GAO issued in May 2011.
section_title: What GAO Found
paragraphs: TSA revised EDS explosives detection requirements in January 2010 to better address current threats and plans to implement these requirements in a phased approach. The first phase, which includes implementation of the previous 2005 requirements, is to take years to fully implement. However, deploying EDSs that meet 2010 requirements could prove difficult given that TSA did not begin deployment of EDSs meeting 2005 requirements until 4 years later in 2009. As of January 2011, some number of the EDSs in TSA's fleet are detecting explosives at the level established in 2005. The remaining EDSs in the fleet are configured to meet the 1998 requirements because TSA either has not activated the included software or has not installed the needed hardware and software to allow these EDSs to meet the 2005 requirements. Developing a plan to deploy and operate EDSs to meet the most recent requirements could help ensure EDSs are operating most effectively and should improve checked-baggage screening. However, TSA has faced challenges in procuring the first 260 EDSs to meet 2010 requirements. For example, due to the danger associated with some explosives, TSA and DHS encountered challenges in developing simulants and collecting data on the explosives' physical and chemical properties needed by vendors and agencies to develop detection software and test EDSs prior to the current acquisition. Also, TSA's decision to pursue EDS procurement during data collection complicated both efforts and resulted in a delay of over 7 months for the current acquisition. Completing data collection for each phase of the 2010 requirements prior to pursuing EDS procurements that meet those requirements could help TSA avoid additional schedule delays. TSA has established a schedule for the current EDS acquisition, but it does not fully comply with best practices, and TSA has not developed a plan to upgrade its EDS fleet. For example, the schedule is not reliable because it does not reflect all planned program activities and does not include a timeline to deploy EDSs or plans to procure EDSs to meet subsequent phases of the 2010 requirements. Developing a reliable schedule would help TSA better monitor and oversee the progress of the EDS acquisition. TSA officials stated that to meet the 2010 requirements, TSA will likely upgrade many of the current fleet of EDSs as well as the first 260 EDS machines to be purchased under the current acquisition. However, TSA has no plan in place outlining how it will approach these upgrades. Because TSA is implementing the 2010 requirements in a phased approach, the same EDS machines may need to be upgraded multiple times. TSA officials stated that they were confident the upgrades could be completed on deployed machines. However, without a plan, it will be difficult for TSA to provide reasonable assurance that the upgrades will be feasible or cost-effective.
section_title: What GAO Recommends
paragraphs: GAO recommends that TSA, among other things, develop a plan to ensure that new machines, as well as those machines currently deployed in airports, will be operated at the levels in established requirements, collect explosives data before initiating new procurements, and develop a reliable schedule for the EBSP. DHS concurred with all of GAO's recommendations and has initiated actions to implement them.

section_title: Letter
paragraphs: Explosives represent a continuing threat to aviation security, according to  the Department of Homeland Security (DHS). The Transportation Security  Administration (TSA), the agency within DHS responsible for overseeing  and ensuring civil aviation security, is seeking to ensure that the  technology it uses to screen checked baggage is capable of detecting  these threats.
TSA’s Electronic Baggage Screening Program (EBSP), one of the largest  acquisition programs within DHS, certifies and acquires systems used to  screen checked baggage at 462 commercial airports throughout the  United States. TSA certifies explosives detection-screening technologies  to ensure they meet explosives detection requirements developed in  conjunction with the DHS Science and Technology Directorate (S&T)  along with input from other agencies, such as the Federal Bureau of  Investigation and Department of Defense (DOD). S&T conducts research  and development of new technologies while its Transportation Security  Laboratory (TSL) is responsible for, among other things, independent test  and evaluation of new technologies, including conducting certification  testing of checked-baggage-screening technologies.
Pursuant to the Aviation and Transportation Security Act (ATSA), enacted  in November 2001, TSA deploys explosives detection systems (EDS) and  explosives trace detection (ETD) machines to screen all checked  baggage transported by U.S. and foreign air carriers departing from U.S.  commercial airports. An EDS machine uses computed tomography (CT)  technology to automatically measure the physical characteristics of  objects in baggage. The system automatically triggers an alarm when  objects that exhibit the physical characteristics of explosives are detected.  An ETD machine is used to chemically analyze trace materials after a  human operator swabs checked baggage to identify any traces of  explosive material.
In January 2010, TSA revised explosives detection requirements for the  EDS (hereinafter referred to as “2010 EDS requirements”) to better  address current threats. The specific details included in the 2010 EDS  requirements such as the physical characteristics and minimum masses  of each of the explosives types that EDS machines must detect are  classified. As highlighted in TSA’s 2010 EBSP acquisition strategy, to  improve its existing checked-baggage-screening capability, TSA plans to  procure 260 EDSs (that is, the current acquisition) capable of meeting the  2010 EDS requirements. In addition to acquiring new EDSs to better  address current threats, TSA is planning to procure new EDSs because  many of the currently deployed machines are nearing the end of their  expected service lives.
You requested that we review TSA’s efforts to enhance explosives  detection requirements for checked-baggage-screening technologies and  to ensure that newly acquired and currently deployed explosives  detection technologies meet the enhanced requirements. Specifically, this  report assesses (1) the extent to which TSA has revised explosives  detection requirements and deployed EDSs and ETDs to meet these  revised requirements; (2) any challenges that TSA and S&T have  experienced in implementing the EDS acquisition; and (3) the extent to  which TSA’s approach to its EDS acquisition meets best practices for  schedule and cost estimates and includes plans for potential upgrades to  deployed EDSs.
This report is a public version of the prior sensitive report that we  provided to you in May 2011. DHS deemed some of the information in the  prior report as sensitive, which must be protected from public disclosure.  Therefore, this report omits certain sensitive information about EDS and  ETD explosives detection requirements, including descriptions of those  requirements as well as timeframes for their implementation, the number  of EDSs meeting the requirements, and challenges associated with data  collection for explosives. This report addresses the same questions as  the sensitive report. Also, the overall methodology used for both reports is  the same.
To determine the status of TSA’s efforts to revise explosives detection  requirements for checked baggage screening, we reviewed TSA’s  explosives detection requirements for EDSs established in January 2010  and compared them to explosives detection requirements previously  established in 2005 and 1998 to determine how the requirements differed.  We also identified, analyzed, and discussed with TSA officials how the  2010 EDS requirements compare with current explosives detection  requirements for the ETD established in 2006. We also visited three of  the Department of Energy’s national laboratories—Lawrence Livermore  National Laboratory, Los Alamos National Laboratory, and Sandia  National Laboratories—to determine the status of additional efforts to  further revise the requirements.
To identify any challenges that TSA is experiencing in implementing the  EDS acquisition, we reviewed documentation from TSA’s EBSP—the  program responsible for operational testing, procurement, deployment, and  maintenance of checked-baggage-screening technologies. Among other  things, we reviewed the 2010 EBSP acquisition strategy, the program’s risk  management plan, the technical specifications for its procurement of new  EDSs, and DHS acquisition guidance and directives. We conducted  multiple interviews with EBSP program officials regarding the program’s  approach to the current EDS acquisition and received updates on revisions  to TSA’s planned approach to the acquisition and timelines. We also  compared TSA’s acquisition efforts with internal control standards. In  addition, we conducted site visits and/or telephone interviews with all six  EDS vendors competing in the current EDS procurement and also obtained  their views regarding TSA’s approach to the competitive procurement.  While information we obtained from these interviews may not be  generalized across the industry as a whole, we were able to obtain the  perspectives of all companies planning to compete for the current EDS  procurement, and they were able to provide an understanding of their  companies’ abilities to develop EDSs that meet the 2010 EDS  requirements.
To assess the extent to which TSA’s approach for its EDS acquisition  meets best practices for schedule and cost estimates, and includes plans  for potential upgrades to deployed EDSs, we assessed the original and  revised schedule for the current EDS acquisition against relevant best  practices in our Cost Estimating and Assessment Guide to determine the  extent to which the schedule reflects key estimating practices that are  fundamental to having a reliable schedule. We compared TSA’s efforts  with recommended practices that we previously identified for sound  acquisition planning. We also conducted interviews with DHS officials  and conducted site visits to S&T’s TSL to obtain additional perspective on  TSA’s efforts to deploy EDSs that meet the 2010 requirements. Further,  we visited Tyndall Air Force Base (AFB), Florida, where the Air Force  Research Laboratory, TSL, and the Department of Energy’s Lawrence  Livermore National Laboratory are assisting S&T’s Explosives Division’s  data collection efforts. Regarding TSA’s planning for potential upgrades to  already deployed EDSs, we interviewed TSA and S&T officials to identify  the explosives detection technologies that are currently used for checked- baggage-screening. We also identified the number of currently deployed  EDSs that meet the 1998 and 2005 EDS explosives detection  requirements, the number of currently deployed ETD that meet the 2002  and 2006 ETD requirements, any challenges involved and expected in  upgrading EDS detection capabilities, and TSA’s plans to upgrade EDSs  to meet its 2010 requirements. Additionally, in our site visits and  telephone interviews with the six vendors as we previously discussed, we  asked vendors to also provide their perspectives on TSA’s approach to  upgrading currently deployed EDSs as well as those to be deployed in the  future.
We conducted this performance audit from September 2009 through May  2011 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. During the  course of our review, we revised the engagement objectives and scope to  facilitate a broader examination of TSA’s efforts to revise its explosives  detection requirements and related schedule for the EDS acquisition, a  revision that increased the time for completing this audit. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Appendix I contains additional  information on the objectives, scope, and methodology of our review.
subsections: section_title: Background
subsections: section_title: The Roles of TSA and S&T After the September 11, 2001, terrorist attacks, the Aviation and
paragraphs: Transportation Security Act (ATSA) was enacted. Among other things,  ATSA required that TSA provide for the screening of all checked baggage  for explosives transported on flights departing U.S. commercial airports.  Pursuant to ATSA, TSA deployed EDS and ETD equipment to screen  checked baggage and identify potential threats from explosives. While  TSA is responsible for operating or overseeing the operation of checked- baggage-screening equipment, TSA and S&T share responsibilities for  the research and development of checked-baggage-screening  technologies. During fiscal year 2006, most research and development  functions within DHS, including TSA, were consolidated within S&T. After  this consolidation, S&T assumed primary responsibility for the research,  development, and related test and evaluation of airport checked- baggage-screening technologies. S&T also assumed responsibility from  TSA for the TSL, which tests and evaluates technologies under  development against TSA-established detection requirements. TSA  continues to be responsible for: identifying the requirements for new  checked-baggage-screening technologies; operationally testing and  evaluating technologies in airports; and procuring, deploying, and  maintaining technologies. TSA relies on S&T as the central coordination  point to manage all work related to explosives that involve the TSL,  Lawrence Livermore National Laboratory, Los Alamos National  Laboratory, Sandia National Laboratories, and the Air Force Research  Laboratory at Tyndall AFB.
Deploying EDSs and ETDs  According to TSA, since fiscal year 2001, TSA has made over $8 billion  available to EBSP for activities related to checked-baggage screening.  TSA uses two types of technology for checked baggage screening—the  EDS and the ETD—at 462 U. S. commercial airports. EDS is used to  identify suspicious bulk items or anomalies in checked baggage that  could be explosives or detonation devices. In airports that have EDS, it is  used for primary screening of checked baggage while ETD machines are  used for secondary screening of checked baggage to help resolve EDS  alarms. Additionally, at airports without EDS, the ETD machines are used  for primary screening of checked baggage. See figure 1 for a photograph  of an EDS machine and figure 2 for a photograph of an ETD machine.
TSA deploys EDSs in multiple configurations, such as an in-line  configuration and a stand-alone configuration. The in-line configuration  integrates EDS with an airport’s baggage handling system—the conveyor  system that sorts and transports baggage for loading onto an aircraft.  EDS in stand-alone configurations are separate baggage screening units  that are not integrated with a baggage-handling system and are typically  located in an airport lobby although they may also be located in other  airport locations. Checked baggage is manually loaded and unloaded on  stand-alone EDS machines.
As of October 2010, TSA had 2,297 EDS machines in its fleet, 1,938 of  which were deployed at airports in the United States. At airports and  terminals that do not use EDSs, ETD machines are used for primary  checked-baggage screening. Typically, ETDs are used for primary  screening of checked baggage at smaller airports. These airports typically  do not have EDSs for primary screening of checked baggage. As of  February 2011, TSA estimated that there were about 5,200 ETD  machines used for the primary or secondary screening of checked  baggage at U.S. commercial airports.
section_title: S&T and TSA’s Certification and Testing Process
paragraphs: TSA certifies the EDS it deploys to commercial airports for screening  checked baggage, based on tests performed by the TSL. Specifically,  TSA certifies that EDSs, alone or as part of an integrated system, can  detect, under realistic operating conditions, the amounts, configurations,  and types of explosive material which would be likely to be used to cause  catastrophic damage to an aircraft, using requirements developed in  consultation with experts from outside TSA. Furthermore, TSA  periodically reviews threats to civil aviation security, including:    explosive material that presents the most significant threat to civil  the minimum amounts, configurations, and types of explosive material  that can cause, or would be expected to cause, catastrophic damage  to aircraft in air transportation; and  the amounts, configurations, and types of explosive material that can  be detected reliably by existing or reasonably anticipated, near-term  explosive detection technologies.
Currently, TSA requires that EDSs undergo three types of testing— certification testing, integration testing, and operational testing—before it  will purchase such equipment. First, TSA verifies that vendors’ explosives  detection systems meet—that is, are capable of detecting in accordance  with—the TSA established explosives detection requirements through the  certification testing process. TSA’s decision to certify an EDS relies on  the results of independent test and evaluation performed at the TSL. Prior  to certification testing, TSL conducts preliminary evaluations of vendors’  EDS, known as certification readiness testing (CRT) and pre-certification,  to determine the extent to which vendors are ready to enter certification  testing. During CRT, TSL provides feedback to vendors on their EDS’s  strengths and weaknesses in detecting explosives in order to help  vendors make necessary adjustments to their detection software.  Second, in addition to being certified that the EDS can meet explosives  detection requirements, EDSs being deployed in an in-line configuration  must also undergo integration testing. As part of this testing, machines  deployed in an in-line configuration must demonstrate in a controlled  environment that they can be successfully integrated within the baggage- handling systems used for checked baggage. Finally, following  certification and integration testing, EDSs undergo operational testing in  an airport setting to demonstrate that they can reliably and effectively  function in a live airport environment.
section_title: TSA Has Revised Explosives Detection Requirements for Checked-Baggage- Screening Systems, but Faces Challenges in Deploying Equipment to Meet the Requirements
paragraphs: In 2005, TSA revised explosives detection requirements for the EDS;  however, some number of the EDSs are currently operating at the levels  to detect explosives as set forth only in the 2005 requirements. When  TSA established the 2005 requirements, it did not have a plan that  identified the appropriate time frames needed to deploy EDSs to meet the  requirements. In January 2010, TSA again revised the EDS explosives  detection requirements and plans to deploy EDSs meeting these  requirements in a tiered and phased approach over a number of years.  One tier of requirements consists of three levels and expanded the  number and types of explosives that EDSs must detect. TSA is in the  process of developing another tier of requirements, which will refine the  amount (for example, minimum mass) of an explosive that can cause  catastrophic damage to an aircraft. If TSA deploys EDSs that fully meet  the one tier of the requirements, TSA must ensure that ETD machines are  capable of detecting all of the explosives that EDSs will be able to detect  to minimize any potential screening difference between the EDS and  ETD.
subsections: section_title: TSA Did Not Establish a Plan to Ensure That Currently Deployed EDSs Meet the 2005 Requirements
paragraphs: In November 2005, TSA revised its explosives detection requirements for  EDS that had previously been established in 1998 by the Federal Aviation  Administration. However, as of January 2011, some number of the EDSs  in TSA’s fleet are configured to detect explosives at the levels established  only in the 2005 requirements. The remaining EDSs are configured to  detect explosives at 1998 levels. When TSA established the 2005  requirements, it did not have a plan with the appropriate time frames  needed to deploy EDSs to meet the requirements. Standard practices for  program and project management state that specific desired outcomes or  results should be conceptualized, defined, and documented in the  planning process as part of a road map, along with the appropriate steps,  time frames, and milestones needed to achieve those results.
Despite the absence of a plan, TSA officials stated that they must conduct  testing to compare the false alarm rates for machines operating at one  level of requirements to those operating at another level of  requirements. According to TSA officials, the results of this testing would  allow them to determine if additional staff are needed at airports to help  resolve false alarms once the EDSs are configured to operate at a certain  level of requirements. TSA officials reported that they had anticipated  this operational testing was to be completed in March 2011.
According to agency officials, TSA did not begin this operational testing  immediately after the previous explosives detection requirements were  established in November 2005 because the agency officials were aware  at the time of a potential further revision of the requirements based upon  a planned computer modeling effort to revise the detection standards that  became known as Project Newton. TSA and S&T officials told us they  had planned to use the results from Project Newton to further revise the  explosives detection requirements to reflect the mass of an explosive that  would cause catastrophic damage to an aircraft. Although Project Newton  did not begin until 2007, TSA officials told us that they were aware of  plans to further revise the requirements prior to the initiation of Project  Newton and delayed operational testing in anticipation of the results of  the computer modeling effort. As of April 2011, the EDS explosives  detection requirements have not been changed based on results of the  computer modeling because Project Newton is still under way, though  TSA officials told us that they plan to use the results of Project Newton to  later define a tier of the 2010 EDS requirements. (We discuss the status  of Project Newton in more detail in app. II.)
However, once it became apparent that the results of Project Newton  would not become available to further revise the requirements, TSA did  not establish a plan with time frames for completing the additional testing  related to staffing. Standard practices for planning and project  management suggest TSA should have defined the operational testing  plan and milestones as part of a road map for assessing potential staffing  changes when these EDSs were first deployed after the 2005  requirements were established. Establishing reasonable time frames to  complete the operational testing could help TSA ensure it achieves its  desired goal of activating EDSs capable of detecting the explosives  established in the 2005 requirements in a timely manner.
section_title: TSA Plans to Deploy EDSs That Meet the 2010 Explosives Detection Requirements Using a Phased Approach
paragraphs: In January 2010, TSA revised the EDS checked-baggage explosives  detection requirements partly in response to credible and immediate  threats to civil aviation. TSA plans to meet the 2010 EDS requirements  using a tiered and phased approach: one tier is to be implemented over a  number of years and expands the types of explosives that EDSs must  detect, and another tier adds the use of the results of the computer- modeling effort known as Project Newton. TSA is not certain when this  tier of requirements will be implemented. TSA plans to deploy EDSs that  meet one tier of the 2010 requirements in a phased approach beginning  in late fiscal year 2011 as part of its planned EDS acquisition. Regarding  the other tier, TSA officials stated that Project Newton is to enhance  TSA’s understanding of explosives effects by simulating hundreds of  explosives tests using computer modeling to determine the effects  explosives will have when placed in different locations within an aircraft.  TSA’s and S&T’s understanding of how explosives affect aircraft has  largely been based on data obtained from live-fire explosive tests on  retired aircraft hulls and other data. Project Newton is jointly managed  and funded by DHS’s S&T and TSA. Through fiscal year 2009, S&T and  TSA had invested about $12.5 million in national laboratories for  computer modeling activities as part of Project Newton, according to a  senior TSA official. We discuss Project Newton and its budget in more  detail in appendix II.
TSA plans to implement one tier of requirements in a phased approach  that consists of three levels (see fig. 3). As our past work has shown, an  incremental or phased approach to implementing requirements can  reduce risk and make a program more achievable by providing more time  to develop and test key technologies. TSA officials told us that the ability  to develop EDS is likely to become increasingly complex as the  implementation of requirements progresses. According to TSA, it expects  to begin procuring EDSs to meet the 2010 requirements in July 2011.  Consequently, if TSA is successful in deploying EDSs that meet all three  levels of the 2010 requirements, TSA’s EDS fleet would be certified to  detect more explosives than a fleet meeting the 1998 or 2005  requirements.
ETD machines are currently certified to detect some different explosives  than those EDSs that meet the 2005 EDS detection requirements.  However, if TSA purchases and deploys EDSs that fully meet Level C of  the 2010 EDS requirements, ETD machines are not required to detect all  of the explosives that can be detected by EDSs. TSA’s existing ETD  explosives detection requirements identify the types and quantities of  explosives materials, that is, traces of explosives, that must be detected  and the minimum detection rate for each category of explosive.
According to TSA officials, the ETD explosives detection requirements  have not been revised because TSA wanted to first focus on revising the  EDS explosives detection requirements in time for its planned EDS  acquisition, which is aimed at replacing and upgrading its fleet of EDSs  used to screen checked baggage. TSA officials stated that they are  developing a combined set of explosives detection requirements that  could eventually result in the ETD and EDS machines detecting the same  explosives. TSA officials stated that the combined set of EDS and ETD  requirements would not be expected to be approved until sometime in  calendar year 2011. Although combined detection requirements for ETD  and EDS are to help ensure that both machines can detect the same  explosives, the machines are not expected to be required to detect the  same amounts of explosives because the purpose of the ETD is to detect  traces of explosives in nanograms while the EDS is designed to detect  larger amounts of explosives.
At all airports that use EDSs to screen checked baggage, ETD machines  are used in conjunction with EDSs to screen checked baggage for  explosives. At these airports, if an EDS alarms—indicating that checked  baggage may contain an explosive or explosive device that cannot be  cleared—ETD machines are used as a secondary screening device in  order to attempt to resolve the alarm. However, the differences between  the EDS and ETD requirements may impact the resolution of EDS alarms  by the ETD in the future. According to TSA’s 2010 EBSP Acquisition  Strategy, additional equipment—other than the ETDs currently  deployed—is to be employed to support alarm resolution when EDSs that  meet the new checked-baggage explosives detection requirements are  deployed. However, the acquisition strategy does not specify what  additional equipment or screening protocols will be employed to resolve  alarms nor does the strategy discuss whether TSA will continue to use  ETD equipment to resolve EDS alarms. According to TSA, the agency is  currently evaluating what additional technologies and/or changes to  screening protocols may be needed to address any potential gap in  capability between the newly certified EDSs and ETDs used for alarm  resolution. If TSA begins operating EDSs that detect explosives in  subsequent phases of the 2010 requirements, this potential screening  difference between EDS and ETD will exist until TSA deploys additional  equipment and/or implements new screening protocols that could be used  for secondary screening.
In commenting on this issue, TSA officials stated that checked-baggage- screening technologies are only one layer of security and that other layers  of security exist to help address potential threats to the aviation security  system. However, officials agreed that they have not yet developed new  screening protocols or deployed additional equipment that will address  the potential gap in screening capability between EDS and ETD if the new  EDSs are deployed. Standards for program management require that  specific desired outcomes or results be conceptualized, defined, and  documented in the planning process as part of a road map, along with the  appropriate steps and time frames needed to achieve those results.  Because TSA decided to revise explosives detection requirements for  EDSs prior to revising the ETD requirements, the differences in the  requirements may affect TSA’s capability to detect the 2010-required  levels until TSA identifies technologies or protocols needed to address  the potential gap. Without a plan to ensure that secondary-screening  devices or protocols are in place to resolve EDS alarms if EDSs are  deployed with additional capability, it will be difficult for TSA to provide  assurances that the potential capability gap has been resolved.
section_title: TSA Has Faced Challenges in Implementing Plans for the Current EDS Acquisition
paragraphs: TSA has developed an EBSP acquisition strategy to guide its efforts to  improve its fleet of checked-baggage-screening machines, but has faced  several challenges in implementing plans for the current EDS acquisition  under this strategy. First, TSA has experienced challenges in collecting  explosives data on the physical and chemical properties of certain  explosives needed by vendors to develop EDS detection software and  needed by TSA before procuring and deploying EDSs to meet the 2010  requirements. TSA and S&T have experienced these challenges because  of problems associated with safely handling and consistently formulating  some explosives. Second, the challenges related to data collection for  certain explosives have resulted in problems carrying out the EDS  procurement as planned. Specifically, attempting to collect data for  certain explosives while simultaneously pursuing the EDS procurement  has delayed the EDS acquisition schedule by at least 7 months. Finally,  EDS vendors have expressed concerns about the extent to which TSA is  communicating with the business community about the current EDS  procurement.
subsections: section_title: TSA’s EBSP Acquisition Strategy Is Being Used to Guide Efforts to Improve the Explosive Detection Capabilities of the EDS
paragraphs: In July 2010, DHS approved TSA’s current acquisition strategy for the  EDS, and under this strategy, TSA plans to increase the threat detection  capabilities of the EDS using a competitive procurement to purchase and  deploy EDS beginning in 2011. According to TSA officials, most of the  previous EDS acquisitions were sole source procurements. However,  TSA is implementing a competitive procurement for the current EDS  acquisition to, in part, meet the EBSP acquisition strategy’s goals and  objectives., Furthermore, the EBSP acquisition strategy calls for  acquiring new EDSs as part of the recapitalization plan to replace aging  EDS. Under the current EDS procurement, TSA plans to award contracts  to purchase 260 EDSs, including those in its recapitalization plan, at an  estimated cost of $256 million during the fourth quarter of fiscal year  2011. Until TSA begins purchasing machines under the current EDS  acquisition to meet 2010 requirements, the agency has continued to  purchase EDSs under existing contracts with current vendors.
section_title: TSA Faces Challenges in Collecting Explosives Data Needed to Procure EDSs That Meet the 2010 Requirements
paragraphs: TSA and S&T have experienced a number of challenges related to  collecting data on some explosives data needed to procure and deploy  EDSs that meet the 2010 requirements. These data are needed both by  vendors to develop EDS detection software and by the TSL for the  certification testing process and includes such information as the physical  and chemical properties of explosives. Participants in the data collection  effort include S&T; the TSL, which is taking the lead; the Air Force  Research Laboratory (AFRL) at Tyndall AFB, Florida; and Lawrence  Livermore National Laboratory. The AFRL is assisting the TSL because the  AFRL facility at Tyndall AFB, Florida, is better equipped to safely handle  certain explosives as part of the testing and data collection efforts than the  TSL facility, according to S&T and Air Force officials.
In the course of collecting data, TSL and AFRL officials determined that  some of the explosives were very unstable or volatile and special care  and procedures were required to reliably and safely handle them. This  caused delays as TSA and S&T were unable to provide vendors with all  of the data on the explosives or simulants, so that vendors could create  and test the software used to detect them. TSL and AFRL also collected  scans of explosives as another means to provide vendors with needed  data. Because micro-computed technology (micro-CT) data are provided  by scanning smaller, less dangerous, amounts of each explosive, TSL  and AFRL officials did not face the same challenges in safely  synthesizing and determining the physical and chemical properties of the  micro-CT data that they faced in working with larger amounts, known as  full threat weight, of explosives. TSL officials told us that they provided  the micro-CT data to help vendors in developing their explosives  detection software. Specifically, TSA was able to distribute the micro-CT  data to vendors in early fiscal year 2010, and five of the six vendors we  interviewed stated that these data were of limited use in developing their  explosives detection software. For example, one vendor stated that the  micro-CT data provided some guidance, but that there were too many  unknowns to fully use the data to develop their explosives detection  software. Further, TSL officials stated that providing the micro-CT data to  vendors only served as an intermediary step to providing full threat-weight  data that vendors needed to develop their explosives detection software.  TSL officials stated that the micro-CT data could not provide vendors with  all of the data they needed to fully develop their explosives detection  software to meet the 2010 EDS requirements because vendors need  scans of the full threat weight of explosives on their respective EDSs to  finish developing their detection software.
Because of the limitations of simulants and micro-CT data, TSA and S&T  decided to collect and distribute scans of the explosives to vendors using  the full threat weight of the explosives specified in the 2010 EDS  requirements. These scans were collected using each vendor’s  respective EDS equipment. TSA has distributed some, but not all, of the  full threat-weight data needed by vendors to develop EDS detection  software. TSL officials stated that they needed full threat-weight data to  conduct certification testing. Additionally, five of the six vendors we  interviewed agreed that the full threat-weight data will be necessary in  order for vendors to develop their explosives detection software.  However, all six vendors noted that because of concerns about the safety  of handling certain explosives, they are relying on TSA for the full threat- weight data. Further, four of the vendors said that in the past they had  access to some explosives and could collect their own data to develop  and test their detection software in order to prepare for certification  testing. However, because the vendors cannot safely handle certain  explosives, they are reliant upon the data provided by TSA.  Consequently, until S&T completes the data collection on all identified  explosives being performed at TSL and the AFRL facility at Tyndall AFB,  TSA cannot provide all of the data that vendors need to develop their  explosives detection software and prepare for certification, nor can the  TSL start certification testing of new equipment as part of the current EDS  acquisition.
section_title: Challenges Related to Collecting Explosives Data Have Delayed the EDS Acquisition Schedule
paragraphs: TSA’s plans to award contracts for the current EDS acquisition have been  delayed by at least 7 months, in part, due to the challenges experienced  by S&T related to collecting explosives data. TSA officials stated that,  initially, they planned to conduct the current EDS acquisition separately  from efforts to collect the data needed to deploy EDSs that meet the 2010  requirements. Specifically, TSA officials stated that they planned to  complete the data collection before initiating the procurement to buy  EDSs that meet the 2010 requirements. However, officials stated that  they subsequently decided to collect explosives data at the same time as  implementing the current EDS acquisition because TSA and other  stakeholders believed that the data collection effort would be  straightforward and that the new requirements could be easily applied to  machines procured in the current EDS acquisition. Additionally, program  officials stated that procuring and deploying EDSs that meet the 2010  requirements in a phased approach (that is, implementing Level C first,  then Level B, then Level A) would help to mitigate any additional  challenges and some of the risks associated with collecting data needed  for the 2010 requirements.
However, TSA and S&T officials acknowledged that pursuing the  competitive procurement and explosives data collection at the same time  had been more challenging than originally anticipated and had presented  problems for the current EDS acquisition. TSA officials stated that all of  the 260 EDSs they plan to purchase in 2011 will be upgraded to meet all  of the 2010 EDS requirements at a later date.
In our prior reports regarding acquisitions, we reported on the elevated  risk of poor program outcomes from the substantial overlap of  development, test, and production activities. Specifically, we have  identified development cost increases, additional delays in manufacturing  and testing schedules, and increased financial risk due to pursuing  procurement before testing is complete. By separating the effort to  collect data on explosives needed to meet the new requirements from the  related competitive procurement, TSA and S&T would have more time to  collect data identifying the physical and chemical properties of explosives,  provide vendors with the time needed to develop detection software, and  attempt to pass CRT and certification testing without the added pressure  of an acquisition deadline. For example, by completing data collection for  each of the phases of the 2010 EDS requirements prior to pursuing  procurements for EDSs that meet those requirements, TSA could avoid  additional delays to the acquisition schedule due to any data collection  challenges. To help avoid these challenges in the future, TSA officials  stated that they do not plan for subsequent procurements of EDS capable  of meeting the more stringent explosives detection requirements until  after the data collection for these explosives has been fully completed.
We recognize that it is difficult in such situations to identify firm  milestones. However, TSA has not documented its revised approach for  conducting the needed data collection and related procurements  sequentially rather than simultaneously. TSA does not yet have a  documented strategy in place for deploying EDSs beyond July 2011; such  a strategy would be valuable because TSA plans to complete the  implementation of all of the requirements at an undetermined time after  July 2011. Standard practices for program management state that the  successful execution of any plan includes identifying in the planning  process the schedule that establishes the timeline for delivering the  plan. Documenting a plan to separate data collection efforts and  certification from future procurements could help TSA ensure it avoids the  challenges it has encountered during the current procurement.
section_title: EDS Vendors Report Concerns about the Extent to Which TSA is Communicating Effectively about the Current Procurement
paragraphs: Officials from five of six EDS vendors we interviewed expressed concerns  about the extent to which TSA has communicated effectively with vendors  interested in the current procurement. Specifically, these five vendors  expressed concerns about the timeliness in which TSA responded to their  questions regarding the current procurement or the manner in which TSA  communicated important schedule changes, or both. Standards for  Internal Control in the Federal Government state that management  should ensure there are adequate means of communicating with and  obtaining information from external stakeholders that may have a  significant impact on the agency achieving its goals. Additionally, the  Federal Acquisition Regulation (FAR) encourages exchanges of  information among all interested parties, from the earliest identification of  a requirement through the receipt of the proposal. The FAR further states  that the purpose of exchanging information is to improve the  understanding of government requirements and industry capabilities,  thereby allowing vendors to judge whether or how they can satisfy the  government’s requirements. The improved understanding resulting from  such information exchange also enhances the government’s ability to  obtain quality supplies and services at reasonable prices and, among  other things, potentially increases efficiency in vendors’ proposal  preparations. However, five out of six vendors we interviewed said TSA  often did not provide information or respond to their questions in a timely  manner, if at all. For example, four out of these five vendors said TSA  did not answer their questions in a timely manner, in one case taking  several months to provide answers to questions posted via a question  tracker accessible online to all interested vendors. Meanwhile four of the  five vendors’ officials stated TSA did not respond at all to some of their  questions, while officials from the fifth vendor stated they were frustrated  with how long it took TSA to answer their questions.
Officials from two vendors stated that the lack of timely communication  regarding schedule changes for the EDS acquisition caused them to incur  additional costs allocating extra resources and time to meet the original  deadline. Specifically, officials from one vendor noted that they spent  additional costs on personnel to aggressively pursue software development  for the planned start of certification readiness testing (CRT), despite not  having all of the full threat-weight explosives data TSA had intended to  provide. Subsequently, these officials told us, TSA did not announce to  vendors that CRT would be delayed until one week prior to the original  deadline. EBSP officials stated that, because vendors had not yet received  all of the full threat weight explosives data, they should have been aware  that CRT was not going to happen according to the established schedule.  However, EBSP officials agreed that providing vendors with a revised  schedule prior to the previously established deadline would have helped  promote greater vendor understanding about the proposed changes to  TSA’s acquisition strategy.
TSA stated that it has taken a number of important steps to alleviate  confusion and provide as much information to the vendors as possible.  Among other things, at the start of the current procurement, TSA  conducted three conferences with industry, called “industry days,” to  provide a forum for sharing information with the vendor community  regarding the current EDS acquisition. TSA also reported sharing multiple  draft versions of the requirements documents and soliciting vendor  comments. Additionally, TSA officials stated that they shared draft copies  of the detection requirements and held individual classified meetings  during the industry days with each interested vendor to obtain input  regarding the acquisition. Finally, TSA stated that it also allowed vendors  to use government owned equipment and paid for engineering services  associated with the testing to help offset vendor costs. Although EBSP  officials stated that they have made a concerted effort to be responsive to  vendors’ questions and to call vendors directly when issues such as  schedule changes arose, EBSP officials agreed that the agency did not  always effectively communicate with vendors in a timely manner.  Establishing a process for more timely communication with vendors  competing for the current EDS procurement could help TSA to ensure  that vendors have all of the information they need to meet TSA’s needs  for new checked baggage screening equipment.
section_title: Improved Acquisition Planning Could Help TSA Avoid Further Delays and Potential Cost Overruns for the EDS Procurement
paragraphs: TSA does not have an integrated master schedule (IMS) for the EBSP,  and TSA’s schedule for the current EDS acquisition, which is only a part  of the program, does not fully meet best practices for preparing an  acquisition schedule. Additionally, while TSA completed an initial cost  estimate for the EBSP, TSA officials reported that the current cost  estimate does not reflect the anticipated costs of purchasing EDSs to  meet the 2010 EDS requirements. To meet the explosives detection  requirements established in January 2010, TSA plans to upgrade the  detection software of a currently unknown number of the deployed EDSs  and 260 of the EDSs to be purchased under the current acquisition after  they are deployed to airports. However, TSA has not yet developed a  plan or cost estimate for the planned upgrades.
subsections: section_title: TSA Does Not Have a Schedule for the EBSP and Has Not Established a Reliable Schedule for the Current EDS Acquisition
paragraphs: As part of EBSP’s responsibility to provide equipment to screen all  checked baggage originating at U.S. commercial airports, it is acquiring  and deploying explosives detection technology to replace aging systems  and meet emerging threats. While TSA established the EBSP as a long- term program to procure, test, deploy, and maintain checked-baggage- screening equipment, TSA officials confirmed in December 2010 that  there is currently no IMS for the EBSP. Among other things, best  practices and related federal guidance call for a program schedule to be  programwide in scope, meaning that it should include the integrated  breakdown of the work to be performed by both the government and its  contractors over the expected life of the program. Without an IMS  identifying long-term plans for the EBSP, it is difficult for TSA to have a  comprehensive program view of the work that must be completed to  deliver explosive detection technology to replace aging systems and meet  emerging threats. Without such a view, a sound basis does not exist for  knowing with any degree of confidence when and how the program will be  completed.
While there is no IMS for the EBSP, TSA has established a schedule for  the current EDS acquisition. However, while the schedule identifies  activities through the first contract award—scheduled for July 2011—of  the current EDS procurement, our analysis shows that it does not identify  activities planned for subsequent award windows. Additionally, TSA has  encountered a number of challenges in implementing the schedule. For  example, according to TSA officials, TSA originally planned to award the  first EDS contract in December 2010 in order to procure machines  required to meet Level C of the 2010 EDS requirements. However, TSA  has since revised the schedule due to the challenges of collecting  explosives data needed before development of EDS can be completed  and certification testing of the machines can begin.
Based on the revised schedule, certification testing began in late 2010,  according to TSA, so that in July 2011 the first EDS contract can be  awarded to procure machines that meet one part of the Level C  explosives detection requirements. Furthermore, while TSA has stated  that it plans to procure and deploy 640 additional EDSs at an estimated  cost of approximately $964 million during fiscal years 2012 through 2015,  it is unclear when TSA plans for those machines to meet the remaining  2010 EDS requirements.
As of March 2011, TSA officials estimate that it will take a number of  years to certify EDSs that meet all three levels—C, B, and A—of the 2010  requirements. However, the officials stated that they cannot fully develop  these plans until they can evaluate the capability of the equipment to  meet these requirements. This is expected to happen during the testing  process associated with the current EDS procurement. TSA officials  stated that they plan to deploy EDSs that meet the full set of Level C, B,  and A requirements, but more precise planning, including establishing  timelines, cannot occur until TSA better understands the potential for the  EDS equipment to meet those requirements. However, best practices  state that a comprehensive schedule should at least reflect all activities  planned for a project even though some activities may be tentative and  there may be uncertainties in schedule estimates due to, among other  things, limited data.
In addition to the challenges TSA has encountered in carrying out the  schedule as originally planned, based on our analysis, the current schedule  leading up to the first contract award is not reliable. Best practices state  that the success of a large-scale system acquisition, such as the current  EDS acquisition, depends in part on having a reliable schedule that  identifies:    when the program’s set of work activities and milestone events will    how long they will take, and    how they are related to one another.
Best practices also call for the schedule to expressly identify and define  the relationships and dependencies among work elements and the  constraints affecting the start and completion of work elements.  Additionally, best practices indicate that a well-defined schedule also  helps to identify the amount of human capital and fiscal resources that are  needed to execute an acquisition. However, based on our assessment of  both the original as well as an updated version of the schedule, TSA’s  schedule for the current EDS acquisition does not fully comply with nine  best practices for preparing a schedule as shown in table 1. Appendix III  has additional information about GAO’s assessment of the extent to  which TSA’s schedule meets each best practice.
Although TSA’s schedule does not fully comply with any of the nine best  practices, TSA has taken action to partially or minimally meet eight of the  best practices. For example, consistent with best practice 4, the schedule  establishes the duration of all activities and properly reflects how long  each activity should take. However, while the schedule establishes the  duration of all activities, 61 percent of activities represented in the  schedule are based on a 7-day calendar that does not account for  holidays. Similarly, our analysis found that, consistent with best practice  5, the schedule is vertically integrated; however, issues with sequencing  logic in the schedule prevent it from being fully horizontally integrated.  Vertical and horizontal integration ensures that products and outcomes  associated with other sequenced activities are arranged in the right order  and that dates for supporting tasks and subtasks are aligned. Other areas  of the schedule that remain unaddressed also reflect weaknesses that  limit its usefulness as a program management tool. For example, the  schedule does not fully identify the resources needed to do the work or  the availability of these resources. Specifically, the schedule does not  reflect what labor, material, and overhead are needed to complete key  activities for the program. Resource information would assist the program  office in forecasting the likelihood of activities being completed based on  their projected end dates. If the current schedule does not allow for  insight into current or projected over-allocation of resources, then the risk  of the program slipping is significantly increased.
Additionally, TSA officials did not complete a schedule risk analysis when  developing the schedule. A schedule risk analysis may be used to  determine the level of uncertainty and to help identify and mitigate the  associated risks. In the absence of a schedule risk analysis, the  acquisition faces the risk of delays to the scheduled completion date if  any delays were to occur on critical path activities. Furthermore, without  this information, TSA is limited in its ability to answer questions such as  how likely it is to complete the project on time and which risks are most  likely to delay the project. Similarly, without a valid critical path, EBSP  management lacks a clear picture of the tasks that must be performed to  achieve the acquisition’s target completion date. While TSA officials noted  that they had no staff or expertise to complete a schedule risk analysis,  TSA provided no explanation as to why a schedule consistent with the  other eight best practices had not been developed. TSA officials stated  that the EDS acquisition is one of the largest acquisition programs in  DHS. However, the absence of a reliable schedule makes it difficult for  management to predict with any degree of confidence whether the  estimated completion date for the acquisition is realistic. Furthermore,  without the development of a schedule that meets scheduling best  practices, TSA is limited in its ability to monitor and oversee the progress  of the billions of dollars being invested in the procurement of new EDSs.
section_title: The EBSP’s Current Life- Cycle Cost Estimate Does Not Reflect Anticipated Costs for Purchasing EDSs That Meet the 2010 EDS Requirements
paragraphs: The EBSP does not yet have an up-to-date approved life-cycle cost  estimate in place, and as a result, DHS has no reliable basis for  understanding how much the program will cost. While TSA reported that it  had completed a life-cycle cost estimate (LCCE) for the EBSP in May  2010, program officials reported in February 2011 that the estimate is  currently being revised to reflect assumptions related to the current EDS  acquisition. Specifically, officials indicated that the May 2010 LCCE did  not include the anticipated costs for purchasing any EDSs that meet the  revised 2010 requirements. TSA officials stated that they are working to  revise the LCCE to reflect the anticipated costs of the current EDS  acquisition and expected to complete the LCCE by the end of April 2011.  Additionally, after conducting a review of the LCCE that was completed in  May 2010, DHS’s Cost Analysis Division (CAD) found that the LCCE  needed more comprehensive data and that its accuracy could not be  determined. As a result, the DHS Acquisition Review Board directed the  CAD to develop an appropriate cost estimate, including a reconciliation  with the EBSP’s LCCE. In January 2011, officials in DHS’s CAD stated  that they had initiated work on the independent cost estimate for the  EBSP but were only able to complete the portion of the estimate related  to current detection capabilities in the Level C requirements for one tier.  Officials stated that the lack of detail in program requirements for some of  Level C and all of Levels B and A limited their ability to develop an  estimate that would be usable for budgetary purposes. CAD officials  further noted that significant portions of the total EBSP program have yet  to be defined and estimated. During the course of our review, the  anticipated completion date of the LCCE has been delayed multiple times  and was expected to be completed at the end of April 2011. As a result,  we were unable to evaluate TSA’s approach to developing the cost  estimates for the program.
We reported in June 2010 that inaccurate or incomplete cost estimates  were often a factor in cost growth for DHS programs we previously  reviewed. We also reported that initial cost estimates for most DHS  programs were often developed after the start of acquisition activities, so  they do not capture earlier cost changes. Further, our best practices for  cost estimation state that estimates are integral to determining and  communicating a realistic view of likely cost and schedule outcomes that  can be used to support a program including planning the work necessary to  develop, produce, and install equipment. However, because TSA had not  established a cost estimate that accurately reflects the anticipated costs of  the acquisition prior to initiating the current EDS procurement, it is unclear  how DHS could determine if the budget for the EBSP is reasonable.  Furthermore, in the absence of an approved cost estimate and baseline  financial information for the current EDS acquisition, including the costs of  purchasing machines that meet the 2010 EDS requirements, TSA has  limited information to make essential cost-informed program decisions.
Although we were unable to evaluate TSA’s cost estimates for the  program, the fact that TSA’s schedule for the EDS acquisition does not  meet best practices for schedule estimating also raises questions about the  credibility of the program’s LCCE. For example, the absence of a schedule  risk analysis would have made it difficult for officials to account for the cost  effects of schedule slippage when developing the LCCE. Best practices for  cost estimation state that because some program costs such as labor,  supervision, rented equipment, and facilities cost more if the program takes  longer, a reliable schedule can contribute to an understanding of the cost  impact if the program does not finish on time. The program’s success  depends on the quality of its schedule and an integrated schedule is key to  managing program performance and is necessary for determining what  work remains and the expected cost to complete the work. In a memo from  the DHS Under Secretary for Management dated July 10, 2008, DHS  endorsed the use of best practices that we identified and stated that DHS  would be utilizing them as a “best practices” approach in the future.  However, in the absence of a reliable schedule to guide cost estimates,  having a current cost estimate that reflects anticipated costs for the EDS  acquisition, or submitting the revised LCCE to DHS for departmental  approval, it is unclear how TSA utilized a best practices approach in  developing cost estimates for the program.
section_title: TSA Has No Plan in Place Outlining How It Will Upgrade Deployed EDSs to Fully Meet the 2010 Requirements
paragraphs: TSA officials stated that they expect to upgrade an unknown number of  the current fleet of 2,297 EDSs and 260 of the EDSs to be purchased  under the current acquisition after they are deployed to airports to fully  meet all phases of the 2010 requirements. However, similar to when  TSA revised the EDS explosives detection requirements in 2005, it has  no plan in place outlining how it will approach these upgrades.  Specifically, TSA has not established an upgrade plan or conducted an  analysis to determine what type of approach to upgrading deployed EDSs  is likely to be most feasible, efficient, or effective. TSA officials stated that  there are too many unknowns at this time regarding potential approaches  to upgrading the fleet of EDSs. Standards for program management  require that specific desired outcomes or results be conceptualized,  defined, and documented in the planning process as part of a road map,  along with the appropriate steps and time frames needed to achieve  those results. Until TSA develops a plan identifying how it will approach  the upgrades for currently deployed EDSs—and the plan includes such  items as estimated costs, the number of machines that can be upgraded,  and the number of times a given machine must be upgraded to meet the  2010 EDS requirements—it will be difficult for TSA to provide reasonable  assurance that its upgrade approach is feasible or cost-effective.
TSA’s 2010 acquisition strategy identifies the planned purchase of 900  EDSs over the next 5 years, but it does not indicate what level  requirements those EDSs will be required to meet. For the currently  deployed equipment that will not be replaced, TSA would need to  upgrade the equipment to meet the 2010 requirements. TSA officials  stated that they may not upgrade all of the current fleet of EDSs to the  2010 EDS requirements because in some cases, certain models of the  EDS may not be upgradeable and in other cases, it may ultimately be  more cost effective to replace older EDSs with new machines. According  to TSA, upgrading EDSs will require an assessment of currently deployed  EDS’ detection capabilities and that the results of the assessment will  affect the EDS program’s schedule, budget, and detection goals. TSA  was working with a consulting firm to modify a computer model that will  be used to project the costs of the upgrades if TSA were to use a time- phased installation for the upgrades. While TSA officials were working  with a consulting firm, they have not yet established a plan for how they  will approach the upgrades. TSA officials further stated that the number of  upgrades TSA performs on currently deployed equipment will depend on  the cost of the upgrades, the level of complexity of the upgrades, and  whether the upgrades can be conducted in the airports or must be  performed in the factory.
TSA’s approach to deploy EDSs that meet the 2010 requirements could  result in the same EDSs being upgraded multiple times in order to first  meet all of the Level C requirements and to then meet the Levels B and A  explosives detection requirements. For example, TSA’s decision to  revise its acquisition strategy and deploy EDSs that meet the Level C  requirements in a phased approach could result in upgrading the same  currently deployed machines twice before they may have to be upgraded  a third time to meet Level B requirements and then upgraded a fourth  time to meet Level A requirements. Moreover, based on TSA’s schedule  for the current EDS acquisition, by the time some or all of the 260 new  EDSs under the current EDS acquisition have been deployed in airports,  TSA may have approved a subsequent tier of the EDS explosives  detection requirements, which could involve upgrading the machines  again or replacing these newly purchased and deployed machines  because they cannot meet the subsequent tier of explosives detection  requirements. Therefore, TSA may procure and deploy 260 EDSs that will  only be used in airports for a short period of time before they will need to  be upgraded, possibly multiple times, or replaced with new machines.
TSA officials told us that they will evaluate the need to upgrade EDSs to a  subsequent tier at the time those requirements are finalized.
TSA officials stated that they initially delayed the analysis of the upgrade  approach until the 2010 EDS explosives detection requirements were  approved, an approval that occurred in January 2010. TSA officials  subsequently stated that their plan to upgrade deployed EDSs is included  in the recapitalization strategy due to be completed at the end of May  2011. According to TSA, vendors that have previously sold EDSs to TSA  are to be asked to also include proposals to upgrade their currently  deployed machines when submitting proposals for the current EDS  procurement. Specifically, vendors are to be asked to include a plan for  upgrading their currently deployed EDS equipment as well as cost  estimates for the upgrades. TSA plans to then analyze the feasibility and  costs of the vendors’ proposals. However, TSA officials stated that the  equipment upgrades may or may not be implemented as part of the  contract award and that TSA has discretion regarding which aspects of  the contracts to implement. According to TSA, the total number of EDSs  to be upgraded and the associated costs will not be known until the  agency receives proposed upgrade plans and cost estimates from EDS  vendors in summer 2011. According to TSA officials, any upgrades are  not to occur until calendar year 2012 at the earliest and will depend on  available funding and complexity of the upgrades.
TSA officials as well as officials from three of six current EDS vendors told  us that they are confident that currently-deployed EDSs can be upgraded  to meet Level C requirements. Specifically, TSA officials stated that the  EDS vendors can rewrite the detection software to provide the capability to  detect the 2010 EDS requirements. Minor hardware changes, such as new  computer chips, are also expected to be made as part of these upgrades  according to the TSA officials. The officials stated that, after they approve  the software and hardware upgrades, EDS vendors will install the upgrades  on the machines in the airports. Officials from three EDS vendors noted  that they believe upgrades to the EDSs can be made in the airports when  regularly-scheduled routine maintenance work is conducted.
Once deployed EDSs have been upgraded to fully meet the Level C  requirements, TSA will have to make decisions about how to ensure  these machines can meet subsequent phases of the 2010 EDS  requirements (Levels B and A). Officials from all six EDS vendors stated  that given the absence of additional data on the explosives that will be  included in subsequent phases of the 2010 EDS requirements, it is  difficult to know precisely what must be done to upgrade newly-purchased  equipment. Therefore, none of the officials from the six vendors could  provide estimates for the cost to upgrade EDSs to meet all of the  requirements for one tier. However, officials from two vendors estimated  the cost to upgrade new EDSs to meet Level C requirements at $50,000  to $150,000 per machine. An official from one vendor stated that the CT  technology currently used in EDSs might not be sufficient to detect Level  A requirements and that an as yet undeveloped technology may be  needed. The official noted that this could result in substantially higher  costs to upgrade the current fleet of machines to Level A requirements  than it would cost to upgrade machines to Level B requirements.  Similarly, officials from two other vendors stated that meeting Level A  requirements may require either new technology or a combination of  current technologies instead of only using an EDS.
Although TSA and vendor officials expressed confidence that deployed  EDSs can be upgraded, TSA officials also confirmed that the agency has  never previously upgraded the detection software of deployed EDS or  ETD machines to meet revised explosives detection requirements.  Additionally, even though TSA has estimated that it will take a number of   years to certify new EDSs to fully meet Levels B and A of the 2010  requirements, TSA has not yet developed similar time frames to upgrade  deployed equipment. Given the number of unknowns associated with  upgrading EDSs, it is unclear how long it will take the agency to upgrade  deployed EDSs to meet Levels C, B, and A of the 2010 requirements.  Furthermore, TSA has identified the EDS upgrade effort as a high  program risk. Consequently, TSA and vendor officials’ confidence that it  will be feasible and cost effective to upgrade deployed machines at  airports may be unwarranted as it has not been based on experience,  supported by analysis, or a documented plan.
section_title: Conclusions
paragraphs: TSA faces a complex task in its efforts to address explosives threats in its  current and future procurements and existing fleet of checked-baggage- screening systems. The complexity of this task is amplified when taking  into account the large volume of checked baggage that TSA must screen  for explosives without disrupting commerce. TSA’s plan to procure and  deploy EDSs that meet the 2010 requirements in a phased approach that  spans a number of years is aimed at allowing more time to collect  necessary explosives data, test key technologies, and provide a means  for TSA to continue to purchase EDSs to meet its needs for new checked- baggage-screening equipment at the nation’s commercial airports.  However, TSA officials recognized that if TSA deploys EDS capable of  detecting all explosives included in the 2010 EDS requirements, TSA  must ensure that ETD machines are capable of detecting all of the  explosives that EDSs will be able to detect to minimize any potential  screening difference between the EDS and ETD. Without a plan to help  ensure that additional screening devices or protocols are in place to  resolve EDS alarms if EDSs are deployed that detect a broader set of  explosives than existing ETD machines used to resolve EDS screening  alarms, it will be difficult for TSA to provide reasonable assurance that a  potential capability gap has been resolved.
By separating the effort to collect data needed to meet the 2010 EDS  requirements from the related competitive procurement, TSA would have  more time to identify the physical and chemical properties of the  explosives, collect full threat weight data, provide vendors with the time  needed to develop detection software, and attempt to pass CRT and  certification testing without the added pressure of an acquisition deadline.  TSA also faces additional challenges related to the agency’s plans for  implementing the current EDS procurement. For example, the lack of  timely communication with vendors may impact vendors’ abilities to  ensure they can meet TSA’s needs for the current EDS acquisition. By  establishing a process to communicate with vendors in a timely manner,  TSA could help ensure that vendors have the information necessary to  meet TSA’s needs for new checked-baggage-screening equipment.  Moreover, by addressing challenges related to planning for the  acquisition, TSA may be able to better avoid further delays and potential  cost overruns for the current procurement. Specifically, completing a  reliable IMS that fully meets the nine best practices could help DHS and  TSA management to predict whether the estimated acquisition completion  date is realistic and manage program performance. Once a reliable  schedule is in place, TSA can in turn revise current cost estimates for the  program to better reflect actual acquisition costs including, for example,  the potential cost impacts resulting from schedule slippage to give  program decision-makers a more accurate and comprehensive view of  current and projected program costs.
As TSA plans to deploy EDSs that meet the 2010 requirements, it is  critical that TSA plans its approach to ensure that all airports with EDS  equipment are capable of detecting the required explosives. Because  TSA has not yet upgraded most of the deployed EDSs to meet certain  requirements, many EDSs are only capable of detecting certain  explosives. Moreover, of the EDSs currently deployed, TSA is currently  operating some number of them at the capability needed to detect the  explosives identified in the 2005 requirements, although activating the  software and operationally testing the machines to detect the 2005  requirements would help address this issue.
As part of TSA’s phased approach to meet the 2010 EDS requirements,  TSA may have to upgrade many of its currently deployed EDSs and  hundreds of newly purchased EDSs over a period of years, upgrades that  may require significant investments in new technologies to help meet  more stringent explosives detection requirements. However, until TSA  develops a plan identifying how it will approach the upgrades for currently  deployed EDSs—and the plan includes such items as estimated costs,  the number of machines that can be upgraded, time frames for upgrading  them, and the number of times a given machine must be upgraded to  meet the 2010 EDS requirements—it will be difficult for TSA to provide  reasonable assurance that its upgrade approach is feasible or cost- effective.
section_title: Recommendations for Executive Action
paragraphs: To help ensure that TSA takes a comprehensive and cost-effective  approach to the procurement and deployment of EDSs that meet the  2010 EDS requirements and any subsequent revisions, we recommend  that the Assistant Secretary for TSA take the following six actions:    Develop a plan to ensure that screening devices or protocols are in  place to resolve EDS alarms if EDSs are deployed that detect a  broader set of explosives than existing ETD machines used to resolve  EDS screening alarms.    Develop a plan to ensure that TSA has the explosives data needed for  each of the planned phases of the 2010 EDS requirements before  starting the procurement process for new EDSs or upgrades included  in each applicable phase.    Establish a process to communicate information to EDS vendors in a  timely manner regarding TSA’s EDS acquisition, including information  such as changes to the schedule.    Develop and maintain an integrated master schedule for the entire  Electronic Baggage Screening Program in accordance with the nine  best practices identified by GAO for preparing a schedule.    Ensure that key elements of the program’s final cost estimate reflect  critical issues, such as the potential cost impacts resulting from  schedule slippage identified once an integrated master schedule for  the Electronic Baggage Screening Program has been developed in  accordance with the nine best practices identified by GAO for  preparing a schedule.    Develop a plan to deploy EDSs that meet the most recent EDS  explosives-detection requirements and ensure that new machines, as  well as machines deployed in airports, will be operated at the levels  established in those requirements. This plan should include the  estimated costs for new machines and upgrading deployed machines,  and the time frames for procuring and deploying new machines and  upgrading deployed machines.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DHS on June 23, 2011, for review  and comment. On July 6, 2011, DHS provided written comments, which  are presented in appendix IV. We also provided relevant excerpts of our  draft report to DOD and the Department of Energy for review and  comment. In commenting on our report, DHS stated that it agreed with  our six recommendations and identified actions planned or under way to  implement them. DOD provided written technical comments and the  Department of Energy provided technical comments in an e-mail. Both  stated that the draft report excerpts related to their respective agencies  contained accurate information.
Overall, DHS stated that, because of the urgent need to meet ongoing  requirements, TSA began addressing many of the issues identified by this  audit while the audit was being conducted. However, as DHS noted in its  letter, TSA still needs to complete many actions to resolve the issues  identified in this report.  Additionally, TSA stated that it suspended the  implementation of the 2005 requirements because of the computer  modeling effort known as “Project Newton” and then issued the 2010  detection standards when Project Newton did not yield timely results.
Thus, in its comments, TSA confirmed that it is using some number of  EDSs that meet requirements established in 1998 by the FAA, as we  reported, an approach that raises questions about how well some of its  deployed equipment detects current explosives threats.
In addition, DHS stated that TSA is currently taking steps to collect the  operational data necessary to support the upgrade of deployed  equipment based on 2010 detection standards and that the operational  data-collection effort is to be completed in 2011. However, as discussed  in the report, this could be a difficult endeavor as TSA is still in the  process of conducting operational testing to determine the staffing  implications of operating EDSs that meet 2005 explosives detection  requirements. Therefore, it will have taken TSA 6 years from the time that  the 2005 EDS explosives detection requirements were issued until this  operational testing is to be completed. Furthermore, if the results of the  operational testing show that operating EDS machines, to meet the 2005  requirements, will require additional TSA staff and/or slow down the rate  of checked-baggage screening, TSA may have to make difficult decisions  and trade-offs that could affect aviation security and commerce, and also  affect the schedule for meeting the 2010 requirements.
DHS concurred with our first recommendation to develop a plan to ensure  that screening devices or protocols are in place to resolve EDS alarms if  EDSs are deployed that detect a broader set of explosives than existing  ETD machines used to resolve EDS screening alarms. DHS stated that  TSA convened a working group to assess capability gaps for secondary  screening technology, evaluate current technology capabilities against the  capabilities of future EDSs, and prepare a plan to procure any additional  technology required to ensure alarms can be resolved. DHS expects this  plan to be finalized by the fourth quarter of fiscal year 2012. While these  actions and planned actions represent positive steps to fully implement  the recommendation, TSA should develop a plan to ensure that screening  devices or protocols are in place to resolve EDS alarms if EDSs are  deployed that detect a broader set of explosives than existing ETD  machines used to resolve EDS screening alarms.
DHS concurred with our second recommendation to develop a plan to  ensure that TSA has the data needed for each of the planned phases of  the 2010 EDS requirements before starting the procurement process for  new EDSs or upgrades included in each applicable phase. DHS  commented that TSA modified its strategy for the EDS’s competitive  procurement in July 2010 in response to the challenges in working with  the explosives for data collection and alerted the vendor community on  September 3, 2010. DHS stated that the new baseline schedule removed  data collection from the acquisition process. Additionally, DHS stated that  TSA is working with DHS S&T to establish a laboratory by summer 2011  to support further data collection and independent test and evaluation.  Although these actions respond in part to the intent of our  recommendation, separating data collection from the acquisition process  does not necessarily ensure that the needed data will be available before  starting the procurement process for the new EDSs or upgrading currently  deployed EDSs. Consequently, we continue to believe that, to fully  address our recommendation, a plan is needed to establish a process for  ensuring that data are available before starting the procurement process  for new EDSs or upgrades for each applicable phase. Developing and  following such a plan would assist TSA in implementing the acquisition  and making upgrades in an efficient and effective manner and would  benefit DHS in its oversight role of TSA by allowing DHS to determine  progress against the plan.
DHS concurred with our third recommendation to establish a process to  communicate information to EDS vendors in a timely manner regarding  TSA’s EDS acquisition, including information such as changes to the  schedule. In the letter, DHS stated that TSA has a process for  communicating information to the vendor community and will continue to  follow this process in adherence with guidelines outlined in the Federal  Acquisition Regulation. DHS also stated that TSA significantly changed  the business model to procure checked-baggage-screening equipment  from what has historically been a sole-source environment to a  competitive environment, resulting in significant improvements in  communication with industry. In addition, according to DHS, TSA has  already made a number of efforts to improve the quality and frequency of  communication with industry, but TSA recognizes the complexity  associated with many of the acquisitions currently ongoing. As such, TSA  acknowledged that there are opportunities to continue to improve  communication with the vendor community and will take steps to ensure  that vendors are provided with the most current information possible in an  efficient manner. Since the agency did not provide us with evidence of  how it plans to ensure more timely and effective communications with  vendors in the future, we continue to believe that such a process is  needed to ensure that TSA officials are aware of the specific guidelines to  follow to communicate with vendors about current and future acquisitions.  Our meetings with vendors indicated that TSA’s communications with  them continue to leave room for improvement.
DHS concurred with our fourth recommendation to develop and maintain  an IMS for the entire EBSP in accordance with the nine best practices  identified by GAO for preparing a schedule. DHS commented that TSA  has already begun working with key stakeholders to develop and define  requirements for an IMS and to ensure that the schedule aligns with the  best practices outlined by GAO. DHS stated that this effort is expected to  be completed by the second quarter of fiscal year 2012. In addition, DHS  stated that, as the program matures and increases its focus on flexible  and upgradeable technology, an IMS will ensure close coordination  among the program’s procurement, deployment, recapitalization, and  upgrade capabilities, and that the EBSP IMS will be updated as a result of  these efforts to be in accordance with the nine best practices. While these  actions and planned actions are steps toward implementing our  recommendation, to fully implement the recommendation, TSA needs to  develop and maintain an IMS for the entire EBSP in accordance with the  nine best practices identified by GAO for preparing a schedule.
DHS concurred with our fifth recommendation to ensure that key elements  of the program’s final cost estimate reflect critical issues, such as the  potential cost impacts resulting from schedule slippage. Such a slippage  might be identified once an IMS for the EBSP has been developed in  accordance with the nine best practices identified by GAO for preparing a  schedule. DHS stated that TSA is working to update the EBSP LCCE to  incorporate cost estimates associated with enhanced detection, work that  should be completed in the fourth quarter of fiscal year 2011. DHS also  stated that, per the recommendations of GAO and DHS, TSA is developing  a master schedule to document timelines associated with various projects.  DHS further stated that risks to the costs and schedules will be analyzed  and that the risk analysis will produce confidence intervals for the life-cycle  costs to the program. Although TSA discussed activities to address the  EBSP LCCE, to fully implement this recommendation, it will be important  that key elements of the program’s final cost estimate reflect critical issues,  such as the potential cost impacts resulting from schedule slippage  identified once an IMS for the EBSP has been developed in accordance  with the nine best practices.
DHS concurred with our sixth recommendation to develop a plan to  deploy EDSs that meet the most recent EDS explosives detection  requirements and ensure that new machines, as well as machines  deployed in airports, will be operated at the levels established in those  requirements. This plan should include the estimated costs for new  machines and upgrading deployed machines, and the time frames for  procuring and deploying new machines and upgrading deployed  machines. DHS commented that TSA has a plan in place to evaluate and  implement the most recent certified algorithms on the existing fleet of  deployed EDSs, assuming the evaluation results in minimal to no  operational impact. In contrast, our recommendation calls for a plan to  deploy new EDSs as well as to upgrade existing EDSs in airports to meet  the 2010 EDS explosives detection requirements and, importantly, ensure  that new machines will be operated at the levels established in those  requirements. As we discussed in the report, some number of the EDSs  in airports are operating at a level that meets the 2005 explosives  detection requirements. Our recommendation is intended to ensure that  TSA operates all EDSs in airports to meet the most recent requirements,  which are currently the 2010 requirements. Consequently, we continue to  believe that a plan is needed describing the approach that TSA will use to  deploy EDSs that meet the most recent EDS explosives detection  requirements and ensure that new machines, as well as machines  deployed in airports, will be operated at the levels established in those  requirements.
TSA also provided written technical comments, which we incorporated in  the report, as appropriate.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 10 days from the  report date. At that time, we will send copies of this report to the  Secretary of Homeland Security, the Assistant Secretary of the  Transportation Security Administration, and appropriate congressional  committees. This report also will be available at no charge on the GAO  Web site at http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  me at (202) 512-8777 or LordS@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report discusses: (1) the extent to which TSA has revised explosives  detection requirements and deployed EDSs and ETDs to meet these revised  requirements; (2) any challenges that TSA and the Department of Homeland  Security’s (DHS) Science and Technology Directorate (S&T) have  experienced in implementing the EDS acquisition; and (3) the extent to which  TSA’s approach to its EDS acquisition meets best practices for schedule and  cost estimates, and includes plans for potential upgrades to deployed EDSs.
To determine the extent to which TSA has revised explosives detection  requirements for checked baggage screening, we reviewed TSA’s EDS  explosives detection requirements for checked baggage screening and  assessed the extent to which the 2010 detection requirements differed from  the 2005 detection requirements for EDSs. We compared specific  explosives 1998, 2005, and 2010 detection requirements to identify  commercial and homemade variants of the explosives. We also identified,  analyzed, and discussed with TSA and S&T officials the differences  between the tiers and multiple levels of explosives detection requirements  in the 2010 EDS explosives detection requirements. We discussed with  TSA officials the 2002 explosives detection requirements for the ETD and  reviewed the 2006 explosives detection requirements for ETD. We also  compared the 2010 EDS requirements with the 2006 ETD requirements  and discussed with TSA and S&T officials the differences between the  explosives detection requirements for the EDS and ETD. We discussed  TSA’s Standard Operating Procedures for resolving EDS and ETD alarms  with TSA officials. Finally, we visited three of the Department of Energy’s  national laboratories, Lawrence Livermore National Laboratory, Los  Alamos National Laboratory, and Sandia National Laboratories, to  determine the status of Project Newton. (For more on Project Newton, see  app. II.)
To identify any challenges that TSA has experienced implementing the  EDS acquisition, we reviewed documentation from TSA’s Electronic  Baggage Screening Program (EBSP), the program responsible  operational testing, procurement, deployment, and maintenance of  checked-baggage-screening technologies. Among other things, we  reviewed available program documentation on the status of its EDS  acquisition including EBSP strategic plans from previous years as well as  the most recent EBSP strategy approved in July 2010. We also reviewed  documentation from the program’s first Acquisition Review Board review,  the EBSP risk management plan, the most recent procurement  specifications for the EDS, information posted by EBSP for interested  vendors on FedBizOpps.gov, and DHS acquisition guidance and  directives. We also interviewed EBSP program officials, including the  EBSP program manager, regarding the program’s approach to the current  EDS acquisition and received updates on revisions to the program’s EDS  acquisition strategy and timelines for the current procurement.
To further understand the challenges TSA and S&T face in preparing for  new EDSs to meet revised detection requirements, we reviewed  documentation provided by TSA outlining the agency’s plan for deploying  EDSs that meet the 2010 requirements as well as documentation regarding  S&T’s approach to testing and certification carried out at the Transportation  Security Laboratory (TSL). We also conducted interviews with TSA and  S&T officials and conducted site visits to the TSL in Atlantic City, New  Jersey, and to the Air Force Research Lab (AFRL) facility at Tyndall Air  Force Base (AFB), Florida, to obtain information on efforts to certify EDSs  that meet the 2010 requirements. We visited the TSL because that is  where S&T tests and evaluates transportation technologies including  checked baggage screening technologies. We visited AFRL because they  are assisting TSL in their efforts to collect data regarding the physical and  chemical properties of explosives included in the 2010 EDS requirements  in preparation to certify EDSs for the current procurement. Additionally, we  conducted site visits and/or telephone interviews with all six EDS vendors  competing in the first phase of the current EDS procurement. These  vendors were able to provide us with an understanding of their companies’  views regarding TSA’s approach to the current procurement as well as  potential challenges they believe vendors face in preparing to compete for  the current EDS procurement. While information we obtained from these  interviews may not be generalized across the industry as a whole, we were  able to obtain the perspectives of all companies planning to compete for  the current EDS procurement, and they were able to provide an  understanding of their companies’ abilities to develop EDSs that meet the  2010 requirements. We also reviewed TSA documentation to identify the  explosives detection technologies that are used for checked baggage  screening. Additionally, we interviewed TSA and S&T officials to identify the  number of currently-deployed explosives detection machines that meet the  previous and most recent detection requirements, and found the data for  the number of machines to be sufficiently reliable.
To determine the extent to which TSA’s approach to its EDS acquisition  meets best practices for schedule and cost estimates and includes plans  for potential upgrades to deployed EDSs, we determined the extent to  which TSA had established an integrated master schedule (IMS) for the  EBSP, and due to the lack of an IMS, assessed the EDS acquisition  schedule against nine best practices in our Cost Estimating and  Assessment Guide. We conducted this assessment to determine the extent  to which the schedule reflects key estimating practices that are  fundamental to having and maintaining a reliable schedule. In doing so, we  independently assessed the schedule for the current EDS acquisition and  its underlying activities against our nine best practices, as provided to us in  July 2010. We subsequently interviewed cognizant program officials to  discuss their use of best practices in creating the schedule and to discuss  the findings resulting from our review of the schedule. After TSA revised  the schedule to reflect changes in some of the timelines and provided it to  us in October 2010, we reviewed the updated schedule and compared it to  information in the original schedule in order to understand how the new  schedule was constructed and to determine to what extent TSA had  resolved weaknesses that we identified in its original schedule. We also  assessed the schedule against relevant best practices in our Cost  Estimating and Assessment Guide to determine the extent to which it  reflects key estimating practices that are fundamental to having a reliable  schedule. We compared TSA’s efforts with internal control standards and  recommended practices we previously identified for sound acquisition  planning.
To further evaluate TSA’s planning for the current EDS acquisition, we  also interviewed TSA, S&T, and EDS vendors’ officials to identify any  challenges involved and expected in upgrading EDS detection  capabilities, and TSA’s plans to upgrade equipment to meet future  implementations of the 2010 EDS requirements. Also, during our site  visits and telephone interviews with the six vendors, as discussed  previously, vendors provided their perspectives on TSA’s approach to  upgrade currently deployed EDSs as well as those to be deployed in the  future. We also obtained from the six vendors their perspectives on how  upgrades to deployed EDSs might be accomplished and potential costs  involved in performing the upgrades.
We conducted this performance audit from September 2009 through May  2011 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. During the  course of our review, we revised the engagement objectives and scope to  facilitate a broader examination of TSA’s efforts to revise its explosives  detection requirements and related schedule for the EDS acquisition,  which increased the time for completing this audit. We believe that the  evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Department of Homeland Security Efforts to Conduct Computer Modeling to Establish a Tier of the Explosives Detection Requirements for the Explosives Detection System
paragraphs: The Department of Homeland Security (DHS) Science and Technology  Directorate (S&T) and Transportation Security Administration (TSA)  began a project in 2007, known as Project Newton, to identify the  minimum mass of explosives that could cause catastrophic damage to an  aircraft. Through 2009, S&T and TSA had invested approximately $12.5  million in Project Newton modeling activities, according to a senior TSA  official. A different senior TSA official stated that TSA allocated an  additional $2.5 million to $3.1 million for Project Newton as of August  2010: $1.0 million to $1.6 million for incremental development of  computer models and $1.5 million to develop a plan to validate the  models. As part of the effort to understand the effects of explosives  detonations on aircraft, S&T and TSA have been working to simulate the  complex dynamics of explosive blast effects on an in-flight aircraft by  using computer models at three Department of Energy national  laboratories—Lawrence Livermore National Laboratory, Los Alamos  National Laboratory, and Sandia National Laboratories. According to TSA  officials, the current understanding of the effects of explosives on aircraft  has been largely based on data from live-fire explosives tests conducted  with retired aircraft hulls at ground level. These officials stated that,  compared to running a computer simulation, live-fire tests can be more  expensive, which limits the number of live-fire tests conducted and,  therefore, the amount of data available for analysis. S&T, TSA, and  national laboratory officials stated that computer modeling can cost- effectively simulate the effects of explosives detonations in various  locations of different types of aircraft at ground level, which can provide  significant data for analysis.
In January 2010, TSA revised the explosives detection requirements for  the explosives detection system (EDS) and established tiers of explosives  that are required to be detected. One tier of requirements is to be  implemented over a number of years. TSA plans to incorporate the  computer-modeling results into the requirements for a subsequent tier.  Although TSA expected that the computer-modeling results would be  used to revise EDS explosives detection requirements as early as 2012,  as of December 2010, TSA officials were uncertain when the computer- modeling results will be used for this purpose because the computer  models had not been validated. In 2009, TSA established a Blue Ribbon  Panel to, among other things, assess the three national laboratories’  computer models and their results and comment on whether they were  valid to be used to revise explosives detection requirements. The panel  members included DHS and TSA officials as well as officials from  academia and the private sector. In March 2010, the Blue Ribbon Panel  recommended that, among other things, before the computer modeling  results are used to revise EDS explosives detection requirements, the  computer models and their results should be validated, according to a  senior TSA official. This official stated that the panel also recommended  specific locations to add to the computer models, so that the models can  simulate the effects of explosives detonations in those additional locations  on the aircraft. Validating the computer models and their results is  essential before relying on them to revise explosives detection  requirements. A senior TSA official stated that it will take a number of  months to validate the computer models, validation that is expected to be  completed later in 2011.
section_title: Appendix III: GAO’s Assessment of the Extent to Which the Electronic Baggage- Screening-Program Schedule Meets Established Best Practices
paragraphs: In determining the extent to which the Transportation Security  Administration’s (TSA’s) Electronic Baggage Screening Program (EBSP)  schedule meets established best practices, we identified that TSA did not  have an integrated master schedule (IMS) for the program. As a result,  we assessed the explosives detection system (EDS) acquisition schedule  against each of nine best practices. Specifically, we assessed TSA’s  initial schedule, which was provided to GAO in July 2010, and met with  TSA officials to discuss our assessment and provided officials with  suggestions for corrective action. TSA later revised the schedule and  provided GAO with an updated version in October 2010. We completed a  separate assessment of TSA’s revised schedule. The following table  presents our two assessments.
section_title: Appendix IV: Comments from the Department of Homeland Security
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Glenn Davis, Assistant Director,  and Joseph E. Dewechter, Analyst-in-Charge, managed this assignment.  Scott Behen, Samantha Carter, and Orlando Copeland made major  contributions to the planning and all other aspects of the work. David  Alexander and Richard Hung assisted with design, methodology, and  data analysis. Jason Lee and Karen Richey assisted with acquisition cost  and schedule analysis. John Hutton and Nathan Tranquilli assisted with  acquisition and contracting issues. Katherine Davis provided assistance  in report preparation. Thomas Lombardi and Tracey King provided legal  support.