Aviation Security: TSA Has Enhanced Its Explosives Detection Requirements for Checked Baggage, but Additional Screening Actions Are Needed
Released: 2011-07-12T00:00:00

Letter

Background

The Roles of TSA and S&T After the September 11, 2001, terrorist attacks, the Aviation and

Transportation Security Act (ATSA) was enacted. Among other things, ATSA required that TSA provide for the screening of all checked baggage for explosives transported on flights departing U.S. commercial airports. Pursuant to ATSA, TSA deployed EDS and ETD equipment to screen checked baggage and identify potential threats from explosives. While TSA is responsible for operating or overseeing the operation of checked- baggage-screening equipment, TSA and S&T share responsibilities for the research and development of checked-baggage-screening technologies. During fiscal year 2006, most research and development functions within DHS, including TSA, were consolidated within S&T. After this consolidation, S&T assumed primary responsibility for the research, development, and related test and evaluation of airport checked- baggage-screening technologies. S&T also assumed responsibility from TSA for the TSL, which tests and evaluates technologies under development against TSA-established detection requirements. TSA continues to be responsible for: identifying the requirements for new checked-baggage-screening technologies; operationally testing and evaluating technologies in airports; and procuring, deploying, and maintaining technologies. TSA relies on S&T as the central coordination point to manage all work related to explosives that involve the TSL, Lawrence Livermore National Laboratory, Los Alamos National Laboratory, Sandia National Laboratories, and the Air Force Research Laboratory at Tyndall AFB.

Deploying EDSs and ETDs According to TSA, since fiscal year 2001, TSA has made over $8 billion available to EBSP for activities related to checked-baggage screening. TSA uses two types of technology for checked baggage screening—the EDS and the ETD—at 462 U. S. commercial airports. EDS is used to identify suspicious bulk items or anomalies in checked baggage that could be explosives or detonation devices. In airports that have EDS, it is used for primary screening of checked baggage while ETD machines are used for secondary screening of checked baggage to help resolve EDS alarms. Additionally, at airports without EDS, the ETD machines are used for primary screening of checked baggage. See figure 1 for a photograph of an EDS machine and figure 2 for a photograph of an ETD machine.

TSA deploys EDSs in multiple configurations, such as an in-line configuration and a stand-alone configuration. The in-line configuration integrates EDS with an airport’s baggage handling system—the conveyor system that sorts and transports baggage for loading onto an aircraft. EDS in stand-alone configurations are separate baggage screening units that are not integrated with a baggage-handling system and are typically located in an airport lobby although they may also be located in other airport locations. Checked baggage is manually loaded and unloaded on stand-alone EDS machines.

As of October 2010, TSA had 2,297 EDS machines in its fleet, 1,938 of which were deployed at airports in the United States. At airports and terminals that do not use EDSs, ETD machines are used for primary checked-baggage screening. Typically, ETDs are used for primary screening of checked baggage at smaller airports. These airports typically do not have EDSs for primary screening of checked baggage. As of February 2011, TSA estimated that there were about 5,200 ETD machines used for the primary or secondary screening of checked baggage at U.S. commercial airports.

S&T and TSA’s Certification and Testing Process

TSA certifies the EDS it deploys to commercial airports for screening checked baggage, based on tests performed by the TSL. Specifically, TSA certifies that EDSs, alone or as part of an integrated system, can detect, under realistic operating conditions, the amounts, configurations, and types of explosive material which would be likely to be used to cause catastrophic damage to an aircraft, using requirements developed in consultation with experts from outside TSA. Furthermore, TSA periodically reviews threats to civil aviation security, including:  explosive material that presents the most significant threat to civil the minimum amounts, configurations, and types of explosive material that can cause, or would be expected to cause, catastrophic damage to aircraft in air transportation; and the amounts, configurations, and types of explosive material that can be detected reliably by existing or reasonably anticipated, near-term explosive detection technologies.

Currently, TSA requires that EDSs undergo three types of testing— certification testing, integration testing, and operational testing—before it will purchase such equipment. First, TSA verifies that vendors’ explosives detection systems meet—that is, are capable of detecting in accordance with—the TSA established explosives detection requirements through the certification testing process. TSA’s decision to certify an EDS relies on the results of independent test and evaluation performed at the TSL. Prior to certification testing, TSL conducts preliminary evaluations of vendors’ EDS, known as certification readiness testing (CRT) and pre-certification, to determine the extent to which vendors are ready to enter certification testing. During CRT, TSL provides feedback to vendors on their EDS’s strengths and weaknesses in detecting explosives in order to help vendors make necessary adjustments to their detection software. Second, in addition to being certified that the EDS can meet explosives detection requirements, EDSs being deployed in an in-line configuration must also undergo integration testing. As part of this testing, machines deployed in an in-line configuration must demonstrate in a controlled environment that they can be successfully integrated within the baggage- handling systems used for checked baggage. Finally, following certification and integration testing, EDSs undergo operational testing in an airport setting to demonstrate that they can reliably and effectively function in a live airport environment.

TSA Has Revised Explosives Detection Requirements for Checked-Baggage- Screening Systems, but Faces Challenges in Deploying Equipment to Meet the Requirements

In 2005, TSA revised explosives detection requirements for the EDS; however, some number of the EDSs are currently operating at the levels to detect explosives as set forth only in the 2005 requirements. When TSA established the 2005 requirements, it did not have a plan that identified the appropriate time frames needed to deploy EDSs to meet the requirements. In January 2010, TSA again revised the EDS explosives detection requirements and plans to deploy EDSs meeting these requirements in a tiered and phased approach over a number of years. One tier of requirements consists of three levels and expanded the number and types of explosives that EDSs must detect. TSA is in the process of developing another tier of requirements, which will refine the amount (for example, minimum mass) of an explosive that can cause catastrophic damage to an aircraft. If TSA deploys EDSs that fully meet the one tier of the requirements, TSA must ensure that ETD machines are capable of detecting all of the explosives that EDSs will be able to detect to minimize any potential screening difference between the EDS and ETD.

TSA Did Not Establish a Plan to Ensure That Currently Deployed EDSs Meet the 2005 Requirements

In November 2005, TSA revised its explosives detection requirements for EDS that had previously been established in 1998 by the Federal Aviation Administration. However, as of January 2011, some number of the EDSs in TSA’s fleet are configured to detect explosives at the levels established only in the 2005 requirements. The remaining EDSs are configured to detect explosives at 1998 levels. When TSA established the 2005 requirements, it did not have a plan with the appropriate time frames needed to deploy EDSs to meet the requirements. Standard practices for program and project management state that specific desired outcomes or results should be conceptualized, defined, and documented in the planning process as part of a road map, along with the appropriate steps, time frames, and milestones needed to achieve those results.

Despite the absence of a plan, TSA officials stated that they must conduct testing to compare the false alarm rates for machines operating at one level of requirements to those operating at another level of requirements. According to TSA officials, the results of this testing would allow them to determine if additional staff are needed at airports to help resolve false alarms once the EDSs are configured to operate at a certain level of requirements. TSA officials reported that they had anticipated this operational testing was to be completed in March 2011.

According to agency officials, TSA did not begin this operational testing immediately after the previous explosives detection requirements were established in November 2005 because the agency officials were aware at the time of a potential further revision of the requirements based upon a planned computer modeling effort to revise the detection standards that became known as Project Newton. TSA and S&T officials told us they had planned to use the results from Project Newton to further revise the explosives detection requirements to reflect the mass of an explosive that would cause catastrophic damage to an aircraft. Although Project Newton did not begin until 2007, TSA officials told us that they were aware of plans to further revise the requirements prior to the initiation of Project Newton and delayed operational testing in anticipation of the results of the computer modeling effort. As of April 2011, the EDS explosives detection requirements have not been changed based on results of the computer modeling because Project Newton is still under way, though TSA officials told us that they plan to use the results of Project Newton to later define a tier of the 2010 EDS requirements. (We discuss the status of Project Newton in more detail in app. II.)

However, once it became apparent that the results of Project Newton would not become available to further revise the requirements, TSA did not establish a plan with time frames for completing the additional testing related to staffing. Standard practices for planning and project management suggest TSA should have defined the operational testing plan and milestones as part of a road map for assessing potential staffing changes when these EDSs were first deployed after the 2005 requirements were established. Establishing reasonable time frames to complete the operational testing could help TSA ensure it achieves its desired goal of activating EDSs capable of detecting the explosives established in the 2005 requirements in a timely manner.

TSA Plans to Deploy EDSs That Meet the 2010 Explosives Detection Requirements Using a Phased Approach

In January 2010, TSA revised the EDS checked-baggage explosives detection requirements partly in response to credible and immediate threats to civil aviation. TSA plans to meet the 2010 EDS requirements using a tiered and phased approach: one tier is to be implemented over a number of years and expands the types of explosives that EDSs must detect, and another tier adds the use of the results of the computer- modeling effort known as Project Newton. TSA is not certain when this tier of requirements will be implemented. TSA plans to deploy EDSs that meet one tier of the 2010 requirements in a phased approach beginning in late fiscal year 2011 as part of its planned EDS acquisition. Regarding the other tier, TSA officials stated that Project Newton is to enhance TSA’s understanding of explosives effects by simulating hundreds of explosives tests using computer modeling to determine the effects explosives will have when placed in different locations within an aircraft. TSA’s and S&T’s understanding of how explosives affect aircraft has largely been based on data obtained from live-fire explosive tests on retired aircraft hulls and other data. Project Newton is jointly managed and funded by DHS’s S&T and TSA. Through fiscal year 2009, S&T and TSA had invested about $12.5 million in national laboratories for computer modeling activities as part of Project Newton, according to a senior TSA official. We discuss Project Newton and its budget in more detail in appendix II.

TSA plans to implement one tier of requirements in a phased approach that consists of three levels (see fig. 3). As our past work has shown, an incremental or phased approach to implementing requirements can reduce risk and make a program more achievable by providing more time to develop and test key technologies. TSA officials told us that the ability to develop EDS is likely to become increasingly complex as the implementation of requirements progresses. According to TSA, it expects to begin procuring EDSs to meet the 2010 requirements in July 2011. Consequently, if TSA is successful in deploying EDSs that meet all three levels of the 2010 requirements, TSA’s EDS fleet would be certified to detect more explosives than a fleet meeting the 1998 or 2005 requirements.

ETD machines are currently certified to detect some different explosives than those EDSs that meet the 2005 EDS detection requirements. However, if TSA purchases and deploys EDSs that fully meet Level C of the 2010 EDS requirements, ETD machines are not required to detect all of the explosives that can be detected by EDSs. TSA’s existing ETD explosives detection requirements identify the types and quantities of explosives materials, that is, traces of explosives, that must be detected and the minimum detection rate for each category of explosive.

According to TSA officials, the ETD explosives detection requirements have not been revised because TSA wanted to first focus on revising the EDS explosives detection requirements in time for its planned EDS acquisition, which is aimed at replacing and upgrading its fleet of EDSs used to screen checked baggage. TSA officials stated that they are developing a combined set of explosives detection requirements that could eventually result in the ETD and EDS machines detecting the same explosives. TSA officials stated that the combined set of EDS and ETD requirements would not be expected to be approved until sometime in calendar year 2011. Although combined detection requirements for ETD and EDS are to help ensure that both machines can detect the same explosives, the machines are not expected to be required to detect the same amounts of explosives because the purpose of the ETD is to detect traces of explosives in nanograms while the EDS is designed to detect larger amounts of explosives.

At all airports that use EDSs to screen checked baggage, ETD machines are used in conjunction with EDSs to screen checked baggage for explosives. At these airports, if an EDS alarms—indicating that checked baggage may contain an explosive or explosive device that cannot be cleared—ETD machines are used as a secondary screening device in order to attempt to resolve the alarm. However, the differences between the EDS and ETD requirements may impact the resolution of EDS alarms by the ETD in the future. According to TSA’s 2010 EBSP Acquisition Strategy, additional equipment—other than the ETDs currently deployed—is to be employed to support alarm resolution when EDSs that meet the new checked-baggage explosives detection requirements are deployed. However, the acquisition strategy does not specify what additional equipment or screening protocols will be employed to resolve alarms nor does the strategy discuss whether TSA will continue to use ETD equipment to resolve EDS alarms. According to TSA, the agency is currently evaluating what additional technologies and/or changes to screening protocols may be needed to address any potential gap in capability between the newly certified EDSs and ETDs used for alarm resolution. If TSA begins operating EDSs that detect explosives in subsequent phases of the 2010 requirements, this potential screening difference between EDS and ETD will exist until TSA deploys additional equipment and/or implements new screening protocols that could be used for secondary screening.

In commenting on this issue, TSA officials stated that checked-baggage- screening technologies are only one layer of security and that other layers of security exist to help address potential threats to the aviation security system. However, officials agreed that they have not yet developed new screening protocols or deployed additional equipment that will address the potential gap in screening capability between EDS and ETD if the new EDSs are deployed. Standards for program management require that specific desired outcomes or results be conceptualized, defined, and documented in the planning process as part of a road map, along with the appropriate steps and time frames needed to achieve those results. Because TSA decided to revise explosives detection requirements for EDSs prior to revising the ETD requirements, the differences in the requirements may affect TSA’s capability to detect the 2010-required levels until TSA identifies technologies or protocols needed to address the potential gap. Without a plan to ensure that secondary-screening devices or protocols are in place to resolve EDS alarms if EDSs are deployed with additional capability, it will be difficult for TSA to provide assurances that the potential capability gap has been resolved.

TSA Has Faced Challenges in Implementing Plans for the Current EDS Acquisition

TSA has developed an EBSP acquisition strategy to guide its efforts to improve its fleet of checked-baggage-screening machines, but has faced several challenges in implementing plans for the current EDS acquisition under this strategy. First, TSA has experienced challenges in collecting explosives data on the physical and chemical properties of certain explosives needed by vendors to develop EDS detection software and needed by TSA before procuring and deploying EDSs to meet the 2010 requirements. TSA and S&T have experienced these challenges because of problems associated with safely handling and consistently formulating some explosives. Second, the challenges related to data collection for certain explosives have resulted in problems carrying out the EDS procurement as planned. Specifically, attempting to collect data for certain explosives while simultaneously pursuing the EDS procurement has delayed the EDS acquisition schedule by at least 7 months. Finally, EDS vendors have expressed concerns about the extent to which TSA is communicating with the business community about the current EDS procurement.

TSA’s EBSP Acquisition Strategy Is Being Used to Guide Efforts to Improve the Explosive Detection Capabilities of the EDS

In July 2010, DHS approved TSA’s current acquisition strategy for the EDS, and under this strategy, TSA plans to increase the threat detection capabilities of the EDS using a competitive procurement to purchase and deploy EDS beginning in 2011. According to TSA officials, most of the previous EDS acquisitions were sole source procurements. However, TSA is implementing a competitive procurement for the current EDS acquisition to, in part, meet the EBSP acquisition strategy’s goals and objectives., Furthermore, the EBSP acquisition strategy calls for acquiring new EDSs as part of the recapitalization plan to replace aging EDS. Under the current EDS procurement, TSA plans to award contracts to purchase 260 EDSs, including those in its recapitalization plan, at an estimated cost of $256 million during the fourth quarter of fiscal year 2011. Until TSA begins purchasing machines under the current EDS acquisition to meet 2010 requirements, the agency has continued to purchase EDSs under existing contracts with current vendors.

TSA Faces Challenges in Collecting Explosives Data Needed to Procure EDSs That Meet the 2010 Requirements

TSA and S&T have experienced a number of challenges related to collecting data on some explosives data needed to procure and deploy EDSs that meet the 2010 requirements. These data are needed both by vendors to develop EDS detection software and by the TSL for the certification testing process and includes such information as the physical and chemical properties of explosives. Participants in the data collection effort include S&T; the TSL, which is taking the lead; the Air Force Research Laboratory (AFRL) at Tyndall AFB, Florida; and Lawrence Livermore National Laboratory. The AFRL is assisting the TSL because the AFRL facility at Tyndall AFB, Florida, is better equipped to safely handle certain explosives as part of the testing and data collection efforts than the TSL facility, according to S&T and Air Force officials.

In the course of collecting data, TSL and AFRL officials determined that some of the explosives were very unstable or volatile and special care and procedures were required to reliably and safely handle them. This caused delays as TSA and S&T were unable to provide vendors with all of the data on the explosives or simulants, so that vendors could create and test the software used to detect them. TSL and AFRL also collected scans of explosives as another means to provide vendors with needed data. Because micro-computed technology (micro-CT) data are provided by scanning smaller, less dangerous, amounts of each explosive, TSL and AFRL officials did not face the same challenges in safely synthesizing and determining the physical and chemical properties of the micro-CT data that they faced in working with larger amounts, known as full threat weight, of explosives. TSL officials told us that they provided the micro-CT data to help vendors in developing their explosives detection software. Specifically, TSA was able to distribute the micro-CT data to vendors in early fiscal year 2010, and five of the six vendors we interviewed stated that these data were of limited use in developing their explosives detection software. For example, one vendor stated that the micro-CT data provided some guidance, but that there were too many unknowns to fully use the data to develop their explosives detection software. Further, TSL officials stated that providing the micro-CT data to vendors only served as an intermediary step to providing full threat-weight data that vendors needed to develop their explosives detection software. TSL officials stated that the micro-CT data could not provide vendors with all of the data they needed to fully develop their explosives detection software to meet the 2010 EDS requirements because vendors need scans of the full threat weight of explosives on their respective EDSs to finish developing their detection software.

Because of the limitations of simulants and micro-CT data, TSA and S&T decided to collect and distribute scans of the explosives to vendors using the full threat weight of the explosives specified in the 2010 EDS requirements. These scans were collected using each vendor’s respective EDS equipment. TSA has distributed some, but not all, of the full threat-weight data needed by vendors to develop EDS detection software. TSL officials stated that they needed full threat-weight data to conduct certification testing. Additionally, five of the six vendors we interviewed agreed that the full threat-weight data will be necessary in order for vendors to develop their explosives detection software. However, all six vendors noted that because of concerns about the safety of handling certain explosives, they are relying on TSA for the full threat- weight data. Further, four of the vendors said that in the past they had access to some explosives and could collect their own data to develop and test their detection software in order to prepare for certification testing. However, because the vendors cannot safely handle certain explosives, they are reliant upon the data provided by TSA. Consequently, until S&T completes the data collection on all identified explosives being performed at TSL and the AFRL facility at Tyndall AFB, TSA cannot provide all of the data that vendors need to develop their explosives detection software and prepare for certification, nor can the TSL start certification testing of new equipment as part of the current EDS acquisition.

Challenges Related to Collecting Explosives Data Have Delayed the EDS Acquisition Schedule

TSA’s plans to award contracts for the current EDS acquisition have been delayed by at least 7 months, in part, due to the challenges experienced by S&T related to collecting explosives data. TSA officials stated that, initially, they planned to conduct the current EDS acquisition separately from efforts to collect the data needed to deploy EDSs that meet the 2010 requirements. Specifically, TSA officials stated that they planned to complete the data collection before initiating the procurement to buy EDSs that meet the 2010 requirements. However, officials stated that they subsequently decided to collect explosives data at the same time as implementing the current EDS acquisition because TSA and other stakeholders believed that the data collection effort would be straightforward and that the new requirements could be easily applied to machines procured in the current EDS acquisition. Additionally, program officials stated that procuring and deploying EDSs that meet the 2010 requirements in a phased approach (that is, implementing Level C first, then Level B, then Level A) would help to mitigate any additional challenges and some of the risks associated with collecting data needed for the 2010 requirements.

However, TSA and S&T officials acknowledged that pursuing the competitive procurement and explosives data collection at the same time had been more challenging than originally anticipated and had presented problems for the current EDS acquisition. TSA officials stated that all of the 260 EDSs they plan to purchase in 2011 will be upgraded to meet all of the 2010 EDS requirements at a later date.

In our prior reports regarding acquisitions, we reported on the elevated risk of poor program outcomes from the substantial overlap of development, test, and production activities. Specifically, we have identified development cost increases, additional delays in manufacturing and testing schedules, and increased financial risk due to pursuing procurement before testing is complete. By separating the effort to collect data on explosives needed to meet the new requirements from the related competitive procurement, TSA and S&T would have more time to collect data identifying the physical and chemical properties of explosives, provide vendors with the time needed to develop detection software, and attempt to pass CRT and certification testing without the added pressure of an acquisition deadline. For example, by completing data collection for each of the phases of the 2010 EDS requirements prior to pursuing procurements for EDSs that meet those requirements, TSA could avoid additional delays to the acquisition schedule due to any data collection challenges. To help avoid these challenges in the future, TSA officials stated that they do not plan for subsequent procurements of EDS capable of meeting the more stringent explosives detection requirements until after the data collection for these explosives has been fully completed.

We recognize that it is difficult in such situations to identify firm milestones. However, TSA has not documented its revised approach for conducting the needed data collection and related procurements sequentially rather than simultaneously. TSA does not yet have a documented strategy in place for deploying EDSs beyond July 2011; such a strategy would be valuable because TSA plans to complete the implementation of all of the requirements at an undetermined time after July 2011. Standard practices for program management state that the successful execution of any plan includes identifying in the planning process the schedule that establishes the timeline for delivering the plan. Documenting a plan to separate data collection efforts and certification from future procurements could help TSA ensure it avoids the challenges it has encountered during the current procurement.

EDS Vendors Report Concerns about the Extent to Which TSA is Communicating Effectively about the Current Procurement

Officials from five of six EDS vendors we interviewed expressed concerns about the extent to which TSA has communicated effectively with vendors interested in the current procurement. Specifically, these five vendors expressed concerns about the timeliness in which TSA responded to their questions regarding the current procurement or the manner in which TSA communicated important schedule changes, or both. Standards for Internal Control in the Federal Government state that management should ensure there are adequate means of communicating with and obtaining information from external stakeholders that may have a significant impact on the agency achieving its goals. Additionally, the Federal Acquisition Regulation (FAR) encourages exchanges of information among all interested parties, from the earliest identification of a requirement through the receipt of the proposal. The FAR further states that the purpose of exchanging information is to improve the understanding of government requirements and industry capabilities, thereby allowing vendors to judge whether or how they can satisfy the government’s requirements. The improved understanding resulting from such information exchange also enhances the government’s ability to obtain quality supplies and services at reasonable prices and, among other things, potentially increases efficiency in vendors’ proposal preparations. However, five out of six vendors we interviewed said TSA often did not provide information or respond to their questions in a timely manner, if at all. For example, four out of these five vendors said TSA did not answer their questions in a timely manner, in one case taking several months to provide answers to questions posted via a question tracker accessible online to all interested vendors. Meanwhile four of the five vendors’ officials stated TSA did not respond at all to some of their questions, while officials from the fifth vendor stated they were frustrated with how long it took TSA to answer their questions.

Officials from two vendors stated that the lack of timely communication regarding schedule changes for the EDS acquisition caused them to incur additional costs allocating extra resources and time to meet the original deadline. Specifically, officials from one vendor noted that they spent additional costs on personnel to aggressively pursue software development for the planned start of certification readiness testing (CRT), despite not having all of the full threat-weight explosives data TSA had intended to provide. Subsequently, these officials told us, TSA did not announce to vendors that CRT would be delayed until one week prior to the original deadline. EBSP officials stated that, because vendors had not yet received all of the full threat weight explosives data, they should have been aware that CRT was not going to happen according to the established schedule. However, EBSP officials agreed that providing vendors with a revised schedule prior to the previously established deadline would have helped promote greater vendor understanding about the proposed changes to TSA’s acquisition strategy.

TSA stated that it has taken a number of important steps to alleviate confusion and provide as much information to the vendors as possible. Among other things, at the start of the current procurement, TSA conducted three conferences with industry, called “industry days,” to provide a forum for sharing information with the vendor community regarding the current EDS acquisition. TSA also reported sharing multiple draft versions of the requirements documents and soliciting vendor comments. Additionally, TSA officials stated that they shared draft copies of the detection requirements and held individual classified meetings during the industry days with each interested vendor to obtain input regarding the acquisition. Finally, TSA stated that it also allowed vendors to use government owned equipment and paid for engineering services associated with the testing to help offset vendor costs. Although EBSP officials stated that they have made a concerted effort to be responsive to vendors’ questions and to call vendors directly when issues such as schedule changes arose, EBSP officials agreed that the agency did not always effectively communicate with vendors in a timely manner. Establishing a process for more timely communication with vendors competing for the current EDS procurement could help TSA to ensure that vendors have all of the information they need to meet TSA’s needs for new checked baggage screening equipment.

Improved Acquisition Planning Could Help TSA Avoid Further Delays and Potential Cost Overruns for the EDS Procurement

TSA does not have an integrated master schedule (IMS) for the EBSP, and TSA’s schedule for the current EDS acquisition, which is only a part of the program, does not fully meet best practices for preparing an acquisition schedule. Additionally, while TSA completed an initial cost estimate for the EBSP, TSA officials reported that the current cost estimate does not reflect the anticipated costs of purchasing EDSs to meet the 2010 EDS requirements. To meet the explosives detection requirements established in January 2010, TSA plans to upgrade the detection software of a currently unknown number of the deployed EDSs and 260 of the EDSs to be purchased under the current acquisition after they are deployed to airports. However, TSA has not yet developed a plan or cost estimate for the planned upgrades.

TSA Does Not Have a Schedule for the EBSP and Has Not Established a Reliable Schedule for the Current EDS Acquisition

As part of EBSP’s responsibility to provide equipment to screen all checked baggage originating at U.S. commercial airports, it is acquiring and deploying explosives detection technology to replace aging systems and meet emerging threats. While TSA established the EBSP as a long- term program to procure, test, deploy, and maintain checked-baggage- screening equipment, TSA officials confirmed in December 2010 that there is currently no IMS for the EBSP. Among other things, best practices and related federal guidance call for a program schedule to be programwide in scope, meaning that it should include the integrated breakdown of the work to be performed by both the government and its contractors over the expected life of the program. Without an IMS identifying long-term plans for the EBSP, it is difficult for TSA to have a comprehensive program view of the work that must be completed to deliver explosive detection technology to replace aging systems and meet emerging threats. Without such a view, a sound basis does not exist for knowing with any degree of confidence when and how the program will be completed.

While there is no IMS for the EBSP, TSA has established a schedule for the current EDS acquisition. However, while the schedule identifies activities through the first contract award—scheduled for July 2011—of the current EDS procurement, our analysis shows that it does not identify activities planned for subsequent award windows. Additionally, TSA has encountered a number of challenges in implementing the schedule. For example, according to TSA officials, TSA originally planned to award the first EDS contract in December 2010 in order to procure machines required to meet Level C of the 2010 EDS requirements. However, TSA has since revised the schedule due to the challenges of collecting explosives data needed before development of EDS can be completed and certification testing of the machines can begin.

Based on the revised schedule, certification testing began in late 2010, according to TSA, so that in July 2011 the first EDS contract can be awarded to procure machines that meet one part of the Level C explosives detection requirements. Furthermore, while TSA has stated that it plans to procure and deploy 640 additional EDSs at an estimated cost of approximately $964 million during fiscal years 2012 through 2015, it is unclear when TSA plans for those machines to meet the remaining 2010 EDS requirements.

As of March 2011, TSA officials estimate that it will take a number of years to certify EDSs that meet all three levels—C, B, and A—of the 2010 requirements. However, the officials stated that they cannot fully develop these plans until they can evaluate the capability of the equipment to meet these requirements. This is expected to happen during the testing process associated with the current EDS procurement. TSA officials stated that they plan to deploy EDSs that meet the full set of Level C, B, and A requirements, but more precise planning, including establishing timelines, cannot occur until TSA better understands the potential for the EDS equipment to meet those requirements. However, best practices state that a comprehensive schedule should at least reflect all activities planned for a project even though some activities may be tentative and there may be uncertainties in schedule estimates due to, among other things, limited data.

In addition to the challenges TSA has encountered in carrying out the schedule as originally planned, based on our analysis, the current schedule leading up to the first contract award is not reliable. Best practices state that the success of a large-scale system acquisition, such as the current EDS acquisition, depends in part on having a reliable schedule that identifies:  when the program’s set of work activities and milestone events will  how long they will take, and  how they are related to one another.

Best practices also call for the schedule to expressly identify and define the relationships and dependencies among work elements and the constraints affecting the start and completion of work elements. Additionally, best practices indicate that a well-defined schedule also helps to identify the amount of human capital and fiscal resources that are needed to execute an acquisition. However, based on our assessment of both the original as well as an updated version of the schedule, TSA’s schedule for the current EDS acquisition does not fully comply with nine best practices for preparing a schedule as shown in table 1. Appendix III has additional information about GAO’s assessment of the extent to which TSA’s schedule meets each best practice.

Although TSA’s schedule does not fully comply with any of the nine best practices, TSA has taken action to partially or minimally meet eight of the best practices. For example, consistent with best practice 4, the schedule establishes the duration of all activities and properly reflects how long each activity should take. However, while the schedule establishes the duration of all activities, 61 percent of activities represented in the schedule are based on a 7-day calendar that does not account for holidays. Similarly, our analysis found that, consistent with best practice 5, the schedule is vertically integrated; however, issues with sequencing logic in the schedule prevent it from being fully horizontally integrated. Vertical and horizontal integration ensures that products and outcomes associated with other sequenced activities are arranged in the right order and that dates for supporting tasks and subtasks are aligned. Other areas of the schedule that remain unaddressed also reflect weaknesses that limit its usefulness as a program management tool. For example, the schedule does not fully identify the resources needed to do the work or the availability of these resources. Specifically, the schedule does not reflect what labor, material, and overhead are needed to complete key activities for the program. Resource information would assist the program office in forecasting the likelihood of activities being completed based on their projected end dates. If the current schedule does not allow for insight into current or projected over-allocation of resources, then the risk of the program slipping is significantly increased.

Additionally, TSA officials did not complete a schedule risk analysis when developing the schedule. A schedule risk analysis may be used to determine the level of uncertainty and to help identify and mitigate the associated risks. In the absence of a schedule risk analysis, the acquisition faces the risk of delays to the scheduled completion date if any delays were to occur on critical path activities. Furthermore, without this information, TSA is limited in its ability to answer questions such as how likely it is to complete the project on time and which risks are most likely to delay the project. Similarly, without a valid critical path, EBSP management lacks a clear picture of the tasks that must be performed to achieve the acquisition’s target completion date. While TSA officials noted that they had no staff or expertise to complete a schedule risk analysis, TSA provided no explanation as to why a schedule consistent with the other eight best practices had not been developed. TSA officials stated that the EDS acquisition is one of the largest acquisition programs in DHS. However, the absence of a reliable schedule makes it difficult for management to predict with any degree of confidence whether the estimated completion date for the acquisition is realistic. Furthermore, without the development of a schedule that meets scheduling best practices, TSA is limited in its ability to monitor and oversee the progress of the billions of dollars being invested in the procurement of new EDSs.

The EBSP’s Current Life- Cycle Cost Estimate Does Not Reflect Anticipated Costs for Purchasing EDSs That Meet the 2010 EDS Requirements

The EBSP does not yet have an up-to-date approved life-cycle cost estimate in place, and as a result, DHS has no reliable basis for understanding how much the program will cost. While TSA reported that it had completed a life-cycle cost estimate (LCCE) for the EBSP in May 2010, program officials reported in February 2011 that the estimate is currently being revised to reflect assumptions related to the current EDS acquisition. Specifically, officials indicated that the May 2010 LCCE did not include the anticipated costs for purchasing any EDSs that meet the revised 2010 requirements. TSA officials stated that they are working to revise the LCCE to reflect the anticipated costs of the current EDS acquisition and expected to complete the LCCE by the end of April 2011. Additionally, after conducting a review of the LCCE that was completed in May 2010, DHS’s Cost Analysis Division (CAD) found that the LCCE needed more comprehensive data and that its accuracy could not be determined. As a result, the DHS Acquisition Review Board directed the CAD to develop an appropriate cost estimate, including a reconciliation with the EBSP’s LCCE. In January 2011, officials in DHS’s CAD stated that they had initiated work on the independent cost estimate for the EBSP but were only able to complete the portion of the estimate related to current detection capabilities in the Level C requirements for one tier. Officials stated that the lack of detail in program requirements for some of Level C and all of Levels B and A limited their ability to develop an estimate that would be usable for budgetary purposes. CAD officials further noted that significant portions of the total EBSP program have yet to be defined and estimated. During the course of our review, the anticipated completion date of the LCCE has been delayed multiple times and was expected to be completed at the end of April 2011. As a result, we were unable to evaluate TSA’s approach to developing the cost estimates for the program.

We reported in June 2010 that inaccurate or incomplete cost estimates were often a factor in cost growth for DHS programs we previously reviewed. We also reported that initial cost estimates for most DHS programs were often developed after the start of acquisition activities, so they do not capture earlier cost changes. Further, our best practices for cost estimation state that estimates are integral to determining and communicating a realistic view of likely cost and schedule outcomes that can be used to support a program including planning the work necessary to develop, produce, and install equipment. However, because TSA had not established a cost estimate that accurately reflects the anticipated costs of the acquisition prior to initiating the current EDS procurement, it is unclear how DHS could determine if the budget for the EBSP is reasonable. Furthermore, in the absence of an approved cost estimate and baseline financial information for the current EDS acquisition, including the costs of purchasing machines that meet the 2010 EDS requirements, TSA has limited information to make essential cost-informed program decisions.

Although we were unable to evaluate TSA’s cost estimates for the program, the fact that TSA’s schedule for the EDS acquisition does not meet best practices for schedule estimating also raises questions about the credibility of the program’s LCCE. For example, the absence of a schedule risk analysis would have made it difficult for officials to account for the cost effects of schedule slippage when developing the LCCE. Best practices for cost estimation state that because some program costs such as labor, supervision, rented equipment, and facilities cost more if the program takes longer, a reliable schedule can contribute to an understanding of the cost impact if the program does not finish on time. The program’s success depends on the quality of its schedule and an integrated schedule is key to managing program performance and is necessary for determining what work remains and the expected cost to complete the work. In a memo from the DHS Under Secretary for Management dated July 10, 2008, DHS endorsed the use of best practices that we identified and stated that DHS would be utilizing them as a “best practices” approach in the future. However, in the absence of a reliable schedule to guide cost estimates, having a current cost estimate that reflects anticipated costs for the EDS acquisition, or submitting the revised LCCE to DHS for departmental approval, it is unclear how TSA utilized a best practices approach in developing cost estimates for the program.

TSA Has No Plan in Place Outlining How It Will Upgrade Deployed EDSs to Fully Meet the 2010 Requirements

TSA officials stated that they expect to upgrade an unknown number of the current fleet of 2,297 EDSs and 260 of the EDSs to be purchased under the current acquisition after they are deployed to airports to fully meet all phases of the 2010 requirements. However, similar to when TSA revised the EDS explosives detection requirements in 2005, it has no plan in place outlining how it will approach these upgrades. Specifically, TSA has not established an upgrade plan or conducted an analysis to determine what type of approach to upgrading deployed EDSs is likely to be most feasible, efficient, or effective. TSA officials stated that there are too many unknowns at this time regarding potential approaches to upgrading the fleet of EDSs. Standards for program management require that specific desired outcomes or results be conceptualized, defined, and documented in the planning process as part of a road map, along with the appropriate steps and time frames needed to achieve those results. Until TSA develops a plan identifying how it will approach the upgrades for currently deployed EDSs—and the plan includes such items as estimated costs, the number of machines that can be upgraded, and the number of times a given machine must be upgraded to meet the 2010 EDS requirements—it will be difficult for TSA to provide reasonable assurance that its upgrade approach is feasible or cost-effective.

TSA’s 2010 acquisition strategy identifies the planned purchase of 900 EDSs over the next 5 years, but it does not indicate what level requirements those EDSs will be required to meet. For the currently deployed equipment that will not be replaced, TSA would need to upgrade the equipment to meet the 2010 requirements. TSA officials stated that they may not upgrade all of the current fleet of EDSs to the 2010 EDS requirements because in some cases, certain models of the EDS may not be upgradeable and in other cases, it may ultimately be more cost effective to replace older EDSs with new machines. According to TSA, upgrading EDSs will require an assessment of currently deployed EDS’ detection capabilities and that the results of the assessment will affect the EDS program’s schedule, budget, and detection goals. TSA was working with a consulting firm to modify a computer model that will be used to project the costs of the upgrades if TSA were to use a time- phased installation for the upgrades. While TSA officials were working with a consulting firm, they have not yet established a plan for how they will approach the upgrades. TSA officials further stated that the number of upgrades TSA performs on currently deployed equipment will depend on the cost of the upgrades, the level of complexity of the upgrades, and whether the upgrades can be conducted in the airports or must be performed in the factory.

TSA’s approach to deploy EDSs that meet the 2010 requirements could result in the same EDSs being upgraded multiple times in order to first meet all of the Level C requirements and to then meet the Levels B and A explosives detection requirements. For example, TSA’s decision to revise its acquisition strategy and deploy EDSs that meet the Level C requirements in a phased approach could result in upgrading the same currently deployed machines twice before they may have to be upgraded a third time to meet Level B requirements and then upgraded a fourth time to meet Level A requirements. Moreover, based on TSA’s schedule for the current EDS acquisition, by the time some or all of the 260 new EDSs under the current EDS acquisition have been deployed in airports, TSA may have approved a subsequent tier of the EDS explosives detection requirements, which could involve upgrading the machines again or replacing these newly purchased and deployed machines because they cannot meet the subsequent tier of explosives detection requirements. Therefore, TSA may procure and deploy 260 EDSs that will only be used in airports for a short period of time before they will need to be upgraded, possibly multiple times, or replaced with new machines.

TSA officials told us that they will evaluate the need to upgrade EDSs to a subsequent tier at the time those requirements are finalized.

TSA officials stated that they initially delayed the analysis of the upgrade approach until the 2010 EDS explosives detection requirements were approved, an approval that occurred in January 2010. TSA officials subsequently stated that their plan to upgrade deployed EDSs is included in the recapitalization strategy due to be completed at the end of May 2011. According to TSA, vendors that have previously sold EDSs to TSA are to be asked to also include proposals to upgrade their currently deployed machines when submitting proposals for the current EDS procurement. Specifically, vendors are to be asked to include a plan for upgrading their currently deployed EDS equipment as well as cost estimates for the upgrades. TSA plans to then analyze the feasibility and costs of the vendors’ proposals. However, TSA officials stated that the equipment upgrades may or may not be implemented as part of the contract award and that TSA has discretion regarding which aspects of the contracts to implement. According to TSA, the total number of EDSs to be upgraded and the associated costs will not be known until the agency receives proposed upgrade plans and cost estimates from EDS vendors in summer 2011. According to TSA officials, any upgrades are not to occur until calendar year 2012 at the earliest and will depend on available funding and complexity of the upgrades.

TSA officials as well as officials from three of six current EDS vendors told us that they are confident that currently-deployed EDSs can be upgraded to meet Level C requirements. Specifically, TSA officials stated that the EDS vendors can rewrite the detection software to provide the capability to detect the 2010 EDS requirements. Minor hardware changes, such as new computer chips, are also expected to be made as part of these upgrades according to the TSA officials. The officials stated that, after they approve the software and hardware upgrades, EDS vendors will install the upgrades on the machines in the airports. Officials from three EDS vendors noted that they believe upgrades to the EDSs can be made in the airports when regularly-scheduled routine maintenance work is conducted.

Once deployed EDSs have been upgraded to fully meet the Level C requirements, TSA will have to make decisions about how to ensure these machines can meet subsequent phases of the 2010 EDS requirements (Levels B and A). Officials from all six EDS vendors stated that given the absence of additional data on the explosives that will be included in subsequent phases of the 2010 EDS requirements, it is difficult to know precisely what must be done to upgrade newly-purchased equipment. Therefore, none of the officials from the six vendors could provide estimates for the cost to upgrade EDSs to meet all of the requirements for one tier. However, officials from two vendors estimated the cost to upgrade new EDSs to meet Level C requirements at $50,000 to $150,000 per machine. An official from one vendor stated that the CT technology currently used in EDSs might not be sufficient to detect Level A requirements and that an as yet undeveloped technology may be needed. The official noted that this could result in substantially higher costs to upgrade the current fleet of machines to Level A requirements than it would cost to upgrade machines to Level B requirements. Similarly, officials from two other vendors stated that meeting Level A requirements may require either new technology or a combination of current technologies instead of only using an EDS.

Although TSA and vendor officials expressed confidence that deployed EDSs can be upgraded, TSA officials also confirmed that the agency has never previously upgraded the detection software of deployed EDS or ETD machines to meet revised explosives detection requirements. Additionally, even though TSA has estimated that it will take a number of years to certify new EDSs to fully meet Levels B and A of the 2010 requirements, TSA has not yet developed similar time frames to upgrade deployed equipment. Given the number of unknowns associated with upgrading EDSs, it is unclear how long it will take the agency to upgrade deployed EDSs to meet Levels C, B, and A of the 2010 requirements. Furthermore, TSA has identified the EDS upgrade effort as a high program risk. Consequently, TSA and vendor officials’ confidence that it will be feasible and cost effective to upgrade deployed machines at airports may be unwarranted as it has not been based on experience, supported by analysis, or a documented plan.

Conclusions

TSA faces a complex task in its efforts to address explosives threats in its current and future procurements and existing fleet of checked-baggage- screening systems. The complexity of this task is amplified when taking into account the large volume of checked baggage that TSA must screen for explosives without disrupting commerce. TSA’s plan to procure and deploy EDSs that meet the 2010 requirements in a phased approach that spans a number of years is aimed at allowing more time to collect necessary explosives data, test key technologies, and provide a means for TSA to continue to purchase EDSs to meet its needs for new checked- baggage-screening equipment at the nation’s commercial airports. However, TSA officials recognized that if TSA deploys EDS capable of detecting all explosives included in the 2010 EDS requirements, TSA must ensure that ETD machines are capable of detecting all of the explosives that EDSs will be able to detect to minimize any potential screening difference between the EDS and ETD. Without a plan to help ensure that additional screening devices or protocols are in place to resolve EDS alarms if EDSs are deployed that detect a broader set of explosives than existing ETD machines used to resolve EDS screening alarms, it will be difficult for TSA to provide reasonable assurance that a potential capability gap has been resolved.

By separating the effort to collect data needed to meet the 2010 EDS requirements from the related competitive procurement, TSA would have more time to identify the physical and chemical properties of the explosives, collect full threat weight data, provide vendors with the time needed to develop detection software, and attempt to pass CRT and certification testing without the added pressure of an acquisition deadline. TSA also faces additional challenges related to the agency’s plans for implementing the current EDS procurement. For example, the lack of timely communication with vendors may impact vendors’ abilities to ensure they can meet TSA’s needs for the current EDS acquisition. By establishing a process to communicate with vendors in a timely manner, TSA could help ensure that vendors have the information necessary to meet TSA’s needs for new checked-baggage-screening equipment. Moreover, by addressing challenges related to planning for the acquisition, TSA may be able to better avoid further delays and potential cost overruns for the current procurement. Specifically, completing a reliable IMS that fully meets the nine best practices could help DHS and TSA management to predict whether the estimated acquisition completion date is realistic and manage program performance. Once a reliable schedule is in place, TSA can in turn revise current cost estimates for the program to better reflect actual acquisition costs including, for example, the potential cost impacts resulting from schedule slippage to give program decision-makers a more accurate and comprehensive view of current and projected program costs.

As TSA plans to deploy EDSs that meet the 2010 requirements, it is critical that TSA plans its approach to ensure that all airports with EDS equipment are capable of detecting the required explosives. Because TSA has not yet upgraded most of the deployed EDSs to meet certain requirements, many EDSs are only capable of detecting certain explosives. Moreover, of the EDSs currently deployed, TSA is currently operating some number of them at the capability needed to detect the explosives identified in the 2005 requirements, although activating the software and operationally testing the machines to detect the 2005 requirements would help address this issue.

As part of TSA’s phased approach to meet the 2010 EDS requirements, TSA may have to upgrade many of its currently deployed EDSs and hundreds of newly purchased EDSs over a period of years, upgrades that may require significant investments in new technologies to help meet more stringent explosives detection requirements. However, until TSA develops a plan identifying how it will approach the upgrades for currently deployed EDSs—and the plan includes such items as estimated costs, the number of machines that can be upgraded, time frames for upgrading them, and the number of times a given machine must be upgraded to meet the 2010 EDS requirements—it will be difficult for TSA to provide reasonable assurance that its upgrade approach is feasible or cost- effective.

Recommendations for Executive Action

To help ensure that TSA takes a comprehensive and cost-effective approach to the procurement and deployment of EDSs that meet the 2010 EDS requirements and any subsequent revisions, we recommend that the Assistant Secretary for TSA take the following six actions:  Develop a plan to ensure that screening devices or protocols are in place to resolve EDS alarms if EDSs are deployed that detect a broader set of explosives than existing ETD machines used to resolve EDS screening alarms.  Develop a plan to ensure that TSA has the explosives data needed for each of the planned phases of the 2010 EDS requirements before starting the procurement process for new EDSs or upgrades included in each applicable phase.  Establish a process to communicate information to EDS vendors in a timely manner regarding TSA’s EDS acquisition, including information such as changes to the schedule.  Develop and maintain an integrated master schedule for the entire Electronic Baggage Screening Program in accordance with the nine best practices identified by GAO for preparing a schedule.  Ensure that key elements of the program’s final cost estimate reflect critical issues, such as the potential cost impacts resulting from schedule slippage identified once an integrated master schedule for the Electronic Baggage Screening Program has been developed in accordance with the nine best practices identified by GAO for preparing a schedule.  Develop a plan to deploy EDSs that meet the most recent EDS explosives-detection requirements and ensure that new machines, as well as machines deployed in airports, will be operated at the levels established in those requirements. This plan should include the estimated costs for new machines and upgrading deployed machines, and the time frames for procuring and deploying new machines and upgrading deployed machines.

Agency Comments and Our Evaluation

We provided a draft of this report to DHS on June 23, 2011, for review and comment. On July 6, 2011, DHS provided written comments, which are presented in appendix IV. We also provided relevant excerpts of our draft report to DOD and the Department of Energy for review and comment. In commenting on our report, DHS stated that it agreed with our six recommendations and identified actions planned or under way to implement them. DOD provided written technical comments and the Department of Energy provided technical comments in an e-mail. Both stated that the draft report excerpts related to their respective agencies contained accurate information.

Overall, DHS stated that, because of the urgent need to meet ongoing requirements, TSA began addressing many of the issues identified by this audit while the audit was being conducted. However, as DHS noted in its letter, TSA still needs to complete many actions to resolve the issues identified in this report. Additionally, TSA stated that it suspended the implementation of the 2005 requirements because of the computer modeling effort known as “Project Newton” and then issued the 2010 detection standards when Project Newton did not yield timely results.

Thus, in its comments, TSA confirmed that it is using some number of EDSs that meet requirements established in 1998 by the FAA, as we reported, an approach that raises questions about how well some of its deployed equipment detects current explosives threats.

In addition, DHS stated that TSA is currently taking steps to collect the operational data necessary to support the upgrade of deployed equipment based on 2010 detection standards and that the operational data-collection effort is to be completed in 2011. However, as discussed in the report, this could be a difficult endeavor as TSA is still in the process of conducting operational testing to determine the staffing implications of operating EDSs that meet 2005 explosives detection requirements. Therefore, it will have taken TSA 6 years from the time that the 2005 EDS explosives detection requirements were issued until this operational testing is to be completed. Furthermore, if the results of the operational testing show that operating EDS machines, to meet the 2005 requirements, will require additional TSA staff and/or slow down the rate of checked-baggage screening, TSA may have to make difficult decisions and trade-offs that could affect aviation security and commerce, and also affect the schedule for meeting the 2010 requirements.

DHS concurred with our first recommendation to develop a plan to ensure that screening devices or protocols are in place to resolve EDS alarms if EDSs are deployed that detect a broader set of explosives than existing ETD machines used to resolve EDS screening alarms. DHS stated that TSA convened a working group to assess capability gaps for secondary screening technology, evaluate current technology capabilities against the capabilities of future EDSs, and prepare a plan to procure any additional technology required to ensure alarms can be resolved. DHS expects this plan to be finalized by the fourth quarter of fiscal year 2012. While these actions and planned actions represent positive steps to fully implement the recommendation, TSA should develop a plan to ensure that screening devices or protocols are in place to resolve EDS alarms if EDSs are deployed that detect a broader set of explosives than existing ETD machines used to resolve EDS screening alarms.

DHS concurred with our second recommendation to develop a plan to ensure that TSA has the data needed for each of the planned phases of the 2010 EDS requirements before starting the procurement process for new EDSs or upgrades included in each applicable phase. DHS commented that TSA modified its strategy for the EDS’s competitive procurement in July 2010 in response to the challenges in working with the explosives for data collection and alerted the vendor community on September 3, 2010. DHS stated that the new baseline schedule removed data collection from the acquisition process. Additionally, DHS stated that TSA is working with DHS S&T to establish a laboratory by summer 2011 to support further data collection and independent test and evaluation. Although these actions respond in part to the intent of our recommendation, separating data collection from the acquisition process does not necessarily ensure that the needed data will be available before starting the procurement process for the new EDSs or upgrading currently deployed EDSs. Consequently, we continue to believe that, to fully address our recommendation, a plan is needed to establish a process for ensuring that data are available before starting the procurement process for new EDSs or upgrades for each applicable phase. Developing and following such a plan would assist TSA in implementing the acquisition and making upgrades in an efficient and effective manner and would benefit DHS in its oversight role of TSA by allowing DHS to determine progress against the plan.

DHS concurred with our third recommendation to establish a process to communicate information to EDS vendors in a timely manner regarding TSA’s EDS acquisition, including information such as changes to the schedule. In the letter, DHS stated that TSA has a process for communicating information to the vendor community and will continue to follow this process in adherence with guidelines outlined in the Federal Acquisition Regulation. DHS also stated that TSA significantly changed the business model to procure checked-baggage-screening equipment from what has historically been a sole-source environment to a competitive environment, resulting in significant improvements in communication with industry. In addition, according to DHS, TSA has already made a number of efforts to improve the quality and frequency of communication with industry, but TSA recognizes the complexity associated with many of the acquisitions currently ongoing. As such, TSA acknowledged that there are opportunities to continue to improve communication with the vendor community and will take steps to ensure that vendors are provided with the most current information possible in an efficient manner. Since the agency did not provide us with evidence of how it plans to ensure more timely and effective communications with vendors in the future, we continue to believe that such a process is needed to ensure that TSA officials are aware of the specific guidelines to follow to communicate with vendors about current and future acquisitions. Our meetings with vendors indicated that TSA’s communications with them continue to leave room for improvement.

DHS concurred with our fourth recommendation to develop and maintain an IMS for the entire EBSP in accordance with the nine best practices identified by GAO for preparing a schedule. DHS commented that TSA has already begun working with key stakeholders to develop and define requirements for an IMS and to ensure that the schedule aligns with the best practices outlined by GAO. DHS stated that this effort is expected to be completed by the second quarter of fiscal year 2012. In addition, DHS stated that, as the program matures and increases its focus on flexible and upgradeable technology, an IMS will ensure close coordination among the program’s procurement, deployment, recapitalization, and upgrade capabilities, and that the EBSP IMS will be updated as a result of these efforts to be in accordance with the nine best practices. While these actions and planned actions are steps toward implementing our recommendation, to fully implement the recommendation, TSA needs to develop and maintain an IMS for the entire EBSP in accordance with the nine best practices identified by GAO for preparing a schedule.

DHS concurred with our fifth recommendation to ensure that key elements of the program’s final cost estimate reflect critical issues, such as the potential cost impacts resulting from schedule slippage. Such a slippage might be identified once an IMS for the EBSP has been developed in accordance with the nine best practices identified by GAO for preparing a schedule. DHS stated that TSA is working to update the EBSP LCCE to incorporate cost estimates associated with enhanced detection, work that should be completed in the fourth quarter of fiscal year 2011. DHS also stated that, per the recommendations of GAO and DHS, TSA is developing a master schedule to document timelines associated with various projects. DHS further stated that risks to the costs and schedules will be analyzed and that the risk analysis will produce confidence intervals for the life-cycle costs to the program. Although TSA discussed activities to address the EBSP LCCE, to fully implement this recommendation, it will be important that key elements of the program’s final cost estimate reflect critical issues, such as the potential cost impacts resulting from schedule slippage identified once an IMS for the EBSP has been developed in accordance with the nine best practices.

DHS concurred with our sixth recommendation to develop a plan to deploy EDSs that meet the most recent EDS explosives detection requirements and ensure that new machines, as well as machines deployed in airports, will be operated at the levels established in those requirements. This plan should include the estimated costs for new machines and upgrading deployed machines, and the time frames for procuring and deploying new machines and upgrading deployed machines. DHS commented that TSA has a plan in place to evaluate and implement the most recent certified algorithms on the existing fleet of deployed EDSs, assuming the evaluation results in minimal to no operational impact. In contrast, our recommendation calls for a plan to deploy new EDSs as well as to upgrade existing EDSs in airports to meet the 2010 EDS explosives detection requirements and, importantly, ensure that new machines will be operated at the levels established in those requirements. As we discussed in the report, some number of the EDSs in airports are operating at a level that meets the 2005 explosives detection requirements. Our recommendation is intended to ensure that TSA operates all EDSs in airports to meet the most recent requirements, which are currently the 2010 requirements. Consequently, we continue to believe that a plan is needed describing the approach that TSA will use to deploy EDSs that meet the most recent EDS explosives detection requirements and ensure that new machines, as well as machines deployed in airports, will be operated at the levels established in those requirements.

TSA also provided written technical comments, which we incorporated in the report, as appropriate.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 10 days from the report date. At that time, we will send copies of this report to the Secretary of Homeland Security, the Assistant Secretary of the Transportation Security Administration, and appropriate congressional committees. This report also will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-8777 or LordS@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

This report discusses: (1) the extent to which TSA has revised explosives detection requirements and deployed EDSs and ETDs to meet these revised requirements; (2) any challenges that TSA and the Department of Homeland Security’s (DHS) Science and Technology Directorate (S&T) have experienced in implementing the EDS acquisition; and (3) the extent to which TSA’s approach to its EDS acquisition meets best practices for schedule and cost estimates, and includes plans for potential upgrades to deployed EDSs.

To determine the extent to which TSA has revised explosives detection requirements for checked baggage screening, we reviewed TSA’s EDS explosives detection requirements for checked baggage screening and assessed the extent to which the 2010 detection requirements differed from the 2005 detection requirements for EDSs. We compared specific explosives 1998, 2005, and 2010 detection requirements to identify commercial and homemade variants of the explosives. We also identified, analyzed, and discussed with TSA and S&T officials the differences between the tiers and multiple levels of explosives detection requirements in the 2010 EDS explosives detection requirements. We discussed with TSA officials the 2002 explosives detection requirements for the ETD and reviewed the 2006 explosives detection requirements for ETD. We also compared the 2010 EDS requirements with the 2006 ETD requirements and discussed with TSA and S&T officials the differences between the explosives detection requirements for the EDS and ETD. We discussed TSA’s Standard Operating Procedures for resolving EDS and ETD alarms with TSA officials. Finally, we visited three of the Department of Energy’s national laboratories, Lawrence Livermore National Laboratory, Los Alamos National Laboratory, and Sandia National Laboratories, to determine the status of Project Newton. (For more on Project Newton, see app. II.)

To identify any challenges that TSA has experienced implementing the EDS acquisition, we reviewed documentation from TSA’s Electronic Baggage Screening Program (EBSP), the program responsible operational testing, procurement, deployment, and maintenance of checked-baggage-screening technologies. Among other things, we reviewed available program documentation on the status of its EDS acquisition including EBSP strategic plans from previous years as well as the most recent EBSP strategy approved in July 2010. We also reviewed documentation from the program’s first Acquisition Review Board review, the EBSP risk management plan, the most recent procurement specifications for the EDS, information posted by EBSP for interested vendors on FedBizOpps.gov, and DHS acquisition guidance and directives. We also interviewed EBSP program officials, including the EBSP program manager, regarding the program’s approach to the current EDS acquisition and received updates on revisions to the program’s EDS acquisition strategy and timelines for the current procurement.

To further understand the challenges TSA and S&T face in preparing for new EDSs to meet revised detection requirements, we reviewed documentation provided by TSA outlining the agency’s plan for deploying EDSs that meet the 2010 requirements as well as documentation regarding S&T’s approach to testing and certification carried out at the Transportation Security Laboratory (TSL). We also conducted interviews with TSA and S&T officials and conducted site visits to the TSL in Atlantic City, New Jersey, and to the Air Force Research Lab (AFRL) facility at Tyndall Air Force Base (AFB), Florida, to obtain information on efforts to certify EDSs that meet the 2010 requirements. We visited the TSL because that is where S&T tests and evaluates transportation technologies including checked baggage screening technologies. We visited AFRL because they are assisting TSL in their efforts to collect data regarding the physical and chemical properties of explosives included in the 2010 EDS requirements in preparation to certify EDSs for the current procurement. Additionally, we conducted site visits and/or telephone interviews with all six EDS vendors competing in the first phase of the current EDS procurement. These vendors were able to provide us with an understanding of their companies’ views regarding TSA’s approach to the current procurement as well as potential challenges they believe vendors face in preparing to compete for the current EDS procurement. While information we obtained from these interviews may not be generalized across the industry as a whole, we were able to obtain the perspectives of all companies planning to compete for the current EDS procurement, and they were able to provide an understanding of their companies’ abilities to develop EDSs that meet the 2010 requirements. We also reviewed TSA documentation to identify the explosives detection technologies that are used for checked baggage screening. Additionally, we interviewed TSA and S&T officials to identify the number of currently-deployed explosives detection machines that meet the previous and most recent detection requirements, and found the data for the number of machines to be sufficiently reliable.

To determine the extent to which TSA’s approach to its EDS acquisition meets best practices for schedule and cost estimates and includes plans for potential upgrades to deployed EDSs, we determined the extent to which TSA had established an integrated master schedule (IMS) for the EBSP, and due to the lack of an IMS, assessed the EDS acquisition schedule against nine best practices in our Cost Estimating and Assessment Guide. We conducted this assessment to determine the extent to which the schedule reflects key estimating practices that are fundamental to having and maintaining a reliable schedule. In doing so, we independently assessed the schedule for the current EDS acquisition and its underlying activities against our nine best practices, as provided to us in July 2010. We subsequently interviewed cognizant program officials to discuss their use of best practices in creating the schedule and to discuss the findings resulting from our review of the schedule. After TSA revised the schedule to reflect changes in some of the timelines and provided it to us in October 2010, we reviewed the updated schedule and compared it to information in the original schedule in order to understand how the new schedule was constructed and to determine to what extent TSA had resolved weaknesses that we identified in its original schedule. We also assessed the schedule against relevant best practices in our Cost Estimating and Assessment Guide to determine the extent to which it reflects key estimating practices that are fundamental to having a reliable schedule. We compared TSA’s efforts with internal control standards and recommended practices we previously identified for sound acquisition planning.

To further evaluate TSA’s planning for the current EDS acquisition, we also interviewed TSA, S&T, and EDS vendors’ officials to identify any challenges involved and expected in upgrading EDS detection capabilities, and TSA’s plans to upgrade equipment to meet future implementations of the 2010 EDS requirements. Also, during our site visits and telephone interviews with the six vendors, as discussed previously, vendors provided their perspectives on TSA’s approach to upgrade currently deployed EDSs as well as those to be deployed in the future. We also obtained from the six vendors their perspectives on how upgrades to deployed EDSs might be accomplished and potential costs involved in performing the upgrades.

We conducted this performance audit from September 2009 through May 2011 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. During the course of our review, we revised the engagement objectives and scope to facilitate a broader examination of TSA’s efforts to revise its explosives detection requirements and related schedule for the EDS acquisition, which increased the time for completing this audit. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Department of Homeland Security Efforts to Conduct Computer Modeling to Establish a Tier of the Explosives Detection Requirements for the Explosives Detection System

The Department of Homeland Security (DHS) Science and Technology Directorate (S&T) and Transportation Security Administration (TSA) began a project in 2007, known as Project Newton, to identify the minimum mass of explosives that could cause catastrophic damage to an aircraft. Through 2009, S&T and TSA had invested approximately $12.5 million in Project Newton modeling activities, according to a senior TSA official. A different senior TSA official stated that TSA allocated an additional $2.5 million to $3.1 million for Project Newton as of August 2010: $1.0 million to $1.6 million for incremental development of computer models and $1.5 million to develop a plan to validate the models. As part of the effort to understand the effects of explosives detonations on aircraft, S&T and TSA have been working to simulate the complex dynamics of explosive blast effects on an in-flight aircraft by using computer models at three Department of Energy national laboratories—Lawrence Livermore National Laboratory, Los Alamos National Laboratory, and Sandia National Laboratories. According to TSA officials, the current understanding of the effects of explosives on aircraft has been largely based on data from live-fire explosives tests conducted with retired aircraft hulls at ground level. These officials stated that, compared to running a computer simulation, live-fire tests can be more expensive, which limits the number of live-fire tests conducted and, therefore, the amount of data available for analysis. S&T, TSA, and national laboratory officials stated that computer modeling can cost- effectively simulate the effects of explosives detonations in various locations of different types of aircraft at ground level, which can provide significant data for analysis.

In January 2010, TSA revised the explosives detection requirements for the explosives detection system (EDS) and established tiers of explosives that are required to be detected. One tier of requirements is to be implemented over a number of years. TSA plans to incorporate the computer-modeling results into the requirements for a subsequent tier. Although TSA expected that the computer-modeling results would be used to revise EDS explosives detection requirements as early as 2012, as of December 2010, TSA officials were uncertain when the computer- modeling results will be used for this purpose because the computer models had not been validated. In 2009, TSA established a Blue Ribbon Panel to, among other things, assess the three national laboratories’ computer models and their results and comment on whether they were valid to be used to revise explosives detection requirements. The panel members included DHS and TSA officials as well as officials from academia and the private sector. In March 2010, the Blue Ribbon Panel recommended that, among other things, before the computer modeling results are used to revise EDS explosives detection requirements, the computer models and their results should be validated, according to a senior TSA official. This official stated that the panel also recommended specific locations to add to the computer models, so that the models can simulate the effects of explosives detonations in those additional locations on the aircraft. Validating the computer models and their results is essential before relying on them to revise explosives detection requirements. A senior TSA official stated that it will take a number of months to validate the computer models, validation that is expected to be completed later in 2011.

Appendix III: GAO’s Assessment of the Extent to Which the Electronic Baggage- Screening-Program Schedule Meets Established Best Practices

In determining the extent to which the Transportation Security Administration’s (TSA’s) Electronic Baggage Screening Program (EBSP) schedule meets established best practices, we identified that TSA did not have an integrated master schedule (IMS) for the program. As a result, we assessed the explosives detection system (EDS) acquisition schedule against each of nine best practices. Specifically, we assessed TSA’s initial schedule, which was provided to GAO in July 2010, and met with TSA officials to discuss our assessment and provided officials with suggestions for corrective action. TSA later revised the schedule and provided GAO with an updated version in October 2010. We completed a separate assessment of TSA’s revised schedule. The following table presents our two assessments.

Appendix IV: Comments from the Department of Homeland Security

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Glenn Davis, Assistant Director, and Joseph E. Dewechter, Analyst-in-Charge, managed this assignment. Scott Behen, Samantha Carter, and Orlando Copeland made major contributions to the planning and all other aspects of the work. David Alexander and Richard Hung assisted with design, methodology, and data analysis. Jason Lee and Karen Richey assisted with acquisition cost and schedule analysis. John Hutton and Nathan Tranquilli assisted with acquisition and contracting issues. Katherine Davis provided assistance in report preparation. Thomas Lombardi and Tracey King provided legal support.