Aviation Security: DHS and TSA Have Researched, Developed, and Begun Deploying Passenger Checkpoint Screening Technologies, but Continue to Face Challenges
Released: 2009-10-29T00:00:00

Letter

Background

TSA’s airport passenger checkpoint screening system is comprised of three elements: the (1) personnel, or screeners, responsible for operating the checkpoint, including the screening of airline passengers and their carry-on items; (2) standard operating procedures that screeners are to follow to conduct screening; and (3) technology used during the screening process. Collectively, these elements determine the effectiveness and efficiency of passenger checkpoint screening. In strengthening one or more elements of its checkpoint screening system, TSA aims to balance its security goals with the need to efficiently process passengers. We previously reported that TSA had made progress in enhancing its passenger checkpoint screening system by strengthening screener training, measuring the performance of screeners and the screening system, and modifying screening procedures to address terrorist threats and efficiency concerns. We made recommendations to DHS designed to strengthen TSA’s efforts to train screeners, modify screening standard operating procedures, and measure the performance of the checkpoint screening system. DHS generally agreed with our recommendations and TSA has taken steps to implement them.

Passenger Checkpoint Screening Process

Passenger screening is a process by which screeners inspect individuals and their property to deter and prevent an act of violence or air piracy, such as the carriage of any unauthorized explosive, incendiary, weapon, or other prohibited item onboard an aircraft or into a sterile area. Screeners inspect individuals for prohibited items at designated screening locations. TSA developed standard operating procedures and the process for screening passengers at airport checkpoints. Figure 1 illustrates the screening functions at a typical passenger checkpoint.

Primary screening is conducted on all airline passengers prior to entering the sterile area of an airport and involves passengers walking through a metal detector and carry-on items being subjected to X-ray screening. Passengers who alarm the walk-through metal detector or are designated as selectees—that is, passengers selected for additional screening—must then undergo secondary screening, as well as passengers whose carry-on items have been identified by the X-ray machine as potentially containing a prohibited item. Secondary screening involves additional means for screening passengers, such as by hand-wand, physical pat-down or, at certain airport locations, an ETP, which is used to detect traces of explosives on passengers by using puffs of air to dislodge particles from their body and clothing into an analyzer. Selectees’ carry-on items are also physically searched or screened for explosives traces by Explosives Trace Detection (ETD) machines. In addition, DHS S&T and TSA have deployed and are pursuing additional technologies to provide improved imaging or anomaly detection capacities to better identify explosives and other threat objects.

Roles and Responsibilities for the RDT&E, Procurement, and Deployment of Checkpoint Screening Technologies

DHS and TSA share responsibility for the screening of passengers and the research, development, and deployment of passenger checkpoint screening technologies. Enacted in November 2001, the Aviation and Transportation Security Act (ATSA) created TSA and charged it with the responsibility of securing civil aviation, which includes the screening of all passengers and their baggage. ATSA also authorized funding to accelerate the RDT&E of new checkpoint screening technologies. The Homeland Security Act of 2002, enacted in November 2002, established DHS, transferred TSA from the Department of Transportation to DHS and, within DHS, established S&T to have primary responsibility for DHS’s RDT&E activities, and for coordinating and integrating all these activities. The Intelligence Reform and Terrorism Prevention Act of 2004 (Intelligence Reform Act), enacted in December 2004, directed the Secretary of Homeland Security to give high priority to developing, testing, improving, and deploying checkpoint screening equipment that detects nonmetallic, chemical, biological, and radiological weapons and explosives, in all forms, on individuals and in their personal property.

Until fiscal year 2006, TSA had primary responsibility for investing in the research and development of new checkpoint screening technologies, and was responsible for developmental and operational test and evaluation of new technologies. However, during fiscal year 2006, research and development functions within DHS were consolidated, for the most part, within S&T. After this consolidation, S&T assumed primary responsibility for funding the research, development, and developmental test and evaluation of airport checkpoint screening technologies. S&T also assumed responsibility from TSA for the Transportation Security Laboratory (TSL) which, among other things, tests and evaluates technologies under development. TSA, through the PSP that was transferred from the Federal Aviation Administration (FAA) to TSA, continues to be responsible for identifying the requirements for new checkpoint technologies; operationally testing and evaluating technologies in airports; and procuring, deploying, and maintaining technologies. This transfer of responsibility from TSA to S&T did not limit TSA’s authority to acquire commercially available technologies for use at the checkpoint.

DHS and TSA’s Processes for the RDT&E, Procurement, and Deployment of Checkpoint Screening Technologies

S&T and TSA’s RDT&E, procurement, and deployment efforts are made up of seven components: basic research, applied research, advanced development, operational testing, procurement, operational integration, and deployment. S&T is responsible for conducting basic and applied research, and advanced development, including developmental test and evaluation. TSA is responsible for conducting operational test and evaluation, operational integration, procurement and deployment of new technologies, including checkpoint screening technologies. These seven components are described below.

Basic research includes scientific efforts and experimentation directed toward increasing knowledge and understanding in the fields of physical, engineering, environmental, social, and life sciences related to long-term national needs.

Applied research includes efforts directed toward solving specific problems with a view toward developing and evaluating the feasibility of proposed solutions.

Advanced development includes efforts directed toward projects that have moved into the development of hardware and software for field experiments and tests, such as acceptance testing.

Operational test and evaluation verifies that new systems are operationally effective, supportable, and suitable before deployment.

Operational integration is the process employed to enable successful transition of viable technologies and systems to the field environment.

Procurement includes the efforts to obtain a product or service. Deployment is a series of actions following the determination that the product meets its requirements and is accepted by the program manager and integrated product team; designated locations are configured for product integration into the screening operating system and the installed product passes site acceptance tests; and logistics support is in place and all users are trained to use the product.

RDT&E, Procurement, and Deployment Funding for Checkpoint Screening Technologies

Over $795 million has been invested by DHS and TSA during fiscal years 2002 through 2008 for the RDT&E, procurement, and deployment of checkpoint screening technologies. During this time, over $91 million was invested in the RDT&E of checkpoint technologies and about $704 million was invested in the procurement and deployment of these technologies. From fiscal years 2002 through 2005, TSA was responsible for the RDT&E of checkpoint technologies; however, TSA officials could not identify the amount of funding the agency invested for these purposes during those years. After fiscal year 2005, TSA invested $14.5 million for test and evaluation of checkpoint technologies, but did not fund the research and development of these technologies because responsibility in general for research and development funding was transferred from TSA to S&T beginning in fiscal year 2006. Therefore, during fiscal years 2006 through 2008, S&T invested $77.0 million in the RDT&E of checkpoint screening technologies. All of the approximately $704 million for the procurement and deployment of checkpoint screening technologies from fiscal years 2002 through 2008 was invested by TSA because the agency has been responsible for procurement and deployment of these technologies since it was created.

Applying a Risk Management Approach to Checkpoint Technology Investments

Risk management is a tool that policy makers can use to help ensure that strategies to develop protective programs and allocate resources target the highest priority security needs. This information helps officials determine which security programs are most important to develop and fund, given that it is not possible to protect the country against all threats because of limited resources. Law and related policy, including the Intelligence Reform Act, the Implementing Recommendations of the 9/11 Commission Act of 2007 (9/11 Commission Act), and Homeland Security Presidential Directive 7, provide that federal agencies with homeland security responsibilities are to apply risk-informed principles to prioritize security needs and allocate resources. Consistent with these provisions, DHS issued the National Strategy for Transportation Security in 2005 that, among other things, describes the policies that DHS is to apply when managing risks to the security of the U.S. transportation system. Further, in June 2006, DHS issued the NIPP, which provides a risk management framework to guide strategies to develop homeland security programs and allocate resources to them. According to the NIPP, its risk management framework consists of six phases that help to identify and assess risks and prioritize investments in programs, as illustrated in figure 2. The NIPP designated TSA as the primary federal agency responsible for coordinating critical infrastructure protection efforts within the transportation sector.

A risk-informed strategy to develop and invest in critical infrastructure protection, according to the NIPP, begins with setting security goals. Setting security goals involves defining specific outcomes, conditions, end points, or performance targets that collectively constitute an effective protective posture. Once security goals are established, decisionmakers are to identify what assets or systems to protect and identify and assess the greatest risks to them, that is, the type of terrorist attack that is most likely to occur and that would result in the most severe consequences. Risk of a terrorist attack, according to the NIPP, is to be assessed by analyzing consequences of an attack; the threat—that is, the likelihood of an attack; and the extent to which an asset or a system, in this case the transportation system, is vulnerable to this type of attack. The potential consequences of any incident, including terrorist attacks and natural or manmade disasters, is the first factor to be considered in a risk assessment. In the context of the NIPP, consequence is measured as the range of loss or damage that can be expected in the event a terrorist attack succeeds. A consequence assessment looks at the expected worst case or reasonable worst case impact of a successful attack. A threat assessment is the identification and evaluation of adverse events that can harm or damage an asset and takes into account certain factors, such as whether the intent and capability to carry out the attack exist. A vulnerability assessment identifies weaknesses or characteristics of an asset or system, such as its design and location, which make it susceptible to a terrorist attack and that may be exploited. This analysis should also take into consideration factors such as protective measures that are in place which may reduce the risk of an attack and the system’s resiliency, that is, ability to recover from an attack.

Once the three components of risk—threat, vulnerability, and consequence—have been assessed for a given asset or system, they are used to provide an estimate of the expected loss considering the likelihood of an attack or other incident. According to the NIPP, calculating a numerical risk score using comparable, credible methodologies provides a systematic and comparable estimate of risk that can help inform national and sector-level risk management decisions. To be considered credible, the NIPP states that a methodology must have a sound basis; be complete; be based on assumptions and produce results that are defensible; and specifically address the three variables of the risk calculus: threat, vulnerability, and consequence. The methodology should also be comparable with other methodologies to support a comparative sector or national risk assessment. To be comparable, a methodology must be documented, transparent, reproducible, accurate, and provide clear and sufficient documentation of the analysis process and the products that result from its use.

The next steps in the DHS risk management framework involve establishing priorities for program development based on risk assessments; implementing these protective programs; and measuring their effectiveness by developing and using performance measures. Identifying and assessing risks helps decisionmakers to identify those assets or systems that are exposed to the greatest risk of attack and, based on this information, prioritize the development and funding of protective programs that provide the greatest mitigation of risk given available resources. The NIPP notes that because resources are limited, risk analysis must be completed before sound priorities can be established. To determine which protective measures provide the greatest mitigation of risk for the resources that are available, the NIPP directs policy makers to evaluate how different options reduce or mitigate threat, vulnerability, or consequence of a terrorist attack. To do so, the NIPP states that cost estimates are combined with risk-mitigation estimates in a cost–benefit analysis to choose between the different options. The last step in the NIPP, measuring the effectiveness of security programs by developing and using performance measures, provides feedback to DHS on its efforts to attain its security goals. Performance metrics are to be developed and used to affirm that specific goals and objectives are being met or to articulate gaps in the national effort or supporting sector efforts. Performance measures enable the identification of corrective actions and provide decisionmakers with a feedback mechanism to help them make appropriate adjustments in their strategies for protecting critical infrastructure.

TSA Has Taken Actions to Prioritize Investments in Passenger Checkpoint Screening Technologies, but Lacks a Risk-Based Strategy

While TSA completed a strategic plan for the PSP in August 2008 that identifies a strategy for researching, developing, and deploying checkpoint screening technologies, the plan and the strategy were not developed based upon all of the key risk management principles outlined in DHS’s NIPP. For instance, TSA has not conducted a complete risk assessment for the PSP, conducted a cost–benefit analysis to prioritize investments, or developed performance measures to assess the extent to which the risk of attack has been reduced or mitigated by investments in technologies. While the agency is currently reviewing a draft of the Aviation Domain Risk Assessment (ADRA), as of September 2009, the ADRA had not been finalized. Officials expect it to be finalized by the end of calendar year 2009. TSA officials could not provide an expected completion date. Therefore, we could not determine when TSA will complete it or to what extent it will be consistent with DHS’s risk management framework. TSA officials acknowledged the importance of a cost–benefit analysis and performance measures to guide technology investments, and stated that they intend to develop them, but could not identify when they would be completed. Until TSA completes these activities, the agency lacks assurances that the PSP strategy addresses the highest priority needs and mitigates the risk of an attack. Further, TSA lacks information to adjust its strategy, if needed.

TSA Completed a Strategic Plan for the PSP that Identifies Goals and Objectives

TSA completed a strategic plan in August 2008 that identifies a strategy and establishes goals and objectives for the PSP, and submitted the plan to congressional committees in September 2008. However, TSA officials stated that the NIPP was not used as guidance in developing the plan. Instead, the officials stated that the specific requirements for a strategic plan, as outlined in the Intelligence Reform Act and 9/11 Commission Act, were used as guidance to construct the plan. The strategic plan identifies three broad trends that have been observed in the types of threats that TSA faces. First, interest in catastrophic destruction of aircraft and facilities has increased, in contrast to hijacking and hostage-taking that characterized the majority of earlier attacks. Second, the range of encountered weapons has expanded, many not previously recognized as threats, nor detected by the technologies that were deployed. Third, terrorists have attacked “soft” airport targets, including airport lobbies, in other countries. To address these challenges, TSA’s strategic plan identifies that the agency’s strategy is to utilize intelligence; partner with law enforcement, industry partners, and the public; and implement security measures that are flexible, widely deployable, mobile, and layered to address the nation’s complex open transportation network. According to the plan, TSA is in the process of implementing and evaluating a fundamental shift in strategy for the security checkpoint that encompasses the critical elements of people, process, and technology. In addition, the plan states that implementing a new security approach called Checkpoint Evolution, which started in the spring 2008, will bring the most significant changes that have occurred in passenger screening since the airport security checkpoint was first established in the 1970s.

TSA’s strategic plan identifies that the key component of TSA’s strategy related to security checkpoints is to improve security effectiveness and resource utilization at the checkpoints. Also, the PSP manager stated that a goal of the PSP strategy is to achieve full operating capability by the dates discussed for each checkpoint screening technology listed in the strategic plan. To meet these goals, the PSP strategic plan identifies three strategic objectives: (1) improve explosive detection capability, (2) improve the behavior detection capability of Transportation Security Officers (TSO), and (3) extend the layers of security throughout the passenger journey. The first objective, improving explosive detection capability, involves combining new technology with procedures that emphasize an element of unpredictability to improve explosive detection capability and prevent would-be attackers from knowing the TSA security process. The second objective, improving the behavior detection capability of TSOs, involves shaping the checkpoint environment to better support and enhance behavior detection capabilities by enabling TSOs to engage a larger number of passengers more frequently throughout the checkpoint queue using informal interviews and SPOT; improving the observation conditions for TSOs trained in SPOT by enhancing the contrast between passengers exhibiting signs of travel stress and those intending to do harm to other passengers, aircraft, or the airport; and providing communications tools for enhanced coordination between TSOs trained in SPOT. The third objective, extending the layers of security throughout the passenger journey, involves enabling additional layers of non-intrusive security beyond the checkpoint and into public spaces; increasing the interaction between TSOs and passengers to provide more opportunities to identify irregular behaviors far ahead of the potential threat reaching the checkpoint; and partnering with airlines, airports, and the private sector to reduce vulnerabilities in soft target areas.

TSA had been directed on multiple occasions to provide strategic plans for explosives detection checkpoint technologies to congressional committees. The Intelligence Reform Act mandated that TSA provide a strategic plan that included, at a minimum, a description of the current efforts to detect explosives on individuals and in their personal property; operational applications of explosive detection equipment at airport checkpoints; quantities of equipment needed to implement the plan and a deployment schedule; funding needed to implement the plan; measures taken and anticipated to be taken to provide explosives detection screening for all passengers identified for additional screening; and recommended legislative actions, if any. The Intelligence Reform Act mandated that such a strategic plan be submitted to congressional committees during the second quarter of fiscal year 2005. According to TSA officials, a strategic plan was developed and delivered to congressional committees on August 9, 2005, in satisfaction of the statutory mandate. However, the 9/11 Commission Act, enacted August 3, 2007, reiterated a requirement for a strategic plan that TSA was mandated to submit in accordance with the Intelligence Reform Act. Specifically, the 9/11 Commission Act required that the Secretary of Homeland Security issue a strategic plan addressing its checkpoint technology program not later than 30 days after enactment of the 9/11 Commission Act (that is, by September 3, 2007) and required implementation of the plan to begin within 1 year of the act’s enactment. In response to the 9/11 Commission Act, TSA provided to Congress the Aviation Security Report: Development of a Passenger Checkpoint Strategic Plan, September 2007. Finally, Division E of the Consolidated Appropriations Act, 2008, enacted on December 26, 2007, required that the Secretary of Homeland Security submit a strategic plan for checkpoint technologies no later than 60 days after enactment of the Act (that is, by February 25, 2008), and further restricted the use of $10,000,000 appropriated to TSA for Transportation Security Support until the Secretary submitted the plan to the Committees on Appropriations of the Senate and House of Representatives. As a result of the mandate for a strategic plan and the funding restriction in the 2008 Consolidated Appropriations Act, TSA officials told us that they interpreted this legislative language to mean that congressional committees considered TSA’s aviation security report in September 2007 to be incomplete and insufficient. After approximately 12 months had elapsed since a strategic plan had been mandated in the 9/11 Commission Act, in August 2008 TSA completed its revised strategic plan and delivered it to the committees in September 2008, which TSA officials stated meets the mandate for a strategic plan in the 9/11 Commission Act, as well as the mandate for a strategic plan in the appropriations act.

As previously discussed, the Intelligence Reform Act included requirements for a deployment schedule, and descriptions of the quantities of equipment and funding needed to implement the plan. However, our analysis of TSA’s August 2008 strategic plan indicates that the strategic plan could include more complete information about these requirements. For example, although TSA provided some deployment information for each emerging checkpoint technology listed in the strategic plan—such as the total quantity to be deployed, expected full operating capability date, and types or categories of airports where the equipment is to be deployed—it does not include a year-by-year schedule showing the number of units for each emerging technology that is expected to be deployed to each specific airport. Regarding information on the funding needed to implement the strategic plan, it includes a funding profile for each fiscal year from 2007 through 2009. However, a number of the emerging technologies are not expected to reach full operating capability until fiscal year 2014. TSA officials stated that they have derived notional (that is, unofficial) quantities to be deployed on an annual basis for each technology through its respective full operating capability date, but the officials stated that the funding profile in the strategic plan does not reflect the funding needed for these future quantities because the funding that will be appropriated for them after fiscal year 2009 is unknown. According to the officials, to implement the strategic plan in the years beyond fiscal year 2009, the agency intends to use a year-by-year approach whereby the quantities to be deployed in a particular year, and the funding needed for that year, would not be officially identified prior to the budget request for that year.

TSA officials stated that they used risk to inform the August 2008 strategic plan and the PSP strategy identified in it. Although TSA may have considered that risk to some degree, our analysis does not confirm that these efforts meet the risk-based framework outlined in the NIPP. Specifically, TSA has not conducted a risk assessment or cost–benefit analyses, or established quantifiable performance measures. As a result, TSA does not have assurance that its efforts are focused on the highest priority security needs, as discussed below.

TSA Has Not Conducted a Risk Assessment to Inform Its PSP Strategy, but Is Finalizing an Assessment and Developing Information that May Help Guide PSP Efforts

TSA has not conducted a risk assessment that includes an assessment of threat, vulnerability, and consequence, which would address passenger checkpoint screening; consequently, the PSP strategy has not been informed by such a risk assessment as required by the NIPP. Agency officials stated that they prepared and are currently reviewing a draft of a risk assessment of the aviation domain, known as the ADRA, which is expected to address checkpoint security and officials expect it to be finalized by the end of calendar year 2009; however, its completion has been delayed multiple times since February 2008. Therefore, it is not clear when this assessment will be completed. The ADRA, when completed, is to provide a scenario-based risk assessment for the aviation system that may augment the information TSA uses to prioritize investments in security measures, including the PSP. However, officials could not provide details regarding the extent to which the ADRA would assess threat, vulnerability, and consequence related to the passenger checkpoint. In 2004, we recommended that the Secretary of Homeland Security and the Assistant Secretary for TSA complete risk assessments—including a consideration of threat, vulnerability, and consequence—for all modes of transportation, and use the results of these assessments to help select and prioritize research and development projects. TSA and DHS concurred with the recommendation, but have not completed these risk assessments. Because TSA has not issued the ADRA or provided details regarding what it will entail, and because it is uncertain when the ADRA will be completed, it is not clear whether the ADRA will provide the risk information needed to support the PSP and TSA’s checkpoint technology strategy. In the meantime, TSA has continued to invest in checkpoint technologies without the benefit of the risk assessment information outlined in the NIPP. Consequently, TSA increases the possibility that its investments will not address the highest priority security needs.

Although TSA has not completed a risk assessment to guide its PSP, officials stated that they identify and assess risks associated with the passenger screening checkpoint by relying on threat information, vulnerability information from Threat Image Projection (TIP) scores, limitations of screening equipment identified during laboratory testing, covert tests, and expert judgment to guide its investment strategy in the PSP. Specifically, TSA’s Office of Intelligence produces civil aviation threat assessments on an annual basis, among other intelligence products. These assessments provide information on individuals who could carry out attacks, tactics they might use, and potential targets. TSA’s most recent aviation threat assessment, dated December 2008, identifies that terrorists worldwide continue to view civil aviation as a viable target for attack and as a weapon that can be used to inflict mass casualties and economic damage. It also concluded that improvised explosive devices (IED) and hijackings pose the most dangerous terrorist threat to commercial airliners in the United States. The assessment identifies that these devices may be concealed on persons, disguised as liquids, or hidden within everyday, familiar objects such as footwear, clothing, toys, and electronics. The threat assessment further identifies that terrorists have various techniques for concealing explosives on their persons. In addition to the annual civil aviation threat assessment, the Office of Intelligence prepares for TSA’s senior leadership team and other officials a (1) daily intelligence briefing, (2) tactical intelligence report that is produced one to four times per week, (3) weekly field intelligence summary, (4) weekly suspicious incident report, and, when necessary, (5) special events update, for example, during major political events. However, according to the NIPP, relying on threat information is not sufficient to identify and assess risks. Rather, threat information, which indicates whether a terrorist is capable of carrying out a particular attack and intends to do so, is to be analyzed along side information on vulnerabilities—weakness in a system that would allow such an attack to occur—and on the consequences of the attack, that is, the results of a specific type of terrorist attack, according to the NIPP.

TSA officials stated that, to guide the PSP, they also rely on programs in place that are designed to assess vulnerabilities at airport checkpoints. To identify vulnerabilities at airport checkpoints, TSA officials stated that TSA analyzes TIP scores, known limitations of screening equipment based on laboratory testing, and information from its covert testing program. TSA conducts national and local covert tests, whereby individuals attempt to enter the secure area of an airport through the passenger checkpoint with a prohibited item in their carry-on bags or hidden on their person. Officials stated they use these sources of information to identify needed changes to standard screening procedures, new technology requirements, and deployment strategies for the PSP. When a checkpoint vulnerability is identified, officials stated that TSA’s Office of Security Technology engages other TSA stakeholders through the PSP’s Integrated Project Team process to identify and develop necessary technology requirements which may lead to new technology initiatives. Officials credited this process with helping TSA identify needed changes to standard screening procedures and deployment strategies for new technologies. For example, according to a TSA official, a technology was developed as a result of tests conducted by GAO that found that prohibited items and components of an IED might be more readily identified if TSA were to develop new screening technologies to screen these items.

Although TSA has obtained information on vulnerabilities at the screening checkpoint, the agency has not assessed vulnerabilities (that is, weaknesses in the system that terrorists could exploit in order to carry out an attack) related to passenger screening technologies that are currently deployed. The NIPP requires a risk assessment to include assessments of threat, vulnerability, and consequence. TSA has not assessed whether there are tactics that terrorists could use, such as the placement of explosives or weapons on specific places on their bodies, to increase the likelihood that the screening equipment would fail to detect the hidden weapons or explosives. Although TIP scores measure how effectively screeners identify prohibited items, they do not indicate whether screening technologies currently deployed may be vulnerable to tactics used by terrorists to disguise prohibited items, such as explosives or weapons, thereby defeating the screening technologies and evading detection. Similarly, TSA’s covert testing programs do not systematically test passenger and baggage screening technologies nationwide to ensure that they identify the threat objects and materials the technologies are designed to detect, nor do the covert testing programs identify vulnerabilities related to these technologies. We reported in August 2008 that, while TSA’s local covert testing program attempts to identify test failures that may be caused by screening equipment not working properly or caused by screeners and the screening procedures they follow, the agency’s national testing program does not attribute a specific cause of the test failure. We recommended, among other things, that TSA require the documentation of specific causes of all national covert testing failures, including documenting failures related to equipment, in the covert testing database to help TSA better identify areas for improvement. TSA concurred with this recommendation and stated that the agency will expand the covert testing database to document test failures related to screening equipment. Moreover, TSA officials stated that it is difficult to attribute a test failure to equipment, because there is a possibility that the threat item used for the test was not designed properly and, therefore, should not have set off the equipment’s alarm. TSA officials also stated that it is difficult to identify a single cause for a test failure because covert testing failures can be caused by multiple factors. As a result, TSA lacks a method to systematically test and identify vulnerabilities in its passenger and baggage screening equipment in an operational airport setting. Consequently, TSA officials do not have complete information to identify the extent to which existing screening technologies mitigate vulnerabilities at the passenger checkpoints, so that they can incorporate this information into the agency’s security strategy, as required by DHS guidance.

TSA’s ADRA, once completed, is to cover the entire aviation domain and include three parts—assessments of over 130 terrorist attack scenarios to determine whether they pose a threat to the aviation system; an assessment of known vulnerabilities or pathways within the aviation system through which these terrorist attacks could be carried out; and an assessment of consequences of these various types of terrorist attacks, such as death, injury, and property loss. TSA officials stated that, through the use of expert panels, the ADRA will evaluate these threat scenarios to assess the likelihood that terrorists might successfully carry out each type of attack on the aviation system, and the likelihood and consequences of these various scenarios will be prioritized to identify the most pressing risks that need to be addressed. In the case of the passenger screening checkpoint, according to officials, TSA will be examining all security measures that a terrorist must breach in order to carry out a specific type of an attack, such as carrying an IED on board an aircraft and detonating it midflight. However, officials could not explain or provide documentation identifying the extent to which the ADRA will provide threat, vulnerability, and consequence assessments in support of the PSP. In addition, the completion date for the ADRA has been delayed multiple times. Because the ADRA has not been finalized and TSA has not described how the ADRA will address the passenger checkpoint, we could not determine the extent to which it will incorporate information on checkpoint vulnerabilities, including vulnerabilities associated with screening technologies and standard operating procedures.

In addition to the ADRA, TSA and DHS S&T are developing other information that could inform their identification and assessments of risks to the aviation transportation system. Specifically, TSA and S&T are reviewing the scientific basis of their current detection standards for explosives detection technologies to screen passengers, carry-on items and checked baggage. As part of this work, TSA and S&T are conducting studies to update their understanding of the effects that explosives may have on aircraft, such as the consequences of detonating explosives on board an in-flight aircraft. Senior TSA and DHS S&T officials stated that the two agencies decided to initiate this review because they could not fully identify or validate the scientific support requiring explosives detection technologies to identify increasingly smaller amounts of some explosives over time as required by TSA policy. Officials stated that they used the best available information to originally develop detection standards for explosives detection technologies. However, according to these officials, TSA’s understanding of how explosives affect aircraft has largely been based on data obtained from live-fire explosive tests on aircraft hulls at ground level. Officials further stated that due to the expense and complexity of live-fire tests, FAA, TSA, and DHS collectively have conducted only a limited number of tests on retired aircraft, which limited the amount of data available for analysis. As part of this ongoing review, TSA and S&T are simulating the complex dynamics of explosive blast effects on an in-flight aircraft by using a computer model based on advanced software developed by the national laboratories. TSA believes that the computer model will be able to accurately simulate hundreds of explosives tests by simulating the effects that explosives will have when placed in different locations within various aircraft models. Officials estimated this work will be completed in 3- to 4-month increments through 2008 and 2009. Officials further stated that the prototype version of the model was validated in the late summer of 2008, and that the model is currently being used. TSA and S&T officials stated that they expect the results of this work will provide a much fuller understanding of the explosive detection requirements and the threat posed by various amounts of different explosives, and will use this information to determine whether any modifications to existing detection standards should be made moving forward.

TSA Has Not Completed a Cost–Benefit Analysis to Help Establish Risk-Based Priorities and Guide Its Investment Strategy

TSA has not completed a cost–benefit analysis to prioritize and fund the PSP’s priorities for investing in checkpoint technologies, as required by the NIPP’s risk management framework. According to the NIPP, policy makers who are designing programs and formulating budgets are to evaluate how different options reduce or mitigate threat, vulnerability, or consequence of a terrorist attack through a cost–benefit analysis that combines cost estimates with risk-mitigation estimates. However, in addition to lacking information on risks to the screening checkpoint, TSA has not conducted a cost–benefit analysis of checkpoint technologies being researched and developed, procured, and deployed. Such a cost– benefit analysis is important because it would help decisionmakers determine which protective measures, for instance, investments in technologies or in other security programs, will provide the greatest mitigation of risk for the resources that are available.

One reason that TSA may have difficulty developing a cost–benefit analysis for the PSP is that it has not developed life-cycle cost estimates of each screening technology the PSP is developing, procuring, or deploying. This information is important because it helps decisionmakers determine, given the cost of various technologies, which technology provides the greatest mitigation of risk for the resources that are available. TSA officials prepared a PSP lifecycle cost estimate in September 2005, but this estimate does not include cost estimates for all technologies currently being researched, developed, tested and evaluated, procured and/or deployed, such as the Advanced Technology Systems, a technology to screen carry-on items that TSA is currently procuring. TSA was subsequently instructed by DHS Joint Requirements Council to complete lifecycle cost estimates for the PSP; in December 2005, the council reviewed the PSP and approved it to proceed to the Investment Review Board for an annual review and potential approval of the PSP’s fiscal year 2006 procurement strategy. However, the council expressed concern about several issues that should be resolved prior to the Investment Review Board’s review, including the need for complete lifecycle cost estimates for the checkpoint screening technologies that were to be developed and procured. TSA officials acknowledged that completing lifecycle cost estimates are important and stated that they have not prepared a lifecycle cost estimate since the council recommended that such an estimate be developed due to lack of staff. These officials further stated that TSA hired four full-time equivalent staff in fiscal year 2008, and two additional full-time equivalent staff are expected to be hired in the fall of 2008. The officials anticipate that these staff will help prepare lifecycle cost estimates. However, the officials did not provide a timeframe for the completion of the estimates.

Although TSA officials identified the technologies they are procuring and deploying, TSA officials could not provide us with information on their priorities for the research and development of checkpoint screening technologies or the processes they followed to develop these priorities. According to S&T officials, TSA provided priorities for near-term applied research and development projects to the S&T Capstone Integrated Product Team (IPT) for Explosives Prevention. This IPT establishes priorities for research projects to be funded by S&T during the fiscal year. S&T officials stated that they rely on TSA and other members of the IPT to use a risk-based approach to identify and prioritize their agencies’ or offices’ individual research and development needs prior to submitting them for consideration to the IPT. However, TSA officials stated they did not submit priorities for research and development to S&T. Without cost– benefit or other analysis to compare the cost and effectiveness of various solutions, the agency cannot determine whether investments in the research and development of new checkpoint technologies or procedures most appropriately mitigate risks with the most cost-effective use of resources. In addition, without knowing the full cost of the technologies that the PSP is developing, procuring, or deploying, TSA could potentially invest in a technology in which the cost outweighs expected benefits.

TSA Lacks Measures to Evaluate the Extent to Which the PSP Reduces the Risk of Terrorist Attacks

TSA’s strategy for the PSP does not have a mechanism—such as performance measures or other evaluation methods—to monitor, assess, or test the extent to which investments in new checkpoint technologies reduce or mitigate the risk of terrorist attacks. The NIPP requires that protective programs be designed to allow measurement, evaluation, and feedback based on risk mitigation so that agencies may re-evaluate risk after programs have been implemented and take corrective action if needed, such as modifying existing programs to counter new risks or implementing alternative programs. The NIPP identifies three types of performance measures—descriptive, process/output, and outcome measures—that can help gauge the effectiveness of protective programs. Although the NIPP requires that protective programs be designed to allow measurement, evaluation, and feedback based on risk mitigation, TSA has not identified quantifiable measures of progress which would allow the agency to assess the PSP’s overall effectiveness. TSA officials stated that they do not have overall performance measures but are currently developing performance goals and measures for the overall program. However, the officials could not provide a time frame for their completion. In September 2004, we recommended that TSA complete strategic plans for its research and development programs which contain measurable objectives. Without measures to monitor the degree to which the TSA’s investments in the research, development, and deployment of new screening technologies reduce or mitigate terrorist threats, the agency is limited in its ability to assess the effectiveness of the PSP or the extent to which it complements other layers of security at the checkpoint.

Ten New Checkpoint Screening Technologies Are in Various Phases of RDT&E, Procurement, and Deployment, but ETP Deployment Has Been Halted

Since TSA’s creation in 2001, 10 new checkpoint screening technologies, including the ETP, have been in various phases of RDT&E, procurement, and deployment, but TSA halted deployment of the ETP due to performance problems and high installation costs. Of the 10 technologies, TSA has initiated deployments for 4 of them, including the ETP and a Bottled Liquids Scanner, but TSA has not deployed any of the 4 technologies to airports nationwide. TSA also initiated procurements of two technologies, including the Whole Body Imager; however, deployment of these two technologies has not begun yet. Four checkpoint technologies are in research and development, such as a shoe scanning device. In June 2006, 6 to 11 months after TSA began to deploy the ETPs to airports, the agency halted their deployment due to performance problems—the machines broke down more frequently than specified by the functional requirements and the machines were more expensive to install and maintain in airports than expected. Because TSA did not follow its acquisition guidance that recommends technologies be tested and evaluated in an operational setting prior to procurement and deployment, the agency lacked assurance that the ETPs performed as required by the system’s requirements. Although TSA officials were aware that tests conducted on earlier ETP models during 2004 and 2005 suggested that they did not operate reliably in an airport environment and that the ETP models that were subsequently deployed to airports had not been tested in an operational environment to prove their effectiveness, TSA deployed the ETPs to airports beginning in July 2005 for the Smiths Detection ETP and beginning in January 2006 for the General Electric ETP without resolving these issues. TSA officials stated that they deployed the ETPs to respond quickly to the threat posed by a potential suicide bomber after suicide bombings had been carried out onboard Russian airliners in 2004. TSA officials stated that they plan to continue to use the 90 ETPs currently deployed to airports. Because the ETPs were deployed without resolving their performance problems and validating all of the functional requirements, the ETPs have not been demonstrated to increase security at the checkpoint. In the future, using validated technologies would enhance TSA’s efforts to improve checkpoint security.

S&T and TSA Investments in RDT&E Resulted in the Procurement or Deployment of Six New Checkpoint Technologies

As a result of S&T and TSA investments in the RDT&E of checkpoint screening technologies since TSA’s creation in 2001, six new screening technologies are being procured and/or deployed, while four checkpoint screening technologies are currently in the research and development phase. Based on S&T and TSA RDT&E efforts, the agency has initiated deployments of four technologies—the ETP, Fido PaxPoint Bottled Liquids Scanner, Advanced Technology Systems, and Cast and Prosthesis Scanner—three of which originated as commercial-off-the-shelf technologies or commercial-off-the-shelf technologies that TSA modified for use as checkpoint screening devices. However, TSA has not completed the deployment for all of these four technologies to airports nationwide. TSA officials stated that they did not deploy additional checkpoint screening technologies because they were primarily focused on deploying explosives detection systems to screen checked baggage, as mandated by ATSA. TSA has also initiated procurements of two additional technologies—Automated Explosives Detection System for Carry-on Baggage and Whole Body Imager—but has not deployed either of them yet. Figure 3 describes the status of the six checkpoint screening technologies for which TSA has initiated procurement and/or deployment.

According to TSA’s August 2008 strategic plan for checkpoint technologies, there are several other ongoing efforts in addition to the technologies discussed in figure 3. S&T and TSA are researching and developing a shoe scanning device that is to conduct automated weapons and explosive detection without requiring passengers to remove their footwear. TSA plans to award a contract in fiscal year 2010, with full operating capability in fiscal year 2015. TSA plans to deploy 1,300 units at all category X through category IV airports. TSA also has two ongoing efforts related to boarding pass and credential authentication, according to the agency’s strategic plan. Starting in 2007, TSA assumed responsibility from airline contractors for travel document checking, which is currently conducted manually. TSA plans to replace the manual system with an automated one. Specifically, the Boarding Pass Scanning System is expected to verify the authenticity of a boarding pass at the checkpoint and enable the use of paperless boarding passes by the airlines. In addition, the Credential Authentication Technology System is planned to be an automated system that authenticates identification presented by passengers and airport employees. According to TSA, the agency plans to eventually combine both of these authentication systems in a single travel document checking system. TSA plans to award a contract for these two systems in fiscal year 2009, with full operating capability expected in fiscal year 2014. TSA plans to deploy a total of 878 units to replace the existing document verification tools at all category X through category IV airports. Another ongoing effort identified in TSA’s strategic plan is the Next Generation ETD. This system is planned to replace legacy ETD systems and to be able to identify a larger range of explosives. Specifically, this system is expected to have enhanced explosive detection capability in terms of sensitivity and the ability to detect new threats, as well as other improvements over legacy systems, which are expected to produce lower lifecycle costs. TSA plans to deploy 1,500 units at all category X through category IV airports.

TSA also has two additional efforts to assess possible technologies. One effort is called Standoff Detection, which is intended to display images to detect anomalies concealed under passengers’ clothing. TSA plans to conduct an operational utility evaluation of test article units during fiscal year 2009 to evaluate the technology’s feasibility within checkpoint screening operations. According to TSA, this technology would assist the agency in applying layered security prior to the checkpoint in soft target areas, such as airport lobbies, to improve early awareness of a potential explosive threat. If the technology proves effective in the checkpoint operation, TSA plans to award a contract in fiscal year 2010, with full operational capability expected by fiscal year 2014, and to deploy 351 units to every checkpoint at category X and category I airports. The other effort is called Explosives Characterization for Trace (Chemical-based) Detection. This effort includes the research and development of trace signatures, detection, and physical properties of explosives to improve the detection and performance of deployed explosives trace detection technologies.

TSA Procured and Deployed ETPs without Assurance that They Would Perform as Intended in an Operational Setting

During 2004 and 2005, prior to deployment of the ETPs, TSA conducted a series of acceptance tests (that is, laboratory tests) of the General Electric and Smiths Detection ETPs that suggested they had not demonstrated reliable performance. Specifically, in 2004, TSA conducted acceptance tests on early models of the General Electric and Smiths Detection ETPs to determine whether the ETPs met key functional requirements. Subsequently, in 2004 a General Electric ETP model was field tested at five airports to determine how well the ETP performed in an operational environment. A Smiths Detection ETP model was also field tested at an airport in 2004. Based on initial test results, both vendors of the ETPs modified the machines, and TSA conducted further laboratory testing. The modified General Electric ETP was tested from December 2004 through February 2005. During the January 2005 to May 2005 time frame, both the General Electric and Smiths Detection ETP models were tested. Even though tests conducted during 2004 and 2005 of the General Electric and Smiths Detection ETPs suggested they had not demonstrated reliable performance, TSA deployed the Smiths Detection ETP and General Electric ETP to airports starting in July 2005 and January 2006, respectively, without resolving identified performance issues.

Further, TSA did not test all 157 of the ETP’s functional requirements prior to procuring and deploying the General Electric and Smiths Detection ETP models. Instead, TSA tested the ETP models against a subset of the functional requirements. According to TSA’s System Development Life Cycle Guidance, testing of a system is to be conducted to prove that the developed system satisfies its requirements in the functional requirements document. TSA officials could not identify the specific requirements that were tested or the reason(s) that all of the requirements were not tested.

A TSA official stated that TSA had intended to resolve problems regarding the ETPs’ performance after they had been deployed, but TSA officials could not explain how these problems were to be resolved. Officials further stated that they worked for over 1 year during 2006 and 2007 with the ETP vendors to correct reliability and maintenance issues after the ETPs were initially deployed, but could not resolve them. Furthermore, according to S&T officials, when TSA conducted limited field tests, the ETP manufacturers provided different configurations from those used during the laboratory tests. According to officials, once this was discovered, it took more than 6 months for the ETP manufacturers to recreate the configurations that had passed the laboratory tests. TSA officials stated that, during this 6-month period, the agency decided to award a sole source contract to General Electric to procure its ETP.

Regarding the reliability of the ETPs, of the 101 ETPs (71 from General Electric and 30 from Smiths Detection) that were originally deployed to 36 airports, the General Electric ETP did not meet the system requirement for operational availability due to frequent breakdowns. Both vendors’ ETPs were also more expensive to maintain than expected, according to the TSA Chief Technology Officer serving during this period. The functional requirements document requires the ETP to be operationally available 98.38 percent of the time. However, the General Electric ETPs were not always able to meet this requirement. TSA officials could not provide information on the operational availability of the Smiths Detection ETPs. For the General Electric ETPs, from January through May 2006, they were operationally available an average of 98.05 percent of the time, although the ETPs met the operational availability requirement for 2 months during that period. Furthermore, TSA’s operational requirements specify that the ETP should function for a minimum of 1,460 hours between critical failures. A critical failure means that an ETP fails to operate and must be repaired as soon as possible. However, the TSA Chief Technology Officer at the time stated that the ETPs operated at a much lower average number of hours before a critical failure occurred because, for example, the dirt and humidity of some airport environments adversely affected the equipment. Specifically, from January 2006 through May 2006, the General Electric ETPs operated for an average of 559 hours before a critical failure, which means that these ETPs operated on average 38 percent of the time that they were required to operate before a critical failure occurred. TSA officials could not provide information on the mean time between critical failures for the Smiths Detection ETPs. TSA officials stated that they tested the ETPs in several airports for several months prior to deployment, but data from these tests did not identify a problem with mean time between critical failures. One reason for this, a TSA official stated, was that not enough data were collected during the field tests. As usage of the ETPs increased, officials stated that they discovered the ETP was not meeting operational availability requirements. The ETPs also required replacement filters and other consumables more often than expected, according to officials, which drove up maintenance costs.

According to TSA officials, because of a variance in operational availability hours among the deployed ETPs, maintenance problems, and the high cost of ETP installation at airports, in June 2006, the agency halted the deployment of the ETP to additional airports and stopped the planned purchase of additional ETPs. TSA officials plan to continue to use the 90 ETPs currently deployed to airports. However, without validating that the ETPs meet their functional requirements, TSA officials do not have assurance that it is worthwhile to continue to use the ETPs in light of the cost to maintain and operate them. In addition, TSA officials are considering what to do with the ETPs that were procured and are currently in storage. As of April 2009, 116 ETPs were in storage.

TSA did not follow the Acquisition Management System (AMS) guidance or a knowledge-based acquisition approach before procuring the ETPs, which contributed to the ETPs not performing as required after they were deployed to airports. Specifically, AMS guidance provides that testing should be conducted in an operational environment to validate that the system meets all functional requirements before deployment. In addition, our reviews have shown that leading commercial firms follow a knowledge-based approach to major acquisitions and do not proceed with large investments unless the product’s design demonstrates its ability to meet functional requirements and be stable. The developer must show that the product can be manufactured within cost, schedule, and quality targets and is reliable before production begins and the system is used in day-to-day operations. As discussed earlier in this report, TSA officials told us that they deployed the ETP despite performance problems because officials wanted to quickly respond to emergent threats. However, TSA did not provide written documentation to us that described the process used at the time to make the decision to deploy the ETP or the process that is currently used to make deployment decisions.

Using Validated Technologies Would Enhance TSA’s Efforts to Improve Checkpoint Security

TSA has relied on technologies in day-to-day airport operations that have not been demonstrated to meet their functional requirements in an operational environment. For example, TSA has substituted existing screening procedures with screening by the Whole Body Imager even though its performance has not yet been validated by testing in an operational environment. In the future, using validated technologies would enhance TSA’s efforts to improve checkpoint security. Furthermore, without retaining existing screening procedures until the effectiveness of future technologies has been validated, TSA officials cannot be sure that checkpoint security will be improved.

DHS Is Addressing Coordination and Collaboration Challenges with Stakeholders to Research, Develop, and Deploy Checkpoint Screening Technologies

DHS S&T and TSA coordinated and collaborated with each other and key stakeholders on their research, development, and deployment activities for airport checkpoint screening technologies, and DHS is taking actions to address challenges and strengthen these efforts. Because S&T and TSA share responsibilities related to the RDT&E, procurement, and deployment of checkpoint screening technologies, the two organizations must coordinate with each other and external stakeholders, such as airport operators and technology vendors. For example, in accordance with provisions of the Homeland Security Act and ATSA, S&T and TSA are to coordinate and collaborate with internal and external stakeholders on matters related to technologies and countermeasures for homeland security missions. S&T and TSA signed an MOU in August 2006 that establishes a framework to coordinate their work at the TSL, which tests and evaluates technologies under development. S&T also established a Capstone IPT for Explosives Prevention in 2006 to bring S&T, TSA, and U.S. Secret Service leadership together to identify gaps in explosives detection capability; prioritize identified gaps; review relevant, ongoing S&T programs; and develop capabilities to meet identified needs. However, inconsistent communication and the lack of an overarching test and evaluation strategy have limited S&T’s and TSA’s ability to coordinate effectively with one another. To coordinate with the aviation community, S&T and TSA have hosted industry days and conference calls to discuss new technologies with airport operators and technology vendors. Although TSA has taken actions to build partnerships with airport operators and vendors, it has not established a systematic process to coordinate with them related to checkpoint screening technologies. However, TSA officials stated that they are in the beginning stages of establishing a systematic process.

S&T and TSA Are Addressing Coordination and Collaboration Challenges with Each Other on New Checkpoint Screening Technologies, but Challenges Remain

S&T and TSA have taken actions to coordinate and collaborate with each other related to the RDT&E of checkpoint screening technologies, such as by communicating priorities and requirements for technologies and working with each other on the Capstone IPT for Explosives Prevention. However, S&T and TSA coordination and collaboration were not always effective due to inconsistent communication and the lack of an overarching test and evaluation strategy. The Homeland Security Act assigned responsibilities within the department for coordinating and integrating the research, development, demonstration, testing, and evaluation activities of the department, as well as for working with federal and private sector stakeholders to develop innovative approaches to produce and deploy the best available technologies for homeland security missions. The act further assigned S&T with responsibility for coordinating with other appropriate executive agencies in developing and carrying out the science and technology agenda of the department to reduce duplication and identify unmet needs. ATSA had also assigned TSA with coordination responsibilities, including the coordination of countermeasures with appropriate departments, agencies, and instrumentalities of the U.S. government.

S&T and TSA have taken several actions to coordinate and collaborate on their research and development activities related to checkpoint screening technologies. First, to coordinate the transition of the TSL from TSA to S&T, minimize disruption of work, and prevent duplication of effort, S&T and TSA signed an MOU that defines the roles and responsibilities for the research and development of homeland security technologies, including checkpoint screening, and establishes a framework for how to coordinate their work. Additionally, S&T created the Capstone IPT for Explosives Prevention, which is co-chaired by the Assistant Secretary for TSA and the Director of the U.S. Secret Service, to identify and prioritize capabilities needed to detect explosives; review relevant, ongoing S&T programs; and develop capabilities to meet the identified needs. The IPT was first convened in December 2006 to identify research and development priorities for explosives detection technologies at airport checkpoints as well as for other transportation modes, and has met periodically since then. According to TSA officials, the Capstone IPT has enabled TSA to establish a clear understanding with S&T of TSA’s needs for technology solutions that meet stringent detection thresholds and throughput requirements to support the aviation sector. Additionally, the officials stated that the Capstone IPT has given TSA a better collective understanding of the technology needs of other DHS components, which will help DHS identify technology solutions that can be combined to benefit multiple users. Finally, to follow through on the priorities established by the Capstone IPT for Explosives Prevention, S&T officials stated that they established project-level IPTs, including one for airport checkpoints and one for homemade explosives. S&T officials stated that they are working with TSA on these project-level IPTs to try to meet the needs identified by the Capstone IPT. TSA officials further stated that they have PSP IPTs or working groups to coordinate on technology projects, establish program goals and objectives, and develop requirements and time lines. These groups meet on a weekly basis, according to TSA officials. In April 2008, S&T dissolved the IPT for explosives detection and replaced it with two separate IPTs, a transportation security IPT, chaired by TSA and a counter-IED IPT, chaired by the Office of Bombing Prevention within the National Protection and Programs Directorate and the United States Secret Service.

Coordination and collaboration efforts between S&T and TSA have helped in identifying checkpoint screening solutions. For example, S&T and TSA officials collaborated on a hand-held vapor detection unit called the Fido PaxPoint. After the August 2006, discovery of the alleged plot to detonate liquid explosives on board commercial air carriers bound for the United States from the United Kingdom, S&T and TSA worked together to identify, develop, and test screening technologies to address this threat. According to TSA officials, S&T learned that the Department of Defense had developed a handheld unit that could detect vapors from explosives. S&T modified the Department of Defense handheld unit, resulting in the Fido PaxPoint unit to screen liquids and gels at airport checkpoints for explosives, and S&T helped TSA test and evaluate the device.

Although S&T and TSA have taken steps to coordinate and collaborate with one another, inconsistent communication and a lack of an overarching test and evaluation strategy have contributed to coordination and collaboration challenges. Specifically, communication between S&T and TSA related to S&T’s basic and applied research efforts and TSA’s efforts to modify commercially available technologies has been lacking at times. For example, TSA officials stated that early in the TSL’s transition to S&T (that is, during fiscal year 2006), TSA did not receive information from S&T regarding which of TSA’s research and development needs S&T would fund, which projects related to airport checkpoint technologies were underway at the TSL, or the time frames to complete those projects. TSA officials stated that, without this information, TSA was unable to determine whether its work on modifying commercially available technologies for screening passengers and carry-on items unnecessarily duplicated S&T’s research and development efforts, although TSA officials were not aware of any duplication that occurred. An S&T official further stated that TSA had not consistently fulfilled its responsibility to provide clearly defined functional requirements for the equipment to be developed by S&T and tested by the TSL, nor has TSA consistently given sufficient notice to the TSL of TSA testing requests. Under the S&T and TSA MOU, TSA has retained responsibility to establish requirements for equipment certification and qualification and acceptance testing. Specifically, an S&T official at the TSL stated that TSA had inadequately defined the functional requirements and allowed too little time for testing several checkpoint screening technologies, including the Advanced Technology Systems, Enhanced Metal Detector II, and Bottled Liquids Scanner. A TSL official acknowledged that when the TSA was responsible for the TSL, the agency had not consistently developed requirements prior to testing or certification of equipment as required by the DHS guidance.

In another example, as previously mentioned in this report, TSA is developing new certification standards and functional requirements for screening technologies, and is working with national laboratories to validate data on aircraft vulnerabilities and generate new computer models to help TSA develop requirements for explosives detection. According to the TSA Chief Technology Officer in 2007, the TSL has custody of the aircraft vulnerability data, but TSL officials had refused to release the data to the national laboratories as requested by TSA. Although the TSL later provided 32 of the 46 requested reports, TSA officials estimated that the TSL’s refusal to release all of the reports had delayed the effort to develop new certification standards and technology requirements by about 1 month. The officials added that most of TSA’s requests to S&T and the TSL had involved similar problems and that, although the MOU provides a framework for coordination, these types of problems are related to day-to-day operations and will have to be resolved as situations arise.

According to S&T and TSA officials, senior-level management turnover at S&T and TSA contributed to these communication difficulties, as well as an S&T reorganization which began in August 2006 with the arrival of a new Under Secretary for Science and Technology. S&T officials further stated that, prior to the establishment of the PSP working groups, there was no mechanism for S&T and TSA to communicate information about priorities, funding, or project timelines. However, through the working groups, S&T officials stated that S&T and TSA are beginning to achieve regular communication and interaction at the working level, which allows for information to be shared in a mutually beneficial way. S&T and TSA officials also stated that communication with each other has improved since the MOU was signed in August 2006 and, in particular since the summer of 2007, although officials from both organizations stated that further improvement is needed. According to S&T officials, the TSL’s independent test and evaluation division and TSA have developed an effective working relationship for several programs, including the Whole Body Imager and Advanced Technology Systems. In addition, S&T officials stated that TSA has come to better understand the processes involving the Capstone IPT and identifying capability needs. According to TSA officials, the agency is in the process of determining whether a position within its Office of Security Technology should be established as a liaison with S&T to improve coordination between S&T and TSA. If the position is created, the TSA liaison would coordinate and collaborate with S&T officials on technology projects by assessing the science that supports the technologies.

The MOU specifies that S&T and TSA will coordinate activities, including developing an integrated, overarching test and evaluation strategy for projects to ensure that test and evaluation functions are not duplicative, adequate resources are outlined and secured for these functions, and activities are scheduled to support the overall project master schedule. However, an overarching test and evaluation strategy for checkpoint technologies has not been developed. The lack of this strategy has presented coordination and collaboration challenges between S&T and TSA, and has resulted in the delay of some technologies. For example, a TSL official stated that the TSL could not accommodate TSA’s request to test the Advanced Technology Systems, in part, because TSA officials had not provided sufficient advance notice of their testing needs. TSA officials said they were working with S&T to develop a project master schedule for the Advanced Technology Systems. S&T and TSA officials stated that they plan to develop a test and evaluation strategy to define a coordinated technology transition process from S&T to TSA by outlining key responsibilities and criteria to initiate field evaluations of technologies, but officials could not tell us when the test and evaluation strategy would be completed.

DHS Is Using Several Approaches to Strengthen Coordination and Collaboration with Airport Operators, Technology Vendors, and Other Federal Agencies

DHS, through S&T and TSA, coordinates with airport operators, private sector partners, such as technology vendors, and other federal agencies on matters related to research and development efforts. This coordination and collaboration between TSA and airport operators and technology vendors is important because the agency relies on airport operators to facilitate the deployment of equipment for testing and day-to-day operations, and on vendors to develop and manufacture new screening equipment. However, TSA does not have a systematic process to coordinate with external stakeholders related to checkpoint screening technologies, but TSA officials stated that the agency has developed a draft communications plan, which is being reviewed.

Although TSA does not have a systematic process to coordinate with technology vendors, airport operators, and other stakeholders related to the RDT&E, procurement, and deployment of checkpoint screening technologies, agency officials stated that they plan to develop and implement such a process. Specifically, TSA officials stated that they have developed a draft communications plan, which is being reviewed, that will document the communications process. However, TSA could not provide an expected completion date for the plan. Although such a plan should help in providing consistency to the agency’s coordination efforts, without knowing the specific activities the plan will include or when it will be implemented, we cannot determine the extent to which the plan may strengthen coordination. In addition, in September 2007, TSA hired an Industry Outreach Manager within its Office of Security Technology to improve relationships with airport operators and communication with internal TSA stakeholders related to screening technologies, including checkpoint technologies. In general, the Industry Outreach Manager is the communications liaison for the Office of Security Technology stakeholders and customers to exchange ideas, information, and operational expertise in support of the office’s mission and goals, and to provide cutting-edge technologies in the most efficient and cost-effective means possible. In addition to these steps, in January 2007, S&T created a Corporate Communications Division to coordinate on a wide variety of science and technology efforts with public and private sector stakeholders. This office is in the process of developing a tool to assess the effectiveness of its outreach efforts to industry stakeholders.

The AMS guidance recommends that TSA coordinate with airport operators to work out all equipment installation issues prior to deployment. According to TSA officials, the role of the airport operator is essential in ensuring that solutions under development are suitable for use in an airport environment, taking into consideration all logistical and operational constraints and possibilities. As described earlier, provisions of the Homeland Security Act address the need to coordinate research and development efforts to further homeland security missions, and reinforce the importance of coordinating and collaborating with airport operators. TSA sponsors monthly conference calls with airport operators to discuss issues of general interest and, according to S&T officials, S&T has conducted pilot studies with airport operators. However, according to many of the 33 airport operators we interviewed, TSA’s coordination on the priorities for and deployment of checkpoint screening technologies has been inconsistent. Specifically, of the 33 airport operators we interviewed, 8 had only positive comments about TSA’s coordination and 16 expressed only concerns regarding TSA’s coordination efforts, while 9 expressed both positive comments and concerns. Eleven of the 33 airport operators told us that TSA had not shared information with them regarding checkpoint technology needs and priorities. For example, an airport operator stated that TSA provided specifications for new screening technologies with sufficient lead time for the airport, which was building a new checkpoint at the time, and that TSA had numerous coordination meetings with airport officials to determine space constraints, power requirements, and other factors. However, this same airport operator expressed a desire for more coordination by TSA in the agency’s selection of the technologies to be pilot tested at this airport. Another airport operator stated that, when TSA asks for volunteers to participate in checkpoint screening technology pilot programs, it is difficult to agree to participate because TSA does not clearly communicate the program’s goals or the capabilities of the technology in the pilot program.

According to airport operators at another airport, TSA officials told them that they would have the latitude to select the ETP from either of two vendors on the TSA contract for purchase. According to the airport officials, after they selected equipment from one of the vendors because it would fit into the physical layout of the airport’s checkpoint, TSA told the airport officials that particular ETP vendor was no longer under contract with TSA. As a result, airport officials stated that they had to redesign the checkpoint, including raising the ceiling, to accommodate the other vendor’s ETP. Senior officials in TSA’s Office of Operational Process and Technology, the office responsible for the development and implementation of security technologies across several modes of transportation, subsequently agreed that coordination with airport managers and other stakeholders could be improved.

According to TSA officials, coordinating with technology vendors is essential in order to determine what technology platform would be appropriate and capable of providing the required detection and throughput capabilities. S&T and TSA have conducted outreach efforts to coordinate with technology vendors. For example, S&T officials stated that they have hosted forums known as industry days and attended conferences to discuss types of technologies needed to be developed and the department’s priorities for research and development. S&T officials also stated that they make presentations at technology-related conferences, symposia, and exhibits, highlighting the work conducted by S&T. At every industry day and conference, officials said, airport security and checkpoint screening technologies have been discussed. In addition, TSA has coordinated with technology vendors through industry days, individual meetings, and conferences. For example, TSA officials stated that TSA held industry days with technology vendors to provide a forum to communicate information to potential vendors on specific technology testing and procurement efforts, and to allow vendors to ask questions regarding technology projects and TSA expectations.

Despite these outreach efforts, of the seven vendors we interviewed who had contracted with TSA to provide checkpoint screening technologies, officials from five vendors expressed concerns about the agency’s ability to coordinate with them on current or future needs for checkpoint technologies. Officials from four of the seven vendors stated that TSA had not communicated a strategic vision for screening technologies that will be needed at the checkpoint in the future, and that TSA did not effectively and clearly communicate standards and requirements for technologies to vendors. For example, just as TSL officials commented that TSA did not always provide clear and quantifiable requirements to conduct tests of screening technologies, vendors stated that TSA had not communicated effectively about its future needs, such as the operational requirements for an advanced, integrated checkpoint screening system. Therefore, a vendor official stated that some of them had taken the initiative to develop integrated screening technologies in the hope that TSA will eventually request this type of integrated system. TSA did not express an opinion regarding the specific concerns raised by the technology vendors, but a senior TSL official stated that TSA should sponsor better briefings for vendors after the agency announces its intentions to develop new technologies. The official stated that these briefings could provide vendors with an opportunity for open dialogue with TSA and clarification of TSA’s needs for new technologies. According to a vendor, without adequate coordination and communication from TSA, the vendors’ ability is limited in deciding how best to invest their resources to develop new checkpoint screening technologies.

In addition to coordinating and collaborating with airport operators and technology vendors, S&T and TSA coordinate and collaborate on the department’s RDT&E efforts with other federal agencies through participation in the Technical Support Working Group, which is co-chaired by the Departments of Defense and State. The Technical Support Working Group is the U.S. national forum that identifies, prioritizes, and coordinates interagency research and development of technologies to combat terrorist acts, including explosives detection technologies. S&T also coordinates with the national laboratories on homeland security research. Specifically, S&T’s Office of National Laboratories coordinates homeland security-related activities and laboratory-directed research conducted within the Department of Energy’s national laboratories. According to an S&T senior official, S&T has worked with the national laboratories to supplement S&T’s research and development of explosives detection technologies by tasking the national laboratories to conduct basic research on the characteristics of homemade explosives.

Conclusions

Researching, developing, testing and evaluating, procuring, and deploying checkpoint technologies capable of detecting ever-changing threats to the commercial aviation system is a daunting task. Although TSA has recently produced a strategic plan that identified a strategy for the PSP, neither the plan nor the agency’s strategy for researching, developing, and deploying checkpoint technologies was informed by some key risk management principles, including a risk assessment, cost–benefit analysis, and performance measures. Without conducting a risk assessment that includes all three elements of risk—threat, vulnerability, and consequence—and completing a cost–benefit analysis to guide the PSP strategy, TSA has limited assurance that its strategy targets the most critical risks and that it invests in the most cost-effective new technologies or other protective measures. Further, without developing performance measures that assess the extent to which checkpoint screening technologies achieve the PSP’s security goals and thereby reduce or mitigate the risk of terrorist attacks, TSA is limited in its ability to determine the success of its strategy and make needed adjustments. Even though TSA has not implemented a risk-informed strategy to ensure that its investments target the most pressing security needs, the agency has moved forward in investing in new checkpoint screening technologies.

Despite limited progress in the RDT&E, procurement, and deployment of new checkpoint screening technologies during the first few years that S&T and TSA had responsibilities related to these technologies, more recently, the organizations have made progress as reflected by the number of technologies for which procurement and deployment has been initiated.

TSA faced challenges with the first new technology that it procured and deployed—the ETP. In the interest of protecting the homeland, it is understandable that TSA may, at times, not follow all established guidance in an effort to deploy technologies quickly to address urgent threats and vulnerabilities. However, deploying the ETP despite unresolved performance concerns identified during testing of earlier ETP models, as well as failing to ensure that ETP models that were ultimately deployed had passed operational testing, increased the risk that the machines would not perform as intended, resulting in a questionable security benefit. TSA did not follow AMS guidance that recommended operational testing of a new technology prior to deployment because it is more cost effective to resolve performance issues then. While TSA deployed the ETPs to provide a much-needed capability to automatically screen higher risk passengers at airport checkpoints, relying on the ETPs could have resulted in airport checkpoints being more vulnerable given the ETPs’ performance problems and lack of operational testing. Also, relying on the ETPs to screen these particular passengers instead of existing screening procedures may not enhance airport checkpoint security because TSA does not know if ETP screening provides an improved detection capability compared to existing screening procedures. Moreover, it is risky to substitute any new technology for existing screening procedures before the technology has been proven to be effective through operational testing. Although TSA is trying to deploy new technologies to address immediate threats, the problems associated with the development and deployment of the ETPs may be repeated with other technologies unless TSA adheres to testing guidance and makes decisions using a knowledge-based acquisition approach. Finally, it is not clear whether it is worthwhile to continue to use the ETPs currently deployed to airports due to the costs associated with maintaining the machines in good, operational condition.

Recommendations for Executive Action

To help ensure that DHS’s Science and Technology Directorate (S&T) and Transportation Security Administration (TSA) take a comprehensive, risk- informed approach to the RDT&E, procurement, and deployment of airport passenger checkpoint screening technologies, and to increase the likelihood of successful procurements and deployments of such technologies, in the restricted version of this report, we recommended that the Assistant Secretary for TSA take the following eight actions: Conduct a complete risk assessment, including threat, vulnerability, and consequence assessments, which would apply to the PSP.

Develop cost–benefit analyses to assist in prioritizing investments in new checkpoint screening technologies.

Develop quantifiable performance measures to assess the extent to which investments in research, development, and deployment of checkpoint screening technologies achieve performance goals for enhancing security at airport passenger checkpoints.

After conducting a complete risk assessment and completing cost-benefit analyses and quantifiable performance measures for the PSP, incorporate the results of these efforts into the PSP strategy as determined appropriate.

To the extent feasible, ensure that operational tests and evaluations have been successfully completed before deploying checkpoint screening technologies to airport checkpoints.

Evaluate whether TSA’s current passenger screening procedures should be revised to require the use of appropriate screening procedures until it is determined that existing emerging technologies meet their functional requirements in an operational environment. In the future, prior to testing or using all checkpoint screening technologies at airports, determine whether TSA’s passenger screening procedures should be revised to require the use of appropriate screening procedures until the performance of the technologies has been validated through successful testing and evaluation.

Evaluate the benefits of the Explosives Trace Portals that are being used in airports, and compare the benefits to the costs to operate and maintain this technology to determine whether it is cost-effective to continue to use the machines in airports.

Agency Comments and Our Evaluation

We provided a draft of our restricted report to DHS for review and comment. On April 7, 2009, DHS provided written comments, which are presented in Appendix II. In commenting on our report, DHS stated that it agreed with our recommendations and identified actions planned or underway to implement them. While DHS is taking steps to address our first and second recommendations related to conducting a risk assessment, the actions DHS reported TSA had taken or plans to take do not fully address the intent of the remaining six recommendations.

In its comments, DHS stated that it concurred with our first recommendation that a risk assessment should be developed for the PSP and that TSA has two efforts currently underway to do so. Completion of TSA’s first effort—the Air Domain Risk Analysis (ADRA)—is expected in the winter of 2009. DHS commented that TSA’s second effort is the Risk Management and Analysis Toolset (RMAT), a model to simulate the potential of some technologies to reduce the risk of certain threat scenarios which will apply specifically to the passenger screening process. DHS reported that it expects initial results from RMAT to be available during the second quarter of 2009. DHS further stated that TSA has made resource allocation and technology decisions that were informed by consideration of risk (including threat, vulnerability, and consequence), although not by comparative assessments of these three elements. However, as we reported, TSA has not conducted a risk assessment for the PSP, and it is unclear to what extent the ADRA would provide risk information needed to support the PSP. Until such a risk assessment is developed and integrated into TSA’s strategy for the PSP, TSA continues to invest in checkpoint technologies without the benefit of a risk-informed strategy and increases the possibility that its investments will not address the highest-priority security needs.

DHS also concurred with our second recommendation that it develop cost- benefit analyses. DHS commented that TSA is developing an approach for selecting cost-effective technologies by developing life-cycle cost estimates and using the RMAT tool to determine how technologies balance risk (based on current threats) with cost. TSA’s decision to collect cost and benefit information is a positive first step. Irrespective of how TSA collects data on the costs and benefits of technologies, it is important, as we reported, that TSA conduct cost-benefit analysis of each checkpoint technology that it invests in that weighs the costs and benefits of technologies relative to the costs and benefits of other solutions. Such analysis is important because it helps decision-makers determine whether investments in technologies or in other security programs will provide the greatest mitigation of risk for the resources that are available.

DHS concurred with our third recommendation that TSA develop quantifiable performance measures to assess the extent to which TSA’s investments in checkpoint screening technologies make the checkpoint more secure, the key mission of the program. DHS commented that it currently collects quantifiable performance attributes for all potential acquisitions with regards to metrics, such as detection, false alarm rate, and operational availability and plans to use information on machines’ attributes as measures of the PSP’s overall effectiveness as a program. However, these actions will not fully address our third recommendation. First, information collected on potential acquisitions prior to their deployment may not reflect their performance in an operational environment; consequently, relying on information about technologies’ attributes rather than measuring the effectiveness of deployed technologies to secure the checkpoint will likely have limited value in terms of measuring the effectiveness of the PSP as a program. Second, as we reported, the ETP example illustrates that TSA did not collect information on the ETP’s performance attributes such as operational availability during laboratory testing prior to procurement and did not collect data on the ETP’s detection capabilities during tests in an operational environment. This raises questions about the completeness of data TSA collects on technologies prior to acquisition and deployment. We could not verify that TSA collects such information on other technologies because TSA did not provide documentation to support this comment. As TSA moves forward in developing performance measures, it is important that these measures reflect not only efficiency of the technologies to process passengers but the effectiveness of technologies and other countermeasures to make the checkpoint more secure and thereby reduce the risks posed by those most pressing threat scenarios that will be identified once TSA completes its risk assessment.

In addition, DHS concurred with our fourth recommendation that it develop a PSP strategic plan that reflects the risk assessment, cost benefit analysis, and performance measures. DHS commented that TSA plans to combine results from the RMAT tool and lifecycle cost estimates for possible technology solutions that strike a balance between risk and efficient use of funding. DHS also stated it will use RMAT to develop proxy measures and general “what-if” analysis and risk insights. However, these actions alone will not satisfy the intent of this recommendation. While it is possible that proxy measures could be developed to assess the extent to which TSA’s investments in the research and development of technologies have achieved program goals of making the checkpoint more secure, to fully address this recommendation, TSA must also conduct a risk assessment that addresses the PSP, develop quantifiable measures that clearly assess the PSP’s progress towards its security goals, and revise its strategic plan accordingly.

DHS concurred with our fifth recommendation that before deploying technologies to airport checkpoints, the technologies should successfully complete testing and evaluation and stated that TSA is taking action to implement a formal testing process. DHS commented that TSA has prepared a Test and Evaluation Master Plan (TEMP) that describes a new formal testing process that is consistent with DHS’s new acquisition directive. However, the TEMP does not address the intent of this recommendation. We deleted from this public report our evaluation of why the TEMP does not address the intent of this recommendation, because TSA determined our evaluation to be sensitive security information.

Further, DHS agreed with our sixth and seventh recommendations that TSA evaluate whether its screening procedures should be revised to require the use of appropriate procedures until it can be determined that emerging technologies or future technologies that may be developed meet all of their requirements in an operational environment. However, DHS’s comments suggest that it does not intend to implement these recommendations. DHS commented that the performance of machines is always measured and confirmed in the laboratory setting prior to operational field testing. However, we disagree that laboratory testing is sufficient to address this recommendation. We deleted from this public report our evaluation of why laboratory testing alone does not address the intent of this recommendation, because TSA determined our evaluation to be sensitive security information.

DHS stated that TSA implemented our eighth recommendation that the agency evaluate the benefits of the ETP, such as its effectiveness, and conduct a cost-benefit analysis to determine whether the technologies should remain in use at airports. However, we disagree that TSA has implemented this recommendation. DHS commented that two actions fulfilled this recommendation: TSA’s current program management reviews in which costs are periodically discussed with vendors and the laboratory testing of the ETP’s detection capabilities. To fully address this recommendation, a cost-benefit analysis and tests of the ETP’s effectiveness to detect explosives in an operational environment are required. As we reported, TSA has not conducted cost-benefit analyses, which, as noted earlier, should compare costs and benefits of alternative solutions. Discussions of maintenance costs with vendors on a periodic basis do not constitute a cost-benefit analysis.

Based on DHS’s written comments, we deleted a reference to the 2004 OMB PART review in a footnote because of updated information from OMB’s 2008 PART review. DHS also provided us with technical comments, which we considered and incorporated in the report where appropriate. In particular, we clarified the wording of a recommendation which originally stated that TSA should develop quantifiable performance measures to assess the extent to which investments in research, development, and deployment of checkpoint screening technologies have mitigated the risks of a terrorist attack. We altered the wording to state that performance measures should be developed to assess progress towards security goals.

As agreed with your offices, unless you publicly announce the contents of this report, we plan no further distribution for 45 days from the report date. At that time, we will send copies of this report to the Secretary of Homeland Security, the Assistant Secretary of the Transportation Security Administration, and appropriate congressional committees.

If you or your staffs have any questions about this report, please contact me at (202) 512-8777 or LordS@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

This report addresses the following questions: (1) To what extent has the Transportation Security Administration (TSA) developed a risk-informed strategy to prioritize investments in the research and development of passenger checkpoint screening technologies? (2) What new passenger checkpoint screening technologies has the Department of Homeland Security (DHS) researched, developed, tested and evaluated, procured, and deployed since its creation, and why did TSA halt the first technology deployment that it initiated—the Explosives Trace Portal (ETP)? (3) To what extent has DHS coordinated the research, development, test and evaluation (RDT&E), procurement, and deployment of passenger checkpoint screening technologies internally and with key stakeholders, such as airport operators and technology vendors?

To determine the extent to which TSA has developed a risk-informed strategy to prioritize investments in the research and development of passenger checkpoint screening technologies, we analyzed program documents, TSA’s August 2008 strategic plan for checkpoint technologies, TSA’s September 2007 report on the development of a strategic plan, technology project plans, and funding. We also compared TSA’s strategic plan and DHS’s responses regarding their efforts to manage their research and development investments, with DHS’s guidance from the National Infrastructure Protection Plan on how to utilize risk management principles to target funding.

To determine the extent to which DHS researched, developed, tested and evaluated, procured, and deployed new checkpoint screening technologies since its creation, and to identify why TSA halted deployment of the ETP, we analyzed TSA’s strategic plan for checkpoint technologies, TSA’s Passenger Screening Program (PSP) documentation, including information on the status of technologies being researched, developed, tested and evaluated, procured, and deployed. Regarding the ETPs, we analyzed the functional requirements for the system, contracts with General Electric and Smiths Detection, and test reports for acceptance tests, regression tests, and operational tests. We also reviewed ETP deployment schedules and documentation on operational availability and mean time between critical failure, and interviewed TSA officials about the reasons that the ETP deployment was halted. We also compared the ETP test approach used by S&T and TSA to the Acquisition Management System (AMS) guidance and knowledge-based acquisition best practices. We also interviewed TSA and S&T officials to obtain information on current investments in the research, development, and deployment of checkpoint technologies, and conducted site visits to the Transportation Security Laboratory in Atlantic City, New Jersey, and Tyndall Air Force Base, Florida, to observe testing of new checkpoint technologies. We visited the TSL because that is where S&T tests and evaluates technologies, including checkpoint screening technologies. We visited Tyndall Air Force Base because technologies to detect bottled liquids explosives were being tested there. Additionally, we analyzed TSA’s passenger screening standard operating procedures and interviewed various TSA headquarters officials, 29 Federal Security Directors, 1 Deputy Federal Security Director, and 5 Assistant Federal Security Directors for Screening, and visited nine airports where the ETPs had been or were to be deployed or new checkpoint screening technologies were undergoing pilot testing. We chose these officials because they are the senior official at the airport in charge of security and manage TSA’s role in deploying new technologies at the airport. We selected these nine locations based on the technologies that had been deployed or were being tested, their geography, size, and proximity to research and development laboratories. Of the nine airports we visited, the ETPs had been or were to be deployed to seven of them, and other new checkpoint screening technologies were undergoing pilot demonstrations or testing at two of them. We visited four airports on the east coast, and three airports on the west coast, and two airports located in the west and southwestern regions of the United States. To determine whether the ETP’s requirements had been tested prior to procuring and deploying them, we selected a non-probability sample of 8 out of the 157 total requirements. We selected the 8 requirements because they were related to some of the ETP’s key functionality requirements, including operational effectiveness, operational suitability, and passenger throughput.

To determine the extent to which DHS has coordinated and collaborated on the RDT&E, procurement, and deployment of passenger screening technologies internally and with key stakeholders, we analyzed program documents, including an August 2006 memorandum of understanding between TSA and S&T for the management of the Transportation Security Laboratory (TSL). Additionally, we interviewed Department of State officials, TSA and S&T officials, seven checkpoint technology vendors, and airport operators and other officials at airports where ETPs were initially deployed. Because we selected nonprobability samples of airports to visit and officials to interview, we cannot generalize the results of what we learned to airports nationwide. However, the information we gathered from these locations and officials provided us with insights and perspectives on DHS’s efforts to operationally test and evaluate, and deploy checkpoint technologies that could only be obtained from officials stationed at locations where the technologies had been tested or deployed. We reviewed the Acquisition Management System, the Aviation and Transportation Security Act, the Homeland Security Act of 2002, and the Intelligence Reform and Terrorism Prevention Act and identified requirements and guidance for coordination and collaboration among S&T, TSA, and other stakeholders. We also reviewed S&T’s and TSA’s coordination activities and compared them to TSA program guidance and GAO’s recommended coordination practices regarding agency coordination with external stakeholders.

We conducted this performance audit from June 2006 through April 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Homeland Security

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Robert Goldenkoff, Acting Director; E. Anne Laffoon and Steve Morris, Assistant Directors; and Joseph E. Dewechter, Analyst-in-Charge, managed this assignment. Carissa Bryant, Chase Cook, Orlando Copeland, Neil Feldman, and Ryan MacMaster made significant contributions to the work. Charles Bausell, Jr., Richard Hung, and Stanley Kostyla assisted with design, methodology, and data analysis. Michele Mackin assisted with acquisition and contracting issues. Sally Williamson, Linda Miller, and Kathryn Godfrey provided assistance in report preparation, and Thomas Lombardi provided legal support.