Aviation Security: DHS and TSA Have Researched, Developed, and Begun Deploying Passenger Checkpoint Screening Technologies, but Continue to Face Challenges
Published: 2009-10-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since fiscal year 2002, the Transportation Security Administration (TSA) and the Department of Homeland Security (DHS) have invested over $795 million in technologies to screen passengers at airport checkpoints. The DHS Science and Technology Directorate (S&T) is responsible, with TSA, for researching and developing technologies, and TSA deploys them. GAO was asked to evaluate the extent to which (1) TSA used a risk-based strategy to prioritize technology investments; (2) DHS researched, developed, and deployed new technologies, and why deployment of the explosives trace portal (ETP) was halted; and (3) DHS coordinated research and development efforts with key stakeholders. To address these objectives, GAO analyzed DHS and TSA plans and documents, conducted site visits to research laboratories and nine airports, and interviewed agency officials, airport operators, and technology vendors.
section_title: What GAO Found
paragraphs: TSA completed a strategic plan to guide research, development, and deployment of passenger checkpoint screening technologies; however, the plan is not risk-based. According to TSA officials, the strategic plan and its underlying strategy for the Passenger Screening Program were developed using risk information, such as threat information. However, the strategic plan and its underlying strategy do not reflect some of the key risk management principles set forth in DHS's National Infrastructure Protection Plan (NIPP), such as conducting a risk assessment based on the three elements of risk--threat, vulnerability, and consequence--and developing a cost-benefit analysis and performance measures. TSA officials stated that, as of September 2009, a draft risk assessment for all of commercial aviation, the Aviation Domain Risk Assessment, was being reviewed internally. However, completion of this risk assessment has been repeatedly delayed, and TSA could not identify the extent to which it will address all three elements of risk. TSA officials also stated that they expect to develop a cost-benefit analysis and establish performance measures, but officials could not provide timeframes for their completion. Without adhering to all key risk management principles as required in the NIPP, TSA lacks assurance that its investments in screening technologies address the highest priority security needs at airport passenger checkpoints. Since TSA's creation, 10 passenger screening technologies have been in various phases of research, development, test and evaluation, procurement, and deployment, but TSA has not deployed any of these technologies to airports nationwide. The ETP, the first new technology deployment initiated by TSA, was halted in June 2006 because of performance problems and high installation costs. Deployment has been initiated for four technologies--the ETP in January 2006, and the advanced technology systems, a cast and prosthesis scanner, and a bottled liquids scanner in 2008. TSA's acquisition guidance and leading commercial firms recommend testing the operational effectiveness and suitability of technologies or products prior to deploying them. However, in the case of the ETP, although TSA tested earlier models, the models ultimately chosen were not operationally tested before they were deployed to ensure they demonstrated effective performance in an operational environment. Without operationally testing technologies prior to deployment, TSA does not have reasonable assurance that technologies will perform as intended. DHS coordinated with stakeholders to research, develop, and deploy checkpoint screening technologies, but coordination challenges remain. Through several mechanisms, DHS is taking steps to strengthen coordination within the department and with airport operators and technology vendors.

section_title: Letter
paragraphs: Commercial aircraft have long been a target of terrorism at the hands of  hijackers and suicide bombers. The Transportation Security  Administration (TSA), the agency with primary responsibility for securing  the nation’s civil aviation system after the September 11, 2001, terrorist  attacks, has identified the need for improved technology to detect  explosives and other threat items at airport passenger screening  checkpoints to strengthen the nation’s defenses against acts of terrorism.  From fiscal years 2002 through 2008, over $795 million has been invested  by TSA and the Department of Homeland Security (DHS) for the research,  development, test and evaluation (RDT&E), procurement, and deployment  of checkpoint screening technologies.
TSA has implemented a multilayered system of security to protect  commercial aviation—the most publicly visible layer being the physical  screening of passengers and their carry-on items at airport screening  checkpoints. TSA’s passenger checkpoint screening system—located at all  airports regulated by TSA—is comprised of three elements: (1) the  personnel, or screeners, responsible for conducting the screening of  airline passengers and their carry-on items; (2) the procedures screeners  are to follow to conduct screening; and (3) the technology used during the  screening process. Collectively, these elements—the people, process and  technology—help to determine the effectiveness and efficiency of  passenger checkpoint screening. We previously reported that TSA had  made efforts to enhance its passenger checkpoint screening system by  strengthening screener training, measuring the performance of screeners  and the screening system, and modifying screening procedures to address  terrorist threats.
Within DHS, the Science and Technology Directorate (S&T) and TSA have  responsibilities for researching, developing, and testing and evaluating  new technologies, including airport checkpoint screening technologies.  Specifically, S&T is responsible for the basic and applied research and  advanced development of new technologies, while TSA, through its  Passenger Screening Program (PSP), identifies the need for new  checkpoint screening technologies, provides input to S&T during the  research and development of new technologies, which TSA then procures  and deploys.
In 2004, we reviewed DHS’s investments in the research and development  of technologies to secure the transportation sector, including aviation, and  found that DHS needed to strengthen the management of its research and  development efforts. In October 2007, we testified that a key challenge  related to securing the homeland involves allocating resources based on  risk. DHS and TSA leadership have identified that risk-informed  considerations will be a cornerstone of departmental and agency policy. In  particular, DHS’s National Infrastructure Protection Plan (NIPP) stated  that TSA should be considering risk management principles when  allocating funding for the research and development of security  technologies. According to the NIPP, security strategies should be  informed by, among other things, a risk assessment that includes threat,  vulnerability, and consequence assessments, information such as cost– benefit analyses to prioritize investments, and performance measures to  assess the extent to which a strategy reduces or mitigates the risk of  terrorist attacks.
In response to your request, this report provides the results of our review  of DHS’s efforts, through S&T and TSA, to research, develop, and deploy  emerging screening technologies for use at airport passenger checkpoints  by addressing the following questions: (1) To what extent has TSA  developed a risk-informed strategy to prioritize investments in the  research and development of passenger checkpoint screening  technologies; (2) What new passenger checkpoint screening technologies  has DHS researched, developed, tested and evaluated, procured, and  deployed since its creation, and why did TSA halt the first technology  deployment that it initiated—the explosives trace portal (ETP); and (3) To  what extent has DHS coordinated the RDT&E, procurement, and  deployment of passenger checkpoint screening technologies internally and  with key stakeholders, such as airport operators and technology vendors?
This report is a public version of a restricted report (GAO-09-21SU) that  we provided to you earlier this year. In this report in three cases we  provide updates regarding the Aviation Domain Risk Assessment (ADRA),  the NIPP, and the number of ETPs in airports. DHS and TSA deemed  some of the information in the restricted report to be sensitive security  information, which must be protected from public disclosure. Although  this report omits that information, such as specific details associated with  the methods and results of testing during the research and development of  the ETPs, it addresses the same questions as the restricted report. Also,  the overall methodology used for both reports is the same.
To determine the extent to which TSA developed a risk-informed strategy  to prioritize investments in the research and development of new  checkpoint technologies, we analyzed program documents, including  TSA’s August 2008 strategic plan for checkpoint technologies, technology  project plans, and budget documents. We also compared TSA’s strategic  plan and DHS’s responses regarding their efforts to develop a risk- informed strategy for their research and development investments with  DHS’s guidance on using risk management principles to prioritize  investments and allocate resources.
To determine what new passenger checkpoint screening technologies DHS  has researched, developed, tested and evaluated, procured, and deployed,  and the reasons why TSA halted the first technology for which it initiated  deployment—the ETP, we analyzed TSA’s strategic plan, TSA’s PSP  documentation, technical and operational requirements for new  technologies, laboratory test reports, and testing data from operational  pilots. Additionally, we interviewed TSA and S&T officials to obtain  information on current technologies being researched, developed, and  deployed, and conducted site visits to the Transportation Security  Laboratory (TSL) and Tyndall Air Force Base to observe testing of new  checkpoint technologies. We visited the TSL because that is where S&T  tests and evaluates transportation technologies, including checkpoint  screening technologies. We visited Tyndall Air Force Base because  technologies to detect bottled liquids explosives were being tested there.  We also interviewed TSA headquarters officials and senior TSA officials  from the airports where TSA had initially deployed or planned to deploy  the ETPs, including 29 Federal Security Directors, 1 Deputy Federal  Security Director, and 5 Assistant Federal Security Directors for  Screening. We chose these officials because they are the senior TSA  officials in charge of security and managing TSA’s role in deploying new  technologies at the airport. We also visited nine airports and selected these  locations based on the technologies that had been deployed or were being  tested on site, their geography, size, and proximity to research and  development laboratories. Of the nine airports we visited, the ETPs had  been deployed or were to be deployed to all of them and other new  checkpoint screening technologies were undergoing pilot demonstrations  or testing at two of them. We visited four airports on the east coast, three  airports on the west coast, and two airports located in the west and  southwestern regions of the United States. We selected these locations  because they represented small-, medium-, and large-sized airports and  different regions in the United States.
To determine the extent to which TSA coordinated and collaborated  internally and with key external stakeholders—airport operators and  technology vendors—on the RDT&E, procurement, and deployment of  checkpoint technologies, we analyzed program documents, including a  memorandum of understanding (MOU) between S&T and TSA.  Additionally, we interviewed S&T and TSA officials, seven checkpoint  technology vendors, and airport operators and other officials at 40  airports where ETPs had initially been or were to be deployed. Because  we selected a nonprobability sample of airports to visit and officials to  interview there, we cannot generalize the results of what we learned to  airports nationwide. However, the information we gathered from these  locations—insights based on observing airport operations and on  perspectives of officials who were involved with DHS’s efforts to  operationally test, evaluate, and deploy checkpoint technologies—could  only be obtained through direct observation or from officials stationed at  these select sites where technologies were being deployed and tested. We  also selected a nonprobability sample of 8 out of the 157 total  requirements for the ETP to determine whether some of its key  requirements had been tested prior to procuring and deploying the  machines. In addition, we reviewed S&T’s and TSA’s coordination and  collaboration activities and compared them to TSA program guidance and  leading practices for collaborating agencies regarding communication,  planning, and federal coordination internally and with external  stakeholders. Appendix I contains additional information on the  objectives, scope, and methodology of our review.
We conducted this performance audit from June 2006 through April 2009,  with some updated information as of September 2009 as previously  disclosed, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: TSA’s airport passenger checkpoint screening system is comprised of  three elements: the (1) personnel, or screeners, responsible for operating  the checkpoint, including the screening of airline passengers and their  carry-on items; (2) standard operating procedures that screeners are to  follow to conduct screening; and (3) technology used during the screening  process. Collectively, these elements determine the effectiveness and  efficiency of passenger checkpoint screening. In strengthening one or  more elements of its checkpoint screening system, TSA aims to balance its  security goals with the need to efficiently process passengers. We  previously reported that TSA had made progress in enhancing its  passenger checkpoint screening system by strengthening screener  training, measuring the performance of screeners and the screening  system, and modifying screening procedures to address terrorist threats  and efficiency concerns. We made recommendations to DHS designed to  strengthen TSA’s efforts to train screeners, modify screening standard  operating procedures, and measure the performance of the checkpoint  screening system. DHS generally agreed with our recommendations and  TSA has taken steps to implement them.
subsections: section_title: Passenger Checkpoint Screening Process
paragraphs: Passenger screening is a process by which screeners inspect individuals  and their property to deter and prevent an act of violence or air piracy,  such as the carriage of any unauthorized explosive, incendiary, weapon, or  other prohibited item onboard an aircraft or into a sterile area. Screeners  inspect individuals for prohibited items at designated screening locations.  TSA developed standard operating procedures and the process for  screening passengers at airport checkpoints. Figure 1 illustrates the  screening functions at a typical passenger checkpoint.
Primary screening is conducted on all airline passengers prior to entering  the sterile area of an airport and involves passengers walking through a  metal detector and carry-on items being subjected to X-ray screening.  Passengers who alarm the walk-through metal detector or are designated  as selectees—that is, passengers selected for additional screening—must   then undergo secondary screening, as well as passengers whose carry-on  items have been identified by the X-ray machine as potentially containing a  prohibited item. Secondary screening involves additional means for  screening passengers, such as by hand-wand, physical pat-down or, at  certain airport locations, an ETP, which is used to detect traces of  explosives on passengers by using puffs of air to dislodge particles from  their body and clothing into an analyzer. Selectees’ carry-on items are also  physically searched or screened for explosives traces by Explosives Trace  Detection (ETD) machines. In addition, DHS S&T and TSA have deployed  and are pursuing additional technologies to provide improved imaging or  anomaly detection capacities to better identify explosives and other threat  objects.
section_title: Roles and Responsibilities for the RDT&E, Procurement, and Deployment of Checkpoint Screening Technologies
paragraphs: DHS and TSA share responsibility for the screening of passengers and the  research, development, and deployment of passenger checkpoint  screening technologies. Enacted in November 2001, the Aviation and  Transportation Security Act (ATSA) created TSA and charged it with the  responsibility of securing civil aviation, which includes the screening of all  passengers and their baggage. ATSA also authorized funding to  accelerate the RDT&E of new checkpoint screening technologies. The  Homeland Security Act of 2002, enacted in November 2002, established  DHS, transferred TSA from the Department of Transportation to DHS and,  within DHS, established S&T to have primary responsibility for DHS’s  RDT&E activities, and for coordinating and integrating all these activities.  The Intelligence Reform and Terrorism Prevention Act of 2004  (Intelligence Reform Act), enacted in December 2004, directed the  Secretary of Homeland Security to give high priority to developing, testing,  improving, and deploying checkpoint screening equipment that detects  nonmetallic, chemical, biological, and radiological weapons and  explosives, in all forms, on individuals and in their personal property.
Until fiscal year 2006, TSA had primary responsibility for investing in the  research and development of new checkpoint screening technologies, and  was responsible for developmental and operational test and evaluation of  new technologies. However, during fiscal year 2006, research and  development functions within DHS were consolidated, for the most part,  within S&T. After this consolidation, S&T assumed primary responsibility  for funding the research, development, and developmental test and  evaluation of airport checkpoint screening technologies. S&T also  assumed responsibility from TSA for the Transportation Security  Laboratory (TSL) which, among other things, tests and evaluates  technologies under development. TSA, through the PSP that was  transferred from the Federal Aviation Administration (FAA) to TSA,  continues to be responsible for identifying the requirements for new  checkpoint technologies; operationally testing and evaluating technologies  in airports; and procuring, deploying, and maintaining technologies. This  transfer of responsibility from TSA to S&T did not limit TSA’s authority to  acquire commercially available technologies for use at the checkpoint.
section_title: DHS and TSA’s Processes for the RDT&E, Procurement, and Deployment of Checkpoint Screening Technologies
paragraphs: S&T and TSA’s RDT&E, procurement, and deployment efforts are made up  of seven components: basic research, applied research, advanced  development, operational testing, procurement, operational integration,  and deployment. S&T is responsible for conducting basic and applied  research, and advanced development, including developmental test and  evaluation. TSA is responsible for conducting operational test and  evaluation, operational integration, procurement and deployment of new  technologies, including checkpoint screening technologies. These seven  components are described below.
Basic research includes scientific efforts and experimentation directed  toward increasing knowledge and understanding in the fields of physical,  engineering, environmental, social, and life sciences related to long-term  national needs.
Applied research includes efforts directed toward solving specific  problems with a view toward developing and evaluating the feasibility of  proposed solutions.
Advanced development includes efforts directed toward projects that have  moved into the development of hardware and software for field  experiments and tests, such as acceptance testing.
Operational test and evaluation verifies that new systems are  operationally effective, supportable, and suitable before deployment.
Operational integration is the process employed to enable successful  transition of viable technologies and systems to the field environment.
Procurement includes the efforts to obtain a product or service.     Deployment is a series of actions following the determination that the  product meets its requirements and is accepted by the program manager  and integrated product team; designated locations are configured for  product integration into the screening operating system and the installed  product passes site acceptance tests; and logistics support is in place and  all users are trained to use the product.
section_title: RDT&E, Procurement, and Deployment Funding for Checkpoint Screening Technologies
paragraphs: Over $795 million has been invested by DHS and TSA during fiscal years  2002 through 2008 for the RDT&E, procurement, and deployment of  checkpoint screening technologies. During this time, over $91 million was  invested in the RDT&E of checkpoint technologies and about $704 million  was invested in the procurement and deployment of these technologies.  From fiscal years 2002 through 2005, TSA was responsible for the RDT&E  of checkpoint technologies; however, TSA officials could not identify the  amount of funding the agency invested for these purposes during those  years. After fiscal year 2005, TSA invested $14.5 million for test and  evaluation of checkpoint technologies, but did not fund the research and  development of these technologies because responsibility in general for  research and development funding was transferred from TSA to S&T  beginning in fiscal year 2006. Therefore, during fiscal years 2006 through  2008, S&T invested $77.0 million in the RDT&E of checkpoint screening  technologies. All of the approximately $704 million for the procurement  and deployment of checkpoint screening technologies from fiscal years  2002 through 2008 was invested by TSA because the agency has been  responsible for procurement and deployment of these technologies since it  was created.
section_title: Applying a Risk Management Approach to Checkpoint Technology Investments
paragraphs: Risk management is a tool that policy makers can use to help ensure that  strategies to develop protective programs and allocate resources target the  highest priority security needs. This information helps officials determine  which security programs are most important to develop and fund, given  that it is not possible to protect the country against all threats because of  limited resources. Law and related policy, including the Intelligence  Reform Act, the Implementing Recommendations of the 9/11 Commission  Act of 2007 (9/11 Commission Act), and Homeland Security Presidential  Directive 7, provide that federal agencies with homeland security  responsibilities are to apply risk-informed principles to prioritize security  needs and allocate resources. Consistent with these provisions, DHS  issued the National Strategy for Transportation Security in 2005 that,  among other things, describes the policies that DHS is to apply when  managing risks to the security of the U.S. transportation system. Further,  in June 2006, DHS issued the NIPP, which provides a risk management  framework to guide strategies to develop homeland security programs and  allocate resources to them. According to the NIPP, its risk management  framework consists of six phases that help to identify and assess risks and  prioritize investments in programs, as illustrated in figure 2. The NIPP  designated TSA as the primary federal agency responsible for coordinating  critical infrastructure protection efforts within the transportation sector.
A risk-informed strategy to develop and invest in critical infrastructure  protection, according to the NIPP, begins with setting security goals.  Setting security goals involves defining specific outcomes, conditions, end  points, or performance targets that collectively constitute an effective  protective posture. Once security goals are established, decisionmakers  are to identify what assets or systems to protect and identify and assess  the greatest risks to them, that is, the type of terrorist attack that is most  likely to occur and that would result in the most severe consequences.  Risk of a terrorist attack, according to the NIPP, is to be assessed by  analyzing consequences of an attack; the threat—that is, the likelihood of  an attack; and the extent to which an asset or a system, in this case the  transportation system, is vulnerable to this type of attack. The potential  consequences of any incident, including terrorist attacks and natural or  manmade disasters, is the first factor to be considered in a risk  assessment. In the context of the NIPP, consequence is measured as the  range of loss or damage that can be expected in the event a terrorist attack  succeeds. A consequence assessment looks at the expected worst case or  reasonable worst case impact of a successful attack. A threat assessment  is the identification and evaluation of adverse events that can harm or  damage an asset and takes into account certain factors, such as whether  the intent and capability to carry out the attack exist. A vulnerability  assessment identifies weaknesses or characteristics of an asset or system,  such as its design and location, which make it susceptible to a terrorist  attack and that may be exploited. This analysis should also take into  consideration factors such as protective measures that are in place which  may reduce the risk of an attack and the system’s resiliency, that is, ability  to recover from an attack.
Once the three components of risk—threat, vulnerability, and  consequence—have been assessed for a given asset or system, they are  used to provide an estimate of the expected loss considering the likelihood  of an attack or other incident. According to the NIPP, calculating a  numerical risk score using comparable, credible methodologies provides a  systematic and comparable estimate of risk that can help inform national  and sector-level risk management decisions. To be considered credible,  the NIPP states that a methodology must have a sound basis; be complete;  be based on assumptions and produce results that are defensible; and  specifically address the three variables of the risk calculus: threat,  vulnerability, and consequence. The methodology should also be  comparable with other methodologies to support a comparative sector or  national risk assessment. To be comparable, a methodology must be  documented, transparent, reproducible, accurate, and provide clear and  sufficient documentation of the analysis process and the products that  result from its use.
The next steps in the DHS risk management framework involve  establishing priorities for program development based on risk  assessments; implementing these protective programs; and measuring  their effectiveness by developing and using performance measures.  Identifying and assessing risks helps decisionmakers to identify those  assets or systems that are exposed to the greatest risk of attack and, based  on this information, prioritize the development and funding of protective  programs that provide the greatest mitigation of risk given available  resources. The NIPP notes that because resources are limited, risk  analysis must be completed before sound priorities can be established. To  determine which protective measures provide the greatest mitigation of  risk for the resources that are available, the NIPP directs policy makers to  evaluate how different options reduce or mitigate threat, vulnerability, or  consequence of a terrorist attack. To do so, the NIPP states that cost  estimates are combined with risk-mitigation estimates in a cost–benefit  analysis to choose between the different options. The last step in the NIPP,  measuring the effectiveness of security programs by developing and using  performance measures, provides feedback to DHS on its efforts to attain  its security goals. Performance metrics are to be developed and used to  affirm that specific goals and objectives are being met or to articulate gaps  in the national effort or supporting sector efforts. Performance measures  enable the identification of corrective actions and provide decisionmakers  with a feedback mechanism to help them make appropriate adjustments in  their strategies for protecting critical infrastructure.
section_title: TSA Has Taken Actions to Prioritize Investments in Passenger Checkpoint Screening Technologies, but Lacks a Risk-Based Strategy
paragraphs: While TSA completed a strategic plan for the PSP in August 2008 that  identifies a strategy for researching, developing, and deploying checkpoint  screening technologies, the plan and the strategy were not developed  based upon all of the key risk management principles outlined in DHS’s  NIPP. For instance, TSA has not conducted a complete risk assessment for  the PSP, conducted a cost–benefit analysis to prioritize investments, or  developed performance measures to assess the extent to which the risk of  attack has been reduced or mitigated by investments in technologies.  While the agency is currently reviewing a draft of the Aviation Domain  Risk Assessment (ADRA), as of September 2009, the ADRA had not been  finalized. Officials expect it to be finalized by the end of calendar year  2009. TSA officials could not provide an expected completion date.  Therefore, we could not determine when TSA will complete it or to what  extent it will be consistent with DHS’s risk management framework. TSA  officials acknowledged the importance of a cost–benefit analysis and  performance measures to guide technology investments, and stated that  they intend to develop them, but could not identify when they would be  completed. Until TSA completes these activities, the agency lacks  assurances that the PSP strategy addresses the highest priority needs and  mitigates the risk of an attack. Further, TSA lacks information to adjust its  strategy, if needed.
subsections: section_title: TSA Completed a Strategic Plan for the PSP that Identifies Goals and Objectives
paragraphs: TSA completed a strategic plan in August 2008 that identifies a strategy  and establishes goals and objectives for the PSP, and submitted the plan to  congressional committees in September 2008. However, TSA officials  stated that the NIPP was not used as guidance in developing the plan.  Instead, the officials stated that the specific requirements for a strategic  plan, as outlined in the Intelligence Reform Act and 9/11 Commission Act,  were used as guidance to construct the plan. The strategic plan identifies  three broad trends that have been observed in the types of threats that  TSA faces. First, interest in catastrophic destruction of aircraft and  facilities has increased, in contrast to hijacking and hostage-taking that  characterized the majority of earlier attacks. Second, the range of  encountered weapons has expanded, many not previously recognized as  threats, nor detected by the technologies that were deployed. Third,  terrorists have attacked “soft” airport targets, including airport lobbies, in  other countries. To address these challenges, TSA’s strategic plan  identifies that the agency’s strategy is to utilize intelligence; partner with  law enforcement, industry partners, and the public; and implement  security measures that are flexible, widely deployable, mobile, and layered  to address the nation’s complex open transportation network. According  to the plan, TSA is in the process of implementing and evaluating a  fundamental shift in strategy for the security checkpoint that encompasses  the critical elements of people, process, and technology. In addition, the  plan states that implementing a new security approach called Checkpoint  Evolution, which started in the spring 2008, will bring the most significant  changes that have occurred in passenger screening since the airport  security checkpoint was first established in the 1970s.
TSA’s strategic plan identifies that the key component of TSA’s strategy  related to security checkpoints is to improve security effectiveness and  resource utilization at the checkpoints. Also, the PSP manager stated that  a goal of the PSP strategy is to achieve full operating capability by the  dates discussed for each checkpoint screening technology listed in the  strategic plan. To meet these goals, the PSP strategic plan identifies three  strategic objectives: (1) improve explosive detection capability,   (2) improve the behavior detection capability of Transportation Security  Officers (TSO), and (3) extend the layers of security throughout the  passenger journey. The first objective, improving explosive detection  capability, involves combining new technology with procedures that  emphasize an element of unpredictability to improve explosive detection  capability and prevent would-be attackers from knowing the TSA security  process. The second objective, improving the behavior detection  capability of TSOs, involves shaping the checkpoint environment to better  support and enhance behavior detection capabilities by enabling TSOs to  engage a larger number of passengers more frequently throughout the  checkpoint queue using informal interviews and SPOT; improving the  observation conditions for TSOs trained in SPOT by enhancing the  contrast between passengers exhibiting signs of travel stress and those  intending to do harm to other passengers, aircraft, or the airport; and  providing communications tools for enhanced coordination between TSOs  trained in SPOT. The third objective, extending the layers of security  throughout the passenger journey, involves enabling additional layers of  non-intrusive security beyond the checkpoint and into public spaces;  increasing the interaction between TSOs and passengers to provide more  opportunities to identify irregular behaviors far ahead of the potential  threat reaching the checkpoint; and partnering with airlines, airports, and  the private sector to reduce vulnerabilities in soft target areas.
TSA had been directed on multiple occasions to provide strategic plans for  explosives detection checkpoint technologies to congressional  committees. The Intelligence Reform Act mandated that TSA provide a  strategic plan that included, at a minimum, a description of the current  efforts to detect explosives on individuals and in their personal property;  operational applications of explosive detection equipment at airport  checkpoints; quantities of equipment needed to implement the plan and a  deployment schedule; funding needed to implement the plan; measures  taken and anticipated to be taken to provide explosives detection  screening for all passengers identified for additional screening; and  recommended legislative actions, if any. The Intelligence Reform Act  mandated that such a strategic plan be submitted to congressional  committees during the second quarter of fiscal year 2005. According to  TSA officials, a strategic plan was developed and delivered to  congressional committees on August 9, 2005, in satisfaction of the  statutory mandate. However, the 9/11 Commission Act, enacted August 3,  2007, reiterated a requirement for a strategic plan that TSA was mandated  to submit in accordance with the Intelligence Reform Act. Specifically, the  9/11 Commission Act required that the Secretary of Homeland Security  issue a strategic plan addressing its checkpoint technology program not  later than 30 days after enactment of the 9/11 Commission Act (that is, by  September 3, 2007) and required implementation of the plan to begin  within 1 year of the act’s enactment. In response to the 9/11 Commission  Act, TSA provided to Congress the Aviation Security Report: Development  of a Passenger Checkpoint Strategic Plan, September 2007. Finally,  Division E of the Consolidated Appropriations Act, 2008, enacted on  December 26, 2007, required that the Secretary of Homeland Security  submit a strategic plan for checkpoint technologies no later than 60 days  after enactment of the Act (that is, by February 25, 2008), and further  restricted the use of $10,000,000 appropriated to TSA for Transportation  Security Support until the Secretary submitted the plan to the Committees  on Appropriations of the Senate and House of Representatives. As a  result of the mandate for a strategic plan and the funding restriction in the  2008 Consolidated Appropriations Act, TSA officials told us that they  interpreted this legislative language to mean that congressional  committees considered TSA’s aviation security report in September 2007  to be incomplete and insufficient. After approximately 12 months had  elapsed since a strategic plan had been mandated in the 9/11 Commission  Act, in August 2008 TSA completed its revised strategic plan and delivered  it to the committees in September 2008, which TSA officials stated meets  the mandate for a strategic plan in the 9/11 Commission Act, as well as the  mandate for a strategic plan in the appropriations act.
As previously discussed, the Intelligence Reform Act included  requirements for a deployment schedule, and descriptions of the quantities  of equipment and funding needed to implement the plan. However, our  analysis of TSA’s August 2008 strategic plan indicates that the strategic  plan could include more complete information about these requirements.  For example, although TSA provided some deployment information for  each emerging checkpoint technology listed in the strategic plan—such as  the total quantity to be deployed, expected full operating capability date,  and types or categories of airports where the equipment is to be  deployed—it does not include a year-by-year schedule showing the  number of units for each emerging technology that is expected to be  deployed to each specific airport. Regarding information on the funding  needed to implement the strategic plan, it includes a funding profile for  each fiscal year from 2007 through 2009. However, a number of the  emerging technologies are not expected to reach full operating capability  until fiscal year 2014. TSA officials stated that they have derived notional  (that is, unofficial) quantities to be deployed on an annual basis for each  technology through its respective full operating capability date, but the  officials stated that the funding profile in the strategic plan does not reflect  the funding needed for these future quantities because the funding that  will be appropriated for them after fiscal year 2009 is unknown. According  to the officials, to implement the strategic plan in the years beyond fiscal  year 2009, the agency intends to use a year-by-year approach whereby the  quantities to be deployed in a particular year, and the funding needed for  that year, would not be officially identified prior to the budget request for  that year.
TSA officials stated that they used risk to inform the August 2008 strategic  plan and the PSP strategy identified in it. Although TSA may have  considered that risk to some degree, our analysis does not confirm that  these efforts meet the risk-based framework outlined in the NIPP.  Specifically, TSA has not conducted a risk assessment or cost–benefit  analyses, or established quantifiable performance measures. As a result,  TSA does not have assurance that its efforts are focused on the highest  priority security needs, as discussed below.
section_title: TSA Has Not Conducted a Risk Assessment to Inform Its PSP Strategy, but Is Finalizing an Assessment and Developing Information that May Help Guide PSP Efforts
paragraphs: TSA has not conducted a risk assessment that includes an assessment of  threat, vulnerability, and consequence, which would address passenger  checkpoint screening; consequently, the PSP strategy has not been  informed by such a risk assessment as required by the NIPP. Agency  officials stated that they prepared and are currently reviewing a draft of a  risk assessment of the aviation domain, known as the ADRA, which is  expected to address checkpoint security and officials expect it to be  finalized by the end of calendar year 2009; however, its completion has  been delayed multiple times since February 2008. Therefore, it is not clear  when this assessment will be completed. The ADRA, when completed, is  to provide a scenario-based risk assessment for the aviation system that  may augment the information TSA uses to prioritize investments in  security measures, including the PSP. However, officials could not provide  details regarding the extent to which the ADRA would assess threat,  vulnerability, and consequence related to the passenger checkpoint. In  2004, we recommended that the Secretary of Homeland Security and the  Assistant Secretary for TSA complete risk assessments—including a  consideration of threat, vulnerability, and consequence—for all modes of  transportation, and use the results of these assessments to help select and  prioritize research and development projects. TSA and DHS concurred  with the recommendation, but have not completed these risk assessments.  Because TSA has not issued the ADRA or provided details regarding what  it will entail, and because it is uncertain when the ADRA will be  completed, it is not clear whether the ADRA will provide the risk  information needed to support the PSP and TSA’s checkpoint technology  strategy. In the meantime, TSA has continued to invest in checkpoint  technologies without the benefit of the risk assessment information  outlined in the NIPP. Consequently, TSA increases the possibility that its  investments will not address the highest priority security needs.
Although TSA has not completed a risk assessment to guide its PSP,  officials stated that they identify and assess risks associated with the  passenger screening checkpoint by relying on threat information,  vulnerability information from Threat Image Projection (TIP) scores,  limitations of screening equipment identified during laboratory testing,  covert tests, and expert judgment to guide its investment strategy in the  PSP. Specifically, TSA’s Office of Intelligence produces civil aviation  threat assessments on an annual basis, among other intelligence products.  These assessments provide information on individuals who could carry  out attacks, tactics they might use, and potential targets. TSA’s most  recent aviation threat assessment, dated December 2008, identifies that  terrorists worldwide continue to view civil aviation as a viable target for  attack and as a weapon that can be used to inflict mass casualties and  economic damage. It also concluded that improvised explosive devices  (IED) and hijackings pose the most dangerous terrorist threat to  commercial airliners in the United States. The assessment identifies that  these devices may be concealed on persons, disguised as liquids, or hidden  within everyday, familiar objects such as footwear, clothing, toys, and  electronics. The threat assessment further identifies that terrorists have  various techniques for concealing explosives on their persons. In addition  to the annual civil aviation threat assessment, the Office of Intelligence  prepares for TSA’s senior leadership team and other officials a (1) daily  intelligence briefing, (2) tactical intelligence report that is produced one to  four times per week, (3) weekly field intelligence summary, (4) weekly  suspicious incident report, and, when necessary, (5) special events update,  for example, during major political events. However, according to the  NIPP, relying on threat information is not sufficient to identify and assess  risks. Rather, threat information, which indicates whether a terrorist is  capable of carrying out a particular attack and intends to do so, is to be  analyzed along side information on vulnerabilities—weakness in a system  that would allow such an attack to occur—and on the consequences of the  attack, that is, the results of a specific type of terrorist attack, according to  the NIPP.
TSA officials stated that, to guide the PSP, they also rely on programs in  place that are designed to assess vulnerabilities at airport checkpoints. To  identify vulnerabilities at airport checkpoints, TSA officials stated that  TSA analyzes TIP scores, known limitations of screening equipment based  on laboratory testing, and information from its covert testing program.  TSA conducts national and local covert tests, whereby individuals attempt  to enter the secure area of an airport through the passenger checkpoint  with a prohibited item in their carry-on bags or hidden on their person.  Officials stated they use these sources of information to identify needed  changes to standard screening procedures, new technology requirements,  and deployment strategies for the PSP. When a checkpoint vulnerability is  identified, officials stated that TSA’s Office of Security Technology  engages other TSA stakeholders through the PSP’s Integrated Project  Team process to identify and develop necessary technology requirements  which may lead to new technology initiatives. Officials credited this  process with helping TSA identify needed changes to standard screening  procedures and deployment strategies for new technologies. For example,  according to a TSA official, a technology was developed as a result of tests  conducted by GAO that found that prohibited items and components of an  IED might be more readily identified if TSA were to develop new  screening technologies to screen these items.
Although TSA has obtained information on vulnerabilities at the screening  checkpoint, the agency has not assessed vulnerabilities (that is,  weaknesses in the system that terrorists could exploit in order to carry out  an attack) related to passenger screening technologies that are currently  deployed. The NIPP requires a risk assessment to include assessments of  threat, vulnerability, and consequence. TSA has not assessed whether  there are tactics that terrorists could use, such as the placement of  explosives or weapons on specific places on their bodies, to increase the  likelihood that the screening equipment would fail to detect the hidden  weapons or explosives. Although TIP scores measure how effectively  screeners identify prohibited items, they do not indicate whether  screening technologies currently deployed may be vulnerable to tactics  used by terrorists to disguise prohibited items, such as explosives or  weapons, thereby defeating the screening technologies and evading  detection. Similarly, TSA’s covert testing programs do not systematically  test passenger and baggage screening technologies nationwide to ensure  that they identify the threat objects and materials the technologies are  designed to detect, nor do the covert testing programs identify  vulnerabilities related to these technologies. We reported in August 2008  that, while TSA’s local covert testing program attempts to identify test  failures that may be caused by screening equipment not working properly  or caused by screeners and the screening procedures they follow, the  agency’s national testing program does not attribute a specific cause of the  test failure. We recommended, among other things, that TSA require the  documentation of specific causes of all national covert testing failures,  including documenting failures related to equipment, in the covert testing  database to help TSA better identify areas for improvement. TSA  concurred with this recommendation and stated that the agency will  expand the covert testing database to document test failures related to  screening equipment. Moreover, TSA officials stated that it is difficult to  attribute a test failure to equipment, because there is a possibility that the  threat item used for the test was not designed properly and, therefore,  should not have set off the equipment’s alarm. TSA officials also stated  that it is difficult to identify a single cause for a test failure because covert  testing failures can be caused by multiple factors. As a result, TSA lacks a  method to systematically test and identify vulnerabilities in its passenger  and baggage screening equipment in an operational airport setting.  Consequently, TSA officials do not have complete information to identify  the extent to which existing screening technologies mitigate  vulnerabilities at the passenger checkpoints, so that they can incorporate  this information into the agency’s security strategy, as required by DHS  guidance.
TSA’s ADRA, once completed, is to cover the entire aviation domain and  include three parts—assessments of over 130 terrorist attack scenarios to  determine whether they pose a threat to the aviation system; an  assessment of known vulnerabilities or pathways within the aviation  system through which these terrorist attacks could be carried out; and an  assessment of consequences of these various types of terrorist attacks,  such as death, injury, and property loss. TSA officials stated that, through  the use of expert panels, the ADRA will evaluate these threat scenarios to  assess the likelihood that terrorists might successfully carry out each type  of attack on the aviation system, and the likelihood and consequences of  these various scenarios will be prioritized to identify the most pressing  risks that need to be addressed. In the case of the passenger screening  checkpoint, according to officials, TSA will be examining all security  measures that a terrorist must breach in order to carry out a specific type  of an attack, such as carrying an IED on board an aircraft and detonating it  midflight. However, officials could not explain or provide documentation  identifying the extent to which the ADRA will provide threat, vulnerability,  and consequence assessments in support of the PSP. In addition, the  completion date for the ADRA has been delayed multiple times. Because  the ADRA has not been finalized and TSA has not described how the  ADRA will address the passenger checkpoint, we could not determine the  extent to which it will incorporate information on checkpoint  vulnerabilities, including vulnerabilities associated with screening  technologies and standard operating procedures.
In addition to the ADRA, TSA and DHS S&T are developing other  information that could inform their identification and assessments of risks  to the aviation transportation system. Specifically, TSA and S&T are  reviewing the scientific basis of their current detection standards for  explosives detection technologies to screen passengers, carry-on items  and checked baggage. As part of this work, TSA and S&T are conducting  studies to update their understanding of the effects that explosives may  have on aircraft, such as the consequences of detonating explosives on  board an in-flight aircraft. Senior TSA and DHS S&T officials stated that  the two agencies decided to initiate this review because they could not  fully identify or validate the scientific support requiring explosives  detection technologies to identify increasingly smaller amounts of some  explosives over time as required by TSA policy. Officials stated that they  used the best available information to originally develop detection  standards for explosives detection technologies. However, according to  these officials, TSA’s understanding of how explosives affect aircraft has  largely been based on data obtained from live-fire explosive tests on  aircraft hulls at ground level. Officials further stated that due to the  expense and complexity of live-fire tests, FAA, TSA, and DHS collectively  have conducted only a limited number of tests on retired aircraft, which  limited the amount of data available for analysis. As part of this ongoing  review, TSA and S&T are simulating the complex dynamics of explosive  blast effects on an in-flight aircraft by using a computer model based on  advanced software developed by the national laboratories. TSA believes  that the computer model will be able to accurately simulate hundreds of  explosives tests by simulating the effects that explosives will have when  placed in different locations within various aircraft models. Officials  estimated this work will be completed in 3- to 4-month increments through  2008 and 2009. Officials further stated that the prototype version of the  model was validated in the late summer of 2008, and that the model is  currently being used. TSA and S&T officials stated that they expect the  results of this work will provide a much fuller understanding of the  explosive detection requirements and the threat posed by various amounts  of different explosives, and will use this information to determine whether  any modifications to existing detection standards should be made moving  forward.
section_title: TSA Has Not Completed a Cost–Benefit Analysis to Help Establish Risk-Based Priorities and Guide Its Investment Strategy
paragraphs: TSA has not completed a cost–benefit analysis to prioritize and fund the  PSP’s priorities for investing in checkpoint technologies, as required by  the NIPP’s risk management framework. According to the NIPP, policy  makers who are designing programs and formulating budgets are to  evaluate how different options reduce or mitigate threat, vulnerability, or  consequence of a terrorist attack through a cost–benefit analysis that  combines cost estimates with risk-mitigation estimates. However, in  addition to lacking information on risks to the screening checkpoint, TSA  has not conducted a cost–benefit analysis of checkpoint technologies  being researched and developed, procured, and deployed. Such a cost– benefit analysis is important because it would help decisionmakers  determine which protective measures, for instance, investments in  technologies or in other security programs, will provide the greatest  mitigation of risk for the resources that are available.
One reason that TSA may have difficulty developing a cost–benefit  analysis for the PSP is that it has not developed life-cycle cost estimates of  each screening technology the PSP is developing, procuring, or deploying.  This information is important because it helps decisionmakers determine,  given the cost of various technologies, which technology provides the  greatest mitigation of risk for the resources that are available. TSA  officials prepared a PSP lifecycle cost estimate in September 2005, but this  estimate does not include cost estimates for all technologies currently  being researched, developed, tested and evaluated, procured and/or  deployed, such as the Advanced Technology Systems, a technology to  screen carry-on items that TSA is currently procuring. TSA was  subsequently instructed by DHS Joint Requirements Council to complete  lifecycle cost estimates for the PSP; in December 2005, the council  reviewed the PSP and approved it to proceed to the Investment Review  Board for an annual review and potential approval of the PSP’s fiscal year  2006 procurement strategy. However, the council expressed concern  about several issues that should be resolved prior to the Investment  Review Board’s review, including the need for complete lifecycle cost  estimates for the checkpoint screening technologies that were to be  developed and procured. TSA officials acknowledged that completing  lifecycle cost estimates are important and stated that they have not  prepared a lifecycle cost estimate since the council recommended that  such an estimate be developed due to lack of staff. These officials further  stated that TSA hired four full-time equivalent staff in fiscal year 2008, and  two additional full-time equivalent staff are expected to be hired in the fall  of 2008. The officials anticipate that these staff will help prepare lifecycle  cost estimates. However, the officials did not provide a timeframe for the  completion of the estimates.
Although TSA officials identified the technologies they are procuring and  deploying, TSA officials could not provide us with information on their  priorities for the research and development of checkpoint screening  technologies or the processes they followed to develop these priorities.  According to S&T officials, TSA provided priorities for near-term applied  research and development projects to the S&T Capstone Integrated  Product Team (IPT) for Explosives Prevention. This IPT establishes  priorities for research projects to be funded by S&T during the fiscal year.  S&T officials stated that they rely on TSA and other members of the IPT to  use a risk-based approach to identify and prioritize their agencies’ or  offices’ individual research and development needs prior to submitting  them for consideration to the IPT. However, TSA officials stated they did  not submit priorities for research and development to S&T. Without cost– benefit or other analysis to compare the cost and effectiveness of various  solutions, the agency cannot determine whether investments in the  research and development of new checkpoint technologies or procedures  most appropriately mitigate risks with the most cost-effective use of  resources. In addition, without knowing the full cost of the technologies  that the PSP is developing, procuring, or deploying, TSA could potentially  invest in a technology in which the cost outweighs expected benefits.
section_title: TSA Lacks Measures to Evaluate the Extent to Which the PSP Reduces the Risk of Terrorist Attacks
paragraphs: TSA’s strategy for the PSP does not have a mechanism—such as  performance measures or other evaluation methods—to monitor, assess,  or test the extent to which investments in new checkpoint technologies  reduce or mitigate the risk of terrorist attacks. The NIPP requires that  protective programs be designed to allow measurement, evaluation, and  feedback based on risk mitigation so that agencies may re-evaluate risk  after programs have been implemented and take corrective action if  needed, such as modifying existing programs to counter new risks or  implementing alternative programs. The NIPP identifies three types of  performance measures—descriptive, process/output, and outcome  measures—that can help gauge the effectiveness of protective programs.  Although the NIPP requires that protective programs be designed to allow  measurement, evaluation, and feedback based on risk mitigation, TSA has  not identified quantifiable measures of progress which would allow the  agency to assess the PSP’s overall effectiveness. TSA officials stated that  they do not have overall performance measures but are currently  developing performance goals and measures for the overall program.  However, the officials could not provide a time frame for their completion.  In September 2004, we recommended that TSA complete strategic plans  for its research and development programs which contain measurable  objectives. Without measures to monitor the degree to which the TSA’s  investments in the research, development, and deployment of new  screening technologies reduce or mitigate terrorist threats, the agency is  limited in its ability to assess the effectiveness of the PSP or the extent to  which it complements other layers of security at the checkpoint.
section_title: Ten New Checkpoint Screening Technologies Are in Various Phases of RDT&E, Procurement, and Deployment, but ETP Deployment Has Been Halted
paragraphs: Since TSA’s creation in 2001, 10 new checkpoint screening technologies,  including the ETP, have been in various phases of RDT&E, procurement,  and deployment, but TSA halted deployment of the ETP due to  performance problems and high installation costs. Of the 10 technologies,  TSA has initiated deployments for 4 of them, including the ETP and a  Bottled Liquids Scanner, but TSA has not deployed any of the 4  technologies to airports nationwide. TSA also initiated procurements of  two technologies, including the Whole Body Imager; however, deployment  of these two technologies has not begun yet. Four checkpoint technologies  are in research and development, such as a shoe scanning device. In June  2006, 6 to 11 months after TSA began to deploy the ETPs to airports, the  agency halted their deployment due to performance problems—the  machines broke down more frequently than specified by the functional  requirements and the machines were more expensive to install and  maintain in airports than expected. Because TSA did not follow its  acquisition guidance that recommends technologies be tested and  evaluated in an operational setting prior to procurement and deployment,  the agency lacked assurance that the ETPs performed as required by the  system’s requirements. Although TSA officials were aware that tests  conducted on earlier ETP models during 2004 and 2005 suggested that  they did not operate reliably in an airport environment and that the ETP  models that were subsequently deployed to airports had not been tested in  an operational environment to prove their effectiveness, TSA deployed the  ETPs to airports beginning in July 2005 for the Smiths Detection ETP and  beginning in January 2006 for the General Electric ETP without resolving  these issues. TSA officials stated that they deployed the ETPs to respond  quickly to the threat posed by a potential suicide bomber after suicide  bombings had been carried out onboard Russian airliners in 2004. TSA  officials stated that they plan to continue to use the 90 ETPs currently  deployed to airports. Because the ETPs were deployed without resolving  their performance problems and validating all of the functional  requirements, the ETPs have not been demonstrated to increase security  at the checkpoint. In the future, using validated technologies would  enhance TSA’s efforts to improve checkpoint security.
subsections: section_title: S&T and TSA Investments in RDT&E Resulted in the Procurement or Deployment of Six New Checkpoint Technologies
paragraphs: As a result of S&T and TSA investments in the RDT&E of checkpoint  screening technologies since TSA’s creation in 2001, six new screening  technologies are being procured and/or deployed, while four checkpoint  screening technologies are currently in the research and development  phase. Based on S&T and TSA RDT&E efforts, the agency has initiated  deployments of four technologies—the ETP, Fido PaxPoint Bottled  Liquids Scanner, Advanced Technology Systems, and Cast and Prosthesis  Scanner—three of which originated as commercial-off-the-shelf  technologies or commercial-off-the-shelf technologies that TSA modified  for use as checkpoint screening devices. However, TSA has not  completed the deployment for all of these four technologies to airports  nationwide. TSA officials stated that they did not deploy additional  checkpoint screening technologies because they were primarily focused  on deploying explosives detection systems to screen checked baggage, as  mandated by ATSA. TSA has also initiated procurements of two additional  technologies—Automated Explosives Detection System for Carry-on  Baggage and Whole Body Imager—but has not deployed either of them  yet. Figure 3 describes the status of the six checkpoint screening  technologies for which TSA has initiated procurement and/or deployment.
According to TSA’s August 2008 strategic plan for checkpoint  technologies, there are several other ongoing efforts in addition to the  technologies discussed in figure 3. S&T and TSA are researching and  developing a shoe scanning device that is to conduct automated weapons  and explosive detection without requiring passengers to remove their  footwear. TSA plans to award a contract in fiscal year 2010, with full  operating capability in fiscal year 2015. TSA plans to deploy 1,300 units at  all category X through category IV airports. TSA also has two ongoing  efforts related to boarding pass and credential authentication, according  to the agency’s strategic plan. Starting in 2007, TSA assumed responsibility  from airline contractors for travel document checking, which is currently  conducted manually. TSA plans to replace the manual system with an  automated one. Specifically, the Boarding Pass Scanning System is  expected to verify the authenticity of a boarding pass at the checkpoint  and enable the use of paperless boarding passes by the airlines. In  addition, the Credential Authentication Technology System is planned to  be an automated system that authenticates identification presented by  passengers and airport employees. According to TSA, the agency plans to  eventually combine both of these authentication systems in a single travel  document checking system. TSA plans to award a contract for these two  systems in fiscal year 2009, with full operating capability expected in fiscal  year 2014. TSA plans to deploy a total of 878 units to replace the existing  document verification tools at all category X through category IV airports.  Another ongoing effort identified in TSA’s strategic plan is the Next  Generation ETD. This system is planned to replace legacy ETD systems  and to be able to identify a larger range of explosives. Specifically, this  system is expected to have enhanced explosive detection capability in  terms of sensitivity and the ability to detect new threats, as well as other  improvements over legacy systems, which are expected to produce lower  lifecycle costs. TSA plans to deploy 1,500 units at all category X through  category IV airports.
TSA also has two additional efforts to assess possible technologies. One  effort is called Standoff Detection, which is intended to display images to  detect anomalies concealed under passengers’ clothing. TSA plans to  conduct an operational utility evaluation of test article units during fiscal  year 2009 to evaluate the technology’s feasibility within checkpoint  screening operations. According to TSA, this technology would assist the  agency in applying layered security prior to the checkpoint in soft target  areas, such as airport lobbies, to improve early awareness of a potential  explosive threat. If the technology proves effective in the checkpoint  operation, TSA plans to award a contract in fiscal year 2010, with full  operational capability expected by fiscal year 2014, and to deploy 351 units  to every checkpoint at category X and category I airports. The other effort  is called Explosives Characterization for Trace (Chemical-based)  Detection. This effort includes the research and development of trace  signatures, detection, and physical properties of explosives to improve the  detection and performance of deployed explosives trace detection  technologies.
section_title: TSA Procured and Deployed ETPs without Assurance that They Would Perform as Intended in an Operational Setting
paragraphs: During 2004 and 2005, prior to deployment of the ETPs, TSA conducted a  series of acceptance tests (that is, laboratory tests) of the General Electric  and Smiths Detection ETPs that suggested they had not demonstrated  reliable performance.  Specifically, in 2004, TSA conducted acceptance  tests on early models of the General Electric and Smiths Detection ETPs  to determine whether the ETPs met key functional requirements.  Subsequently, in 2004 a General Electric ETP model was field tested at five  airports to determine how well the ETP performed in an operational  environment. A Smiths Detection ETP model was also field tested at an  airport in 2004.  Based on initial test results, both vendors of the ETPs  modified the machines, and TSA conducted further laboratory testing. The  modified General Electric ETP was tested from December 2004 through  February 2005.  During the January 2005 to May 2005 time frame, both the  General Electric and Smiths Detection ETP models were tested. Even  though tests conducted during 2004 and 2005 of the General Electric and  Smiths Detection ETPs suggested they had not demonstrated reliable  performance, TSA deployed the Smiths Detection ETP and General  Electric ETP to airports starting in July 2005 and January 2006,  respectively, without resolving identified performance issues.
Further, TSA did not test all 157 of the ETP’s functional requirements prior  to procuring and deploying the General Electric and Smiths Detection ETP  models.  Instead, TSA tested the ETP models against a subset of the  functional requirements. According to TSA’s System Development Life  Cycle Guidance, testing of a system is to be conducted to prove that the  developed system satisfies its requirements in the functional requirements  document. TSA officials could not identify the specific requirements that  were tested or the reason(s) that all of the requirements were not tested.
A TSA official stated that TSA had intended to resolve problems regarding  the ETPs’ performance after they had been deployed, but TSA officials  could not explain how these problems were to be resolved. Officials  further stated that they worked for over 1 year during 2006 and 2007 with  the ETP vendors to correct reliability and maintenance issues after the  ETPs were initially deployed, but could not resolve them. Furthermore,  according to S&T officials, when TSA conducted limited field tests, the  ETP manufacturers provided different configurations from those used  during the laboratory tests. According to officials, once this was  discovered, it took more than 6 months for the ETP manufacturers to  recreate the configurations that had passed the laboratory tests. TSA  officials stated that, during this 6-month period, the agency decided to  award a sole source contract to General Electric to procure its ETP.
Regarding the reliability of the ETPs, of the 101 ETPs (71 from General  Electric and 30 from Smiths Detection) that were originally deployed to 36  airports, the General Electric ETP did not meet the system requirement for  operational availability due to frequent breakdowns. Both vendors’ ETPs  were also more expensive to maintain than expected, according to the  TSA Chief Technology Officer serving during this period. The functional  requirements document requires the ETP to be operationally available  98.38 percent of the time. However, the General Electric ETPs were not  always able to meet this requirement. TSA officials could not provide  information on the operational availability of the Smiths Detection ETPs.  For the General Electric ETPs, from January through May 2006, they were  operationally available an average of 98.05 percent of the time, although  the ETPs met the operational availability requirement for 2 months during  that period. Furthermore, TSA’s operational requirements specify that the  ETP should function for a minimum of 1,460 hours between critical  failures. A critical failure means that an ETP fails to operate and must be  repaired as soon as possible. However, the TSA Chief Technology Officer  at the time stated that the ETPs operated at a much lower average number  of hours before a critical failure occurred because, for example, the dirt  and humidity of some airport environments adversely affected the  equipment. Specifically, from January 2006 through May 2006, the General  Electric ETPs operated for an average of 559 hours before a critical  failure, which means that these ETPs operated on average 38 percent of  the time that they were required to operate before a critical failure  occurred. TSA officials could not provide information on the mean time  between critical failures for the Smiths Detection ETPs. TSA officials  stated that they tested the ETPs in several airports for several months  prior to deployment, but data from these tests did not identify a problem  with mean time between critical failures. One reason for this, a TSA  official stated, was that not enough data were collected during the field  tests. As usage of the ETPs increased, officials stated that they discovered  the ETP was not meeting operational availability requirements. The ETPs  also required replacement filters and other consumables more often than  expected, according to officials, which drove up maintenance costs.
According to TSA officials, because of a variance in operational  availability hours among the deployed ETPs, maintenance problems, and  the high cost of ETP installation at airports, in June 2006, the agency  halted the deployment of the ETP to additional airports and stopped the  planned purchase of additional ETPs. TSA officials plan to continue to use  the 90 ETPs currently deployed to airports. However, without validating  that the ETPs meet their functional requirements, TSA officials do not  have assurance that it is worthwhile to continue to use the ETPs in light of  the cost to maintain and operate them. In addition, TSA officials are  considering what to do with the ETPs that were procured and are  currently in storage. As of April 2009, 116 ETPs were in storage.
TSA did not follow the Acquisition Management System (AMS) guidance  or a knowledge-based acquisition approach before procuring the ETPs,  which contributed to the ETPs not performing as required after they were  deployed to airports. Specifically, AMS guidance provides that testing  should be conducted in an operational environment to validate that the  system meets all functional requirements before deployment. In addition,  our reviews have shown that leading commercial firms follow a  knowledge-based approach to major acquisitions and do not proceed with  large investments unless the product’s design demonstrates its ability to  meet functional requirements and be stable. The developer must show  that the product can be manufactured within cost, schedule, and quality  targets and is reliable before production begins and the system is used in  day-to-day operations. As discussed earlier in this report, TSA officials told  us that they deployed the ETP despite performance problems because  officials wanted to quickly respond to emergent threats. However, TSA did  not provide written documentation to us that described the process used  at the time to make the decision to deploy the ETP or the process that is  currently used to make deployment decisions.
section_title: Using Validated Technologies Would Enhance TSA’s Efforts to Improve Checkpoint Security
paragraphs: TSA has relied on technologies in day-to-day airport operations that have  not been demonstrated to meet their functional requirements in an  operational environment. For example, TSA has substituted existing  screening procedures with screening by the Whole Body Imager even  though its performance has not yet been validated by testing in an  operational environment. In the future, using validated technologies would  enhance TSA’s efforts to improve checkpoint security. Furthermore,  without retaining existing screening procedures until the effectiveness of  future technologies has been validated, TSA officials cannot be sure that  checkpoint security will be improved.
section_title: DHS Is Addressing Coordination and Collaboration Challenges with Stakeholders to Research, Develop, and Deploy Checkpoint Screening Technologies
paragraphs: DHS S&T and TSA coordinated and collaborated with each other and key  stakeholders on their research, development, and deployment activities  for airport checkpoint screening technologies, and DHS is taking actions  to address challenges and strengthen these efforts. Because S&T and  TSA share responsibilities related to the RDT&E, procurement, and  deployment of checkpoint screening technologies, the two organizations  must coordinate with each other and external stakeholders, such as  airport operators and technology vendors. For example, in accordance  with provisions of the Homeland Security Act and ATSA, S&T and TSA are  to coordinate and collaborate with internal and external stakeholders on  matters related to technologies and countermeasures for homeland  security missions. S&T and TSA signed an MOU in August 2006 that  establishes a framework to coordinate their work at the TSL, which tests  and evaluates technologies under development. S&T also established a  Capstone IPT for Explosives Prevention in 2006 to bring S&T, TSA, and  U.S. Secret Service leadership together to identify gaps in explosives  detection capability; prioritize identified gaps; review relevant, ongoing  S&T programs; and develop capabilities to meet identified needs.  However, inconsistent communication and the lack of an overarching test  and evaluation strategy have limited S&T’s and TSA’s ability to coordinate  effectively with one another. To coordinate with the aviation community,  S&T and TSA have hosted industry days and conference calls to discuss  new technologies with airport operators and technology vendors.  Although TSA has taken actions to build partnerships with airport  operators and vendors, it has not established a systematic process to  coordinate with them related to checkpoint screening technologies.  However, TSA officials stated that they are in the beginning stages of  establishing a systematic process.
subsections: section_title: S&T and TSA Are Addressing Coordination and Collaboration Challenges with Each Other on New Checkpoint Screening Technologies, but Challenges Remain
paragraphs: S&T and TSA have taken actions to coordinate and collaborate with each  other related to the RDT&E of checkpoint screening technologies, such as  by communicating priorities and requirements for technologies and  working with each other on the Capstone IPT for Explosives Prevention.  However, S&T and TSA coordination and collaboration were not always  effective due to inconsistent communication and the lack of an  overarching test and evaluation strategy. The Homeland Security Act  assigned responsibilities within the department for coordinating and  integrating the research, development, demonstration, testing, and  evaluation activities of the department, as well as for working with federal  and private sector stakeholders to develop innovative approaches to  produce and deploy the best available technologies for homeland security  missions. The act further assigned S&T with responsibility for  coordinating with other appropriate executive agencies in developing and  carrying out the science and technology agenda of the department to  reduce duplication and identify unmet needs. ATSA had also assigned TSA  with coordination responsibilities, including the coordination of  countermeasures with appropriate departments, agencies, and  instrumentalities of the U.S. government.
S&T and TSA have taken several actions to coordinate and collaborate on  their research and development activities related to checkpoint screening  technologies. First, to coordinate the transition of the TSL from TSA to  S&T, minimize disruption of work, and prevent duplication of effort, S&T  and TSA signed an MOU that defines the roles and responsibilities for the  research and development of homeland security technologies, including  checkpoint screening, and establishes a framework for how to coordinate  their work. Additionally, S&T created the Capstone IPT for Explosives  Prevention, which is co-chaired by the Assistant Secretary for TSA and the  Director of the U.S. Secret Service, to identify and prioritize capabilities  needed to detect explosives; review relevant, ongoing S&T programs; and  develop capabilities to meet the identified needs. The IPT was first  convened in December 2006 to identify research and development  priorities for explosives detection technologies at airport checkpoints as  well as for other transportation modes, and has met periodically since  then. According to TSA officials, the Capstone IPT has enabled TSA to  establish a clear understanding with S&T of TSA’s needs for technology  solutions that meet stringent detection thresholds and throughput  requirements to support the aviation sector. Additionally, the officials  stated that the Capstone IPT has given TSA a better collective  understanding of the technology needs of other DHS components, which  will help DHS identify technology solutions that can be combined to  benefit multiple users. Finally, to follow through on the priorities  established by the Capstone IPT for Explosives Prevention, S&T officials  stated that they established project-level IPTs, including one for airport  checkpoints and one for homemade explosives. S&T officials stated that  they are working with TSA on these project-level IPTs to try to meet the  needs identified by the Capstone IPT. TSA officials further stated that they  have PSP IPTs or working groups to coordinate on technology projects,  establish program goals and objectives, and develop requirements and  time lines. These groups meet on a weekly basis, according to TSA  officials. In April 2008, S&T dissolved the IPT for explosives detection and  replaced it with two separate IPTs, a transportation security IPT, chaired  by TSA and a counter-IED IPT, chaired by the Office of Bombing  Prevention within the National Protection and Programs Directorate and  the United States Secret Service.
Coordination and collaboration efforts between S&T and TSA have helped  in identifying checkpoint screening solutions. For example, S&T and TSA  officials collaborated on a hand-held vapor detection unit called the Fido  PaxPoint. After the August 2006, discovery of the alleged plot to detonate  liquid explosives on board commercial air carriers bound for the United  States from the United Kingdom, S&T and TSA worked together to  identify, develop, and test screening technologies to address this threat.  According to TSA officials, S&T learned that the Department of Defense  had developed a handheld unit that could detect vapors from explosives.  S&T modified the Department of Defense handheld unit, resulting in the  Fido PaxPoint unit to screen liquids and gels at airport checkpoints for  explosives, and S&T helped TSA test and evaluate the device.
Although S&T and TSA have taken steps to coordinate and collaborate  with one another, inconsistent communication and a lack of an  overarching test and evaluation strategy have contributed to coordination  and collaboration challenges. Specifically, communication between S&T  and TSA related to S&T’s basic and applied research efforts and TSA’s  efforts to modify commercially available technologies has been lacking at  times. For example, TSA officials stated that early in the TSL’s transition  to S&T (that is, during fiscal year 2006), TSA did not receive information  from S&T regarding which of TSA’s research and development needs S&T  would fund, which projects related to airport checkpoint technologies  were underway at the TSL, or the time frames to complete those projects.  TSA officials stated that, without this information, TSA was unable to  determine whether its work on modifying commercially available  technologies for screening passengers and carry-on items unnecessarily  duplicated S&T’s research and development efforts, although TSA officials  were not aware of any duplication that occurred. An S&T official further  stated that TSA had not consistently fulfilled its responsibility to provide  clearly defined functional requirements for the equipment to be developed  by S&T and tested by the TSL, nor has TSA consistently given sufficient  notice to the TSL of TSA testing requests. Under the S&T and TSA MOU,  TSA has retained responsibility to establish requirements for equipment  certification and qualification and acceptance testing. Specifically, an S&T  official at the TSL stated that TSA had inadequately defined the functional  requirements and allowed too little time for testing several checkpoint  screening technologies, including the Advanced Technology Systems,  Enhanced Metal Detector II, and Bottled Liquids Scanner. A TSL official  acknowledged that when the TSA was responsible for the TSL, the agency  had not consistently developed requirements prior to testing or  certification of equipment as required by the DHS guidance.
In another example, as previously mentioned in this report, TSA is  developing new certification standards and functional requirements for  screening technologies, and is working with national laboratories to  validate data on aircraft vulnerabilities and generate new computer  models to help TSA develop requirements for explosives detection.  According to the TSA Chief Technology Officer in 2007, the TSL has  custody of the aircraft vulnerability data, but TSL officials had refused to  release the data to the national laboratories as requested by TSA. Although  the TSL later provided 32 of the 46 requested reports, TSA officials  estimated that the TSL’s refusal to release all of the reports had delayed  the effort to develop new certification standards and technology  requirements by about 1 month. The officials added that most of TSA’s  requests to S&T and the TSL had involved similar problems and that,  although the MOU provides a framework for coordination, these types of  problems are related to day-to-day operations and will have to be resolved  as situations arise.
According to S&T and TSA officials, senior-level management turnover at  S&T and TSA contributed to these communication difficulties, as well as  an S&T reorganization which began in August 2006 with the arrival of a  new Under Secretary for Science and Technology. S&T officials further  stated that, prior to the establishment of the PSP working groups, there  was no mechanism for S&T and TSA to communicate information about  priorities, funding, or project timelines. However, through the working  groups, S&T officials stated that S&T and TSA are beginning to achieve  regular communication and interaction at the working level, which allows  for information to be shared in a mutually beneficial way. S&T and TSA  officials also stated that communication with each other has improved  since the MOU was signed in August 2006 and, in particular since the  summer of 2007, although officials from both organizations stated that  further improvement is needed. According to S&T officials, the TSL’s  independent test and evaluation division and TSA have developed an  effective working relationship for several programs, including the Whole  Body Imager and Advanced Technology Systems. In addition, S&T officials  stated that TSA has come to better understand the processes involving the  Capstone IPT and identifying capability needs. According to TSA officials,  the agency is in the process of determining whether a position within its  Office of Security Technology should be established as a liaison with S&T  to improve coordination between S&T and TSA. If the position is created,  the TSA liaison would coordinate and collaborate with S&T officials on  technology projects by assessing the science that supports the  technologies.
The MOU specifies that S&T and TSA will coordinate activities, including  developing an integrated, overarching test and evaluation strategy for  projects to ensure that test and evaluation functions are not duplicative,  adequate resources are outlined and secured for these functions, and  activities are scheduled to support the overall project master schedule.  However, an overarching test and evaluation strategy for checkpoint  technologies has not been developed. The lack of this strategy has  presented coordination and collaboration challenges between S&T and  TSA, and has resulted in the delay of some technologies. For example, a  TSL official stated that the TSL could not accommodate TSA’s request to  test the Advanced Technology Systems, in part, because TSA officials had  not provided sufficient advance notice of their testing needs. TSA officials  said they were working with S&T to develop a project master schedule for  the Advanced Technology Systems. S&T and TSA officials stated that they  plan to develop a test and evaluation strategy to define a coordinated  technology transition process from S&T to TSA by outlining key  responsibilities and criteria to initiate field evaluations of technologies,  but officials could not tell us when the test and evaluation strategy would  be completed.
section_title: DHS Is Using Several Approaches to Strengthen Coordination and Collaboration with Airport Operators, Technology Vendors, and Other Federal Agencies
paragraphs: DHS, through S&T and TSA, coordinates with airport operators, private  sector partners, such as technology vendors, and other federal agencies on  matters related to research and development efforts. This coordination  and collaboration between TSA and airport operators and technology  vendors is important because the agency relies on airport operators to  facilitate the deployment of equipment for testing and day-to-day  operations, and on vendors to develop and manufacture new screening  equipment. However, TSA does not have a systematic process to  coordinate with external stakeholders related to checkpoint screening  technologies, but TSA officials stated that the agency has developed a  draft communications plan, which is being reviewed.
Although TSA does not have a systematic process to coordinate with  technology vendors, airport operators, and other stakeholders related to  the RDT&E, procurement, and deployment of checkpoint screening  technologies, agency officials stated that they plan to develop and  implement such a process. Specifically, TSA officials stated that they have  developed a draft communications plan, which is being reviewed, that will  document the communications process. However, TSA could not provide  an expected completion date for the plan. Although such a plan should  help in providing consistency to the agency’s coordination efforts, without  knowing the specific activities the plan will include or when it will be  implemented, we cannot determine the extent to which the plan may  strengthen coordination. In addition, in September 2007, TSA hired an  Industry Outreach Manager within its Office of Security Technology to  improve relationships with airport operators and communication with  internal TSA stakeholders related to screening technologies, including  checkpoint technologies. In general, the Industry Outreach Manager is the  communications liaison for the Office of Security Technology  stakeholders and customers to exchange ideas, information, and  operational expertise in support of the office’s mission and goals, and to  provide cutting-edge technologies in the most efficient and cost-effective  means possible. In addition to these steps, in January 2007, S&T created a  Corporate Communications Division to coordinate on a wide variety of  science and technology efforts with public and private sector  stakeholders. This office is in the process of developing a tool to assess  the effectiveness of its outreach efforts to industry stakeholders.
The AMS guidance recommends that TSA coordinate with airport  operators to work out all equipment installation issues prior to  deployment. According to TSA officials, the role of the airport operator is  essential in ensuring that solutions under development are suitable for use  in an airport environment, taking into consideration all logistical and  operational constraints and possibilities. As described earlier, provisions  of the Homeland Security Act address the need to coordinate research and  development efforts to further homeland security missions, and reinforce  the importance of coordinating and collaborating with airport operators.  TSA sponsors monthly conference calls with airport operators to discuss  issues of general interest and, according to S&T officials, S&T has  conducted pilot studies with airport operators. However, according to  many of the 33 airport operators we interviewed, TSA’s coordination on  the priorities for and deployment of checkpoint screening technologies  has been inconsistent. Specifically, of the 33 airport operators we  interviewed, 8 had only positive comments about TSA’s coordination and  16 expressed only concerns regarding TSA’s coordination efforts, while 9  expressed both positive comments and concerns. Eleven of the 33 airport  operators told us that TSA had not shared information with them  regarding checkpoint technology needs and priorities. For example, an  airport operator stated that TSA provided specifications for new screening  technologies with sufficient lead time for the airport, which was building a  new checkpoint at the time, and that TSA had numerous coordination  meetings with airport officials to determine space constraints, power  requirements, and other factors. However, this same airport operator  expressed a desire for more coordination by TSA in the agency’s selection  of the technologies to be pilot tested at this airport. Another airport  operator stated that, when TSA asks for volunteers to participate in  checkpoint screening technology pilot programs, it is difficult to agree to  participate because TSA does not clearly communicate the program’s  goals or the capabilities of the technology in the pilot program.
According to airport operators at another airport, TSA officials told them  that they would have the latitude to select the ETP from either of two  vendors on the TSA contract for purchase. According to the airport  officials, after they selected equipment from one of the vendors because it  would fit into the physical layout of the airport’s checkpoint, TSA told the  airport officials that particular ETP vendor was no longer under contract  with TSA. As a result, airport officials stated that they had to redesign the  checkpoint, including raising the ceiling, to accommodate the other  vendor’s ETP. Senior officials in TSA’s Office of Operational Process and  Technology, the office responsible for the development and  implementation of security technologies across several modes of  transportation, subsequently agreed that coordination with airport  managers and other stakeholders could be improved.
According to TSA officials, coordinating with technology vendors is  essential in order to determine what technology platform would be  appropriate and capable of providing the required detection and  throughput capabilities. S&T and TSA have conducted outreach efforts to  coordinate with technology vendors. For example, S&T officials stated  that they have hosted forums known as industry days and attended  conferences to discuss types of technologies needed to be developed and  the department’s priorities for research and development. S&T officials  also stated that they make presentations at technology-related  conferences, symposia, and exhibits, highlighting the work conducted by  S&T. At every industry day and conference, officials said, airport security  and checkpoint screening technologies have been discussed. In addition,  TSA has coordinated with technology vendors through industry days,  individual meetings, and conferences. For example, TSA officials stated  that TSA held industry days with technology vendors to provide a forum to  communicate information to potential vendors on specific technology  testing and procurement efforts, and to allow vendors to ask questions  regarding technology projects and TSA expectations.
Despite these outreach efforts, of the seven vendors we interviewed who  had contracted with TSA to provide checkpoint screening technologies,  officials from five vendors expressed concerns about the agency’s ability  to coordinate with them on current or future needs for checkpoint  technologies. Officials from four of the seven vendors stated that TSA had  not communicated a strategic vision for screening technologies that will  be needed at the checkpoint in the future, and that TSA did not effectively  and clearly communicate standards and requirements for technologies to  vendors. For example, just as TSL officials commented that TSA did not  always provide clear and quantifiable requirements to conduct tests of  screening technologies, vendors stated that TSA had not communicated  effectively about its future needs, such as the operational requirements for  an advanced, integrated checkpoint screening system. Therefore, a  vendor official stated that some of them had taken the initiative to develop  integrated screening technologies in the hope that TSA will eventually  request this type of integrated system. TSA did not express an opinion  regarding the specific concerns raised by the technology vendors, but a  senior TSL official stated that TSA should sponsor better briefings for  vendors after the agency announces its intentions to develop new  technologies. The official stated that these briefings could provide vendors  with an opportunity for open dialogue with TSA and clarification of TSA’s  needs for new technologies. According to a vendor, without adequate  coordination and communication from TSA, the vendors’ ability is limited  in deciding how best to invest their resources to develop new checkpoint  screening technologies.
In addition to coordinating and collaborating with airport operators and  technology vendors, S&T and TSA coordinate and collaborate on the  department’s RDT&E efforts with other federal agencies through  participation in the Technical Support Working Group, which is co-chaired  by the Departments of Defense and State. The Technical Support Working  Group is the U.S. national forum that identifies, prioritizes, and  coordinates interagency research and development of technologies to  combat terrorist acts, including explosives detection technologies. S&T  also coordinates with the national laboratories on homeland security  research. Specifically, S&T’s Office of National Laboratories coordinates  homeland security-related activities and laboratory-directed research  conducted within the Department of Energy’s national laboratories.  According to an S&T senior official, S&T has worked with the national  laboratories to supplement S&T’s research and development of explosives  detection technologies by tasking the national laboratories to conduct  basic research on the characteristics of homemade explosives.
section_title: Conclusions
paragraphs: Researching, developing, testing and evaluating, procuring, and deploying  checkpoint technologies capable of detecting ever-changing threats to the  commercial aviation system is a daunting task. Although TSA has recently  produced a strategic plan that identified a strategy for the PSP, neither the  plan nor the agency’s strategy for researching, developing, and deploying  checkpoint technologies was informed by some key risk management  principles, including a risk assessment, cost–benefit analysis, and  performance measures. Without conducting a risk assessment that  includes all three elements of risk—threat, vulnerability, and  consequence—and completing a cost–benefit analysis to guide the PSP  strategy, TSA has limited assurance that its strategy targets the most  critical risks and that it invests in the most cost-effective new technologies  or other protective measures. Further, without developing performance  measures that assess the extent to which checkpoint screening  technologies achieve the PSP’s security goals and thereby reduce or  mitigate the risk of terrorist attacks, TSA is limited in its ability to  determine the success of its strategy and make needed adjustments. Even  though TSA has not implemented a risk-informed strategy to ensure that  its investments target the most pressing security needs, the agency has  moved forward in investing in new checkpoint screening technologies.
Despite limited progress in the RDT&E, procurement, and deployment of  new checkpoint screening technologies during the first few years that S&T  and TSA had responsibilities related to these technologies, more recently,  the organizations have made progress as reflected by the number of  technologies for which procurement and deployment has been initiated.
TSA faced challenges with the first new technology that it procured and  deployed—the ETP. In the interest of protecting the homeland, it is  understandable that TSA may, at times, not follow all established guidance  in an effort to deploy technologies quickly to address urgent threats and  vulnerabilities. However, deploying the ETP despite unresolved  performance concerns identified during testing of earlier ETP models, as  well as failing to ensure that ETP models that were ultimately deployed  had passed operational testing, increased the risk that the machines would  not perform as intended, resulting in a questionable security benefit. TSA  did not follow AMS guidance that recommended operational testing of a  new technology prior to deployment because it is more cost effective to  resolve performance issues then. While TSA deployed the ETPs to provide  a much-needed capability to automatically screen higher risk passengers  at airport checkpoints, relying on the ETPs could have resulted in airport  checkpoints being more vulnerable given the ETPs’ performance problems  and lack of operational testing. Also, relying on the ETPs to screen these  particular passengers instead of existing screening procedures may not  enhance airport checkpoint security because TSA does not know if ETP  screening provides an improved detection capability compared to existing  screening procedures. Moreover, it is risky to substitute any new  technology for existing screening procedures before the technology has  been proven to be effective through operational testing. Although TSA is  trying to deploy new technologies to address immediate threats, the  problems associated with the development and deployment of the ETPs  may be repeated with other technologies unless TSA adheres to testing  guidance and makes decisions using a knowledge-based acquisition  approach. Finally, it is not clear whether it is worthwhile to continue to  use the ETPs currently deployed to airports due to the costs associated  with maintaining the machines in good, operational condition.
section_title: Recommendations for Executive Action
paragraphs: To help ensure that DHS’s Science and Technology Directorate (S&T) and  Transportation Security Administration (TSA) take a comprehensive, risk- informed approach to the RDT&E, procurement, and deployment of  airport passenger checkpoint screening technologies, and to increase the  likelihood of successful procurements and deployments of such  technologies, in the restricted version of this report, we recommended that  the Assistant Secretary for TSA take the following eight actions:    Conduct a complete risk assessment, including threat, vulnerability, and  consequence assessments, which would apply to the PSP.
Develop cost–benefit analyses to assist in prioritizing investments in new  checkpoint screening technologies.
Develop quantifiable performance measures to assess the extent to which  investments in research, development, and deployment of checkpoint  screening technologies achieve performance goals for enhancing security  at airport passenger checkpoints.
After conducting a complete risk assessment and completing cost-benefit  analyses and quantifiable performance measures for the PSP, incorporate  the results of these efforts into the PSP strategy as determined  appropriate.
To the extent feasible, ensure that operational tests and evaluations have  been successfully completed before deploying checkpoint screening  technologies to airport checkpoints.
Evaluate whether TSA’s current passenger screening procedures should  be revised to require the use of appropriate screening procedures until it is  determined that existing emerging technologies meet their functional  requirements in an operational environment.   In the future, prior to testing or using all checkpoint screening  technologies at airports, determine whether TSA’s passenger screening  procedures should be revised to require the use of appropriate screening  procedures until the performance of the technologies has been validated  through successful testing and evaluation.
Evaluate the benefits of the Explosives Trace Portals that are being used  in airports, and compare the benefits to the costs to operate and maintain  this technology to determine whether it is cost-effective to continue to use  the machines in airports.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of our restricted report to DHS for review and  comment. On April 7, 2009, DHS provided written comments, which are  presented in Appendix II. In commenting on our report, DHS stated that it  agreed with our recommendations and identified actions planned or  underway to implement them. While DHS is taking steps to address our  first and second recommendations related to conducting a risk  assessment, the actions DHS reported TSA had taken or plans to take do  not fully address the intent of the remaining six recommendations.
In its comments, DHS stated that it concurred with our first  recommendation that a risk assessment should be developed for the PSP  and that TSA has two efforts currently underway to do so. Completion of  TSA’s first effort—the Air Domain Risk Analysis (ADRA)—is expected in  the winter of 2009. DHS commented that TSA’s second effort is the Risk  Management and Analysis Toolset (RMAT), a model to simulate the  potential of some technologies to reduce the risk of certain threat  scenarios which will apply specifically to the passenger screening process.  DHS reported that it expects initial results from RMAT to be available  during the second quarter of 2009. DHS further stated that TSA has made  resource allocation and technology decisions that were informed by  consideration of risk (including threat, vulnerability, and consequence),  although not by comparative assessments of these three elements.  However, as we reported, TSA has not conducted a risk assessment for the  PSP, and it is unclear to what extent the ADRA would provide risk  information needed to support the PSP. Until such a risk assessment is  developed and integrated into TSA’s strategy for the PSP, TSA continues  to invest in checkpoint technologies without the benefit of a risk-informed  strategy and increases the possibility that its investments will not address  the highest-priority security needs.
DHS also concurred with our second recommendation that it develop cost-  benefit analyses. DHS commented that TSA is developing an approach for  selecting cost-effective technologies by developing life-cycle cost  estimates and using the RMAT tool to determine how technologies balance  risk (based on current threats) with cost. TSA’s decision to collect cost  and benefit information is a positive first step. Irrespective of how TSA  collects data on the costs and benefits of technologies, it is important, as  we reported, that TSA conduct cost-benefit analysis of each checkpoint  technology that it invests in that weighs the costs and benefits of  technologies relative to the costs and benefits of other solutions. Such  analysis is important because it helps decision-makers determine whether  investments in technologies or in other security programs will provide the  greatest mitigation of risk for the resources that are available.
DHS concurred with our third recommendation that TSA develop  quantifiable performance measures to assess the extent to which TSA’s  investments in checkpoint screening technologies make the checkpoint  more secure, the key mission of the program. DHS commented that it  currently collects quantifiable performance attributes for all potential  acquisitions with regards to metrics, such as detection, false alarm rate,  and operational availability and plans to use information on machines’  attributes as measures of the PSP’s overall effectiveness as a program.  However, these actions will not fully address our third recommendation.  First, information collected on potential acquisitions prior to their  deployment may not reflect their performance in an operational  environment; consequently, relying on information about technologies’  attributes rather than measuring the effectiveness of deployed  technologies to secure the checkpoint will likely have limited value in  terms of measuring the effectiveness of the PSP as a program. Second, as  we reported, the ETP example illustrates that TSA did not collect  information on the ETP’s performance attributes such as operational  availability during laboratory testing prior to procurement and did not  collect data on the ETP’s detection capabilities during tests in an  operational environment. This raises questions about the completeness of  data TSA collects on technologies prior to acquisition and deployment. We  could not verify that TSA collects such information on other technologies  because TSA did not provide documentation to support this comment. As  TSA moves forward in developing performance measures, it is important  that these measures reflect not only efficiency of the technologies to  process passengers but the effectiveness of technologies and other  countermeasures to make the checkpoint more secure and thereby reduce  the risks posed by those most pressing threat scenarios that will be  identified once TSA completes its risk assessment.
In addition, DHS concurred with our fourth recommendation that it  develop a PSP strategic plan that reflects the risk assessment, cost benefit  analysis, and performance measures. DHS commented that TSA plans to  combine results from the RMAT tool and lifecycle cost estimates for  possible technology solutions that strike a balance between risk and  efficient use of funding. DHS also stated it will use RMAT to develop proxy  measures and general “what-if” analysis and risk insights. However, these  actions alone will not satisfy the intent of this recommendation. While it is  possible that proxy measures could be developed to assess the extent to  which TSA’s investments in the research and development of technologies  have achieved program goals of making the checkpoint more secure, to  fully address this recommendation, TSA must also conduct a risk  assessment that addresses the PSP, develop quantifiable measures that  clearly assess the PSP’s progress towards its security goals, and revise its  strategic plan accordingly.
DHS concurred with our fifth recommendation that before deploying  technologies to airport checkpoints, the technologies should successfully  complete testing and evaluation and stated that TSA is taking action to  implement a formal testing process. DHS commented that TSA has  prepared a Test and Evaluation Master Plan (TEMP) that describes a new  formal testing process that is consistent with DHS’s new acquisition  directive. However, the TEMP does not address the intent of this  recommendation. We deleted from this public report our evaluation of  why the TEMP does not address the intent of this recommendation,  because TSA determined our evaluation to be sensitive security  information.
Further, DHS agreed with our sixth and seventh recommendations that  TSA evaluate whether its screening procedures should be revised to  require the use of appropriate procedures until it can be determined that  emerging technologies or future technologies that may be developed meet  all of their requirements in an operational environment. However, DHS’s  comments suggest that it does not intend to implement these  recommendations. DHS commented that the performance of machines is  always measured and confirmed in the laboratory setting prior to  operational field testing. However, we disagree that laboratory testing is  sufficient to address this recommendation. We deleted from this public  report our evaluation of why laboratory testing alone does not address the  intent of this recommendation, because TSA determined our evaluation to  be sensitive security information.
DHS stated that TSA implemented our eighth recommendation that the  agency evaluate the benefits of the ETP, such as its effectiveness, and  conduct a cost-benefit analysis to determine whether the technologies  should remain in use at airports. However, we disagree that TSA has  implemented this recommendation. DHS commented that two actions  fulfilled this recommendation: TSA’s current program management  reviews in which costs are periodically discussed with vendors and the  laboratory testing of the ETP’s detection capabilities. To fully address this  recommendation, a cost-benefit analysis and tests of the ETP’s  effectiveness to detect explosives in an operational environment are  required. As we reported, TSA has not conducted cost-benefit analyses,  which, as noted earlier, should compare costs and benefits of alternative  solutions. Discussions of maintenance costs with vendors on a periodic  basis do not constitute a cost-benefit analysis.
Based on DHS’s written comments, we deleted a reference to the 2004  OMB PART review in a footnote because of updated information from  OMB’s 2008 PART review. DHS also provided us with technical comments,  which we considered and incorporated in the report where appropriate. In  particular, we clarified the wording of a recommendation which originally  stated that TSA should develop quantifiable performance measures to  assess the extent to which investments in research, development, and  deployment of checkpoint screening technologies have mitigated the risks  of a terrorist attack. We altered the wording to state that performance  measures should be developed to assess progress towards security goals.
As agreed with your offices, unless you publicly announce the contents of  this report, we plan no further distribution for 45 days from the report  date. At that time, we will send copies of this report to the Secretary of  Homeland Security, the Assistant Secretary of the Transportation Security  Administration, and appropriate congressional committees.
If you or your staffs have any questions about this report, please contact  me at (202) 512-8777 or LordS@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report addresses the following questions: (1) To what extent has the  Transportation Security Administration (TSA) developed a risk-informed  strategy to prioritize investments in the research and development of  passenger checkpoint screening technologies? (2) What new passenger  checkpoint screening technologies has the Department of Homeland  Security (DHS) researched, developed, tested and evaluated, procured,  and deployed since its creation, and why did TSA halt the first technology  deployment that it initiated—the Explosives Trace Portal (ETP)? (3) To  what extent has DHS coordinated the research, development, test and  evaluation (RDT&E), procurement, and deployment of passenger  checkpoint screening technologies internally and with key stakeholders,  such as airport operators and technology vendors?
To determine the extent to which TSA has developed a risk-informed  strategy to prioritize investments in the research and development of  passenger checkpoint screening technologies, we analyzed program  documents, TSA’s August 2008 strategic plan for checkpoint technologies,  TSA’s September 2007 report on the development of a strategic plan,  technology project plans, and funding. We also compared TSA’s strategic  plan and DHS’s responses regarding their efforts to manage their research  and development investments, with DHS’s guidance from the National  Infrastructure Protection Plan on how to utilize risk management  principles to target funding.
To determine the extent to which DHS researched, developed, tested and  evaluated, procured, and deployed new checkpoint screening technologies  since its creation, and to identify why TSA halted deployment of the ETP,  we analyzed TSA’s strategic plan for checkpoint technologies, TSA’s  Passenger Screening Program (PSP) documentation, including information  on the status of technologies being researched, developed, tested and  evaluated, procured, and deployed. Regarding the ETPs, we analyzed the  functional requirements for the system, contracts with General Electric  and Smiths Detection, and test reports for acceptance tests, regression  tests, and operational tests. We also reviewed ETP deployment schedules  and documentation on operational availability and mean time between  critical failure, and interviewed TSA officials about the reasons that the  ETP deployment was halted. We also compared the ETP test approach  used by S&T and TSA to the Acquisition Management System (AMS)  guidance and knowledge-based acquisition best practices. We also  interviewed TSA and S&T officials to obtain information on current  investments in the research, development, and deployment of checkpoint  technologies, and conducted site visits to the Transportation Security  Laboratory in Atlantic City, New Jersey, and Tyndall Air Force Base,  Florida, to observe testing of new checkpoint technologies. We visited the  TSL because that is where S&T tests and evaluates technologies, including  checkpoint screening technologies. We visited Tyndall Air Force Base  because technologies to detect bottled liquids explosives were being  tested there. Additionally, we analyzed TSA’s passenger screening  standard operating procedures and interviewed various TSA headquarters  officials, 29 Federal Security Directors, 1 Deputy Federal Security  Director, and 5 Assistant Federal Security Directors for Screening, and  visited nine airports where the ETPs had been or were to be deployed or  new checkpoint screening technologies were undergoing pilot testing. We  chose these officials because they are the senior official at the airport in  charge of security and manage TSA’s role in deploying new technologies at  the airport. We selected these nine locations based on the technologies  that had been deployed or were being tested, their geography, size, and  proximity to research and development laboratories.  Of the nine airports  we visited, the ETPs had been or were to be deployed to seven of them,  and other new checkpoint screening technologies were undergoing pilot  demonstrations or testing at two of them. We visited four airports on the  east coast, and three airports on the west coast, and two airports located  in the west and southwestern regions of the United States. To determine  whether the ETP’s requirements had been tested prior to procuring and  deploying them, we selected a non-probability sample of 8 out of the 157  total requirements. We selected the 8 requirements because they were  related to some of the ETP’s key functionality requirements, including  operational effectiveness, operational suitability, and passenger  throughput.
To determine the extent to which DHS has coordinated and collaborated  on the RDT&E, procurement, and deployment of passenger screening  technologies internally and with key stakeholders, we analyzed program  documents, including an August 2006 memorandum of understanding  between TSA and S&T for the management of the Transportation Security  Laboratory (TSL). Additionally, we interviewed Department of State  officials, TSA and S&T officials, seven checkpoint technology vendors, and  airport operators and other officials at airports where ETPs were initially  deployed. Because we selected nonprobability samples of airports to visit  and officials to interview, we cannot generalize the results of what we  learned to airports nationwide. However, the information we gathered  from these locations and officials provided us with insights and  perspectives on DHS’s efforts to operationally test and evaluate, and  deploy checkpoint technologies that could only be obtained from officials  stationed at locations where the technologies had been tested or deployed.  We reviewed the Acquisition Management System, the Aviation and  Transportation Security Act, the Homeland Security Act of 2002, and the  Intelligence Reform and Terrorism Prevention Act and identified  requirements and guidance for coordination and collaboration among  S&T, TSA, and other stakeholders. We also reviewed S&T’s and TSA’s  coordination activities and compared them to TSA program guidance and  GAO’s recommended coordination practices regarding agency  coordination with external stakeholders.
We conducted this performance audit from June 2006 through April 2009  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Homeland Security
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Robert Goldenkoff, Acting  Director; E. Anne Laffoon and Steve Morris, Assistant Directors; and  Joseph E. Dewechter, Analyst-in-Charge, managed this assignment.  Carissa Bryant, Chase Cook, Orlando Copeland, Neil Feldman, and Ryan  MacMaster made significant contributions to the work. Charles Bausell,  Jr., Richard Hung, and Stanley Kostyla assisted with design, methodology,  and data analysis. Michele Mackin assisted with acquisition and  contracting issues. Sally Williamson, Linda Miller, and Kathryn Godfrey  provided assistance in report preparation, and Thomas Lombardi provided  legal support.