Aviation Security: TSA Has Taken Steps to Improve Oversight of Key Programs, but Additional Actions Are Needed
Published: 2015-06-09T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since the attacks of September 11, 2001 exposed vulnerabilities in the nation's aviation system, billions of dollars have been spent on a wide range of programs designed to enhance aviation security. Securing commercial aviation remains a daunting task, and continuing fiscal pressure highlights the need for TSA to determine how to allocate its finite resources for the greatest impact. GAO previously reported on TSA's oversight of its aviation security programs, including the extent to which TSA has the information needed to assess the programs.
This testimony focuses on TSA's oversight of aviation security measures including, among other things (1) Secure Flight, (2) Advanced Imaging Technology, and (3) Managed Inclusion. This statement is based on reports and testimonies issued from December 2011 through May 2015. For prior work, GAO analyzed TSA documents and interviewed TSA officials, among other things.
section_title: What GAO Found
paragraphs: The Transportation Security Administration (TSA) has taken steps to improve oversight of Secure Flight—a passenger prescreening program that matches passenger information against watch lists and assigns each passenger a risk category—but could take further action to address screening errors. In September 2014, GAO reported that TSA lacked timely and reliable information on system matching errors—instances where Secure Flight did not identify passengers who were actual matches to watch lists. GAO recommended that TSA systematically document such errors to help TSA determine if actions can be taken to prevent similar errors from occurring. The Department of Homeland Security (DHS) concurred and has developed a mechanism to do so, but has not yet shown how it will use this information to improve system performance. In September 2014, GAO also found that screening personnel made errors in screening passengers at the checkpoint at a level consistent with their Secure Flight risk determinations and that TSA did not have a systematic process for evaluating the root causes of these errors across airports. GAO recommended that TSA develop a process for evaluating the root causes and implement corrective measures to address them. DHS concurred and has developed such a process but has not yet demonstrated implementation of corrective measures.
In March 2014, GAO found that TSA performance assessments of certain full-body scanners used to screen passengers at airports did not account for all factors affecting the systems. GAO reported that the effectiveness of Advanced Imaging Technology (AIT) systems equipped with automated target recognition software (AIT-ATR)—which displays anomalies on a generic passenger outline instead of actual passenger bodies—relied on both the technology's capability to identify potential threat items and its operators' ability to resolve them. However, GAO found that TSA did not include these factors in determining overall AIT-ATR system performance. GAO also found that TSA evaluated the technology's performance in the laboratory—a practice that does not reflect how well the technology will perform with actual human operators. In considering procurement of the next generation of AIT systems (AIT-2), GAO recommended that TSA measure system effectiveness based on the performance of both the technology and the screening personnel. DHS concurred and in January 2015 reported that it has evaluated the AIT-2 technology and screening personnel as a system but has not yet provided sufficient documentation of this effort.
In December 2014, GAO found that TSA had not tested the effectiveness of its overall Managed Inclusion process—a process to assess passenger risk in real time at the airport and provide expedited screening to certain passengers—but had plans to do so. Specifically, GAO found that TSA had tested the effectiveness of individual components of the Managed Inclusion process, such as canine teams, but had not yet tested the effectiveness of the overall process. TSA officials stated that they had plans to conduct such testing. Given that GAO has previously reported on TSA challenges testing the effectiveness of its security programs, GAO recommended that TSA ensure its planned testing of the Managed Inclusion process adhere to established evaluation design practices. DHS concurred and has plans to use a test and evaluation process for its planned testing of Managed Inclusion.
section_title: What GAO Recommends
paragraphs: GAO has previously made recommendations to DHS to strengthen TSA's oversight of aviation security programs. DHS generally agreed and has actions underway to address them. Consequently, GAO is not making any new recommendations in this testimony.

section_title: Letter
paragraphs: I am pleased to be here today to discuss our past work examining the  Transportation Security Administration’s (TSA) oversight of its passenger  and airport worker screening programs. It has been nearly 14 years since  the attacks of September 11, 2001 exposed vulnerabilities in the nation’s  aviation system. Since then, billions of dollars have been spent on a wide  range of programs designed to enhance aviation security. However,  securing commercial aviation operations remains a daunting task—with  hundreds of airports, thousands of aircraft, and thousands of flights daily  carrying millions of passengers and pieces of carry-on and checked  baggage. According to TSA, the threat to civil aviation has not  diminished—underscoring the need for effective passenger and airport  worker screening programs. As the fiscal pressures facing the  government continue, so too does the need for TSA to determine how to  allocate its finite resources to have the greatest impact on addressing  threats and strengthening the effectiveness of its programs and activities.  GAO previously reported on TSA’s oversight of its aviation security  programs, including the extent to which TSA has the information needed  to assess the programs.
As requested, my testimony today focuses on TSA’s oversight of four key  aviation security measures:
Secure Flight: a passenger prescreening program that matches  passenger information against federal government watch lists and  other information to assign each passenger to a risk category;
Advanced Imaging Technology (AIT): a full body scanner used to  screen passengers in the nation’s airports;
Managed Inclusion: a process that TSA uses to determine  passengers’ eligibility for expedited screening at some passenger  screening checkpoints, via Pre✓TM lanes; and
Aviation Workers: a program by which TSA and airports, in  collaboration with the Federal Bureau of Investigation (FBI), vet  applicants against the FBI’s criminal history records, among other  databases, and issue credentials to qualifying airport facility workers,  retail employees, and airline employees, among others.
This statement is based on our reports and testimonies issued from  December 2011 through May 2015 related to TSA’s efforts to oversee its  aviation security measures. For our past work, we reviewed applicable  laws, regulations, and policies as well as TSA program documents;  results from AIT testing and screener performance reviews; decision  memorandums; and other documents. We also visited airports—six for  our Managed Inclusion work and nine for our Secure Flight work—which  we selected based on a variety of factors, such as volume of passengers  screened and geographic dispersion, and interviewed Department of  Homeland Security (DHS), TSA, FBI officials, among other things. Further  details on the scope and methodology for the previously issued reports  and testimonies are available within each of the published products. We  conducted the work on which this statement is based in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.
subsections: section_title: Background
paragraphs: The Aviation and Transportation Security Act (ATSA) established TSA as  the primary federal agency with responsibility for securing the nation’s  civil aviation system. This responsibility includes the screening of all passengers and property transported from and within the United States  by commercial passenger aircraft. In accordance with ATSA, all  passengers, their accessible property, and their checked baggage are  screened pursuant to TSA-established procedures at the more than 450  airports at which TSA performs, or oversees the performance of, security  screening operations. These procedures generally provide, among other  things, that passengers pass through security checkpoints where their  person, identification documents, and accessible property, are checked  by screening personnel.
subsections: section_title: Secure Flight
paragraphs: Since its implementation, in 2009, Secure Flight has changed from a  program that identifies passengers as high risk solely by matching them  against federal government watch lists—primarily the No Fly List,  comprised of individuals who should be precluded from boarding an  aircraft, and the Selectee List, composed of individuals who should  receive enhanced screening at the passenger security checkpoint—to  one that uses additional lists and risk-based criteria to assign passengers  to a risk category: high risk, low risk, or unknown risk. In 2010, following the December 2009 attempted attack on a U.S.-bound flight, which  exposed gaps in how agencies used watch lists to screen individuals,  TSA began using risk-based criteria to create additional lists for Secure  Flight screening. These lists are composed of high-risk passengers who  may not be in the Terrorist Screening Database (TSDB), but who TSA  has determined should be subject to enhanced screening procedures.Further, in 2011, TSA began screening passengers against additional  identities in the TSDB that are not included on the No Fly or Selectee  Lists. In addition, as part of TSA Pre✓™, a 2011 program through which  TSA designates passengers as low risk for expedited screening, TSA  began screening against several new lists of preapproved low-risk  travelers. TSA also began conducting TSA Pre✓™ risk assessments, an  activity distinct from matching against lists that uses the Secure Flight  system to assign passengers scores based upon their travel-related data,  for the purpose of identifying them as low risk for a specific flight.
section_title: AIT Systems
paragraphs: According to TSA officials, AIT systems, also referred to as full-body  scanners, provide enhanced security benefits compared with those of  walk-through metal detectors by identifying nonmetallic objects and  liquids. Following the deployment of AIT, the public and others raised  privacy concerns because AIT systems produced images of passengers’  bodies that image operators analyzed to identify objects or anomalies that  could pose a threat to an aircraft or to the traveling public. To mitigate  those concerns, TSA began installing automated target recognition (ATR)  software on deployed AIT systems in July 2011.with ATR (AIT-ATR) automatically interpret the image and display  anomalies on a generic outline of a passenger instead of displaying  images of actual passenger bodies. Screening officers use the generic  image of a passenger to identify and resolve anomalies on-site in the  presence of the passenger.
section_title: TSA’s Managed Inclusion Process
paragraphs: TSA Pre✓TM is intended to allow TSA to devote more time and resources  at the airport to screening the passengers TSA determined to be higher or  unknown risk, while providing expedited screening to those passengers  determined to pose a lower risk to the aviation system. To assess  whether a passenger is eligible for expedited screening, TSA considers,  in general, (1) inclusion on an approved TSA Pre✓TM list of known  travelers; (2) results from the automated TSA Pre✓TM risk assessments  of all passengers; and (3) real-time threat assessments of passengers,  known as Managed Inclusion, conducted at airport checkpoints. Managed  Inclusion uses several layers of security, including procedures that  randomly select passengers for expedited screening and a combination of  behavior detection officers (BDO), who observe passengers to identify  high-risk behaviors at TSA-regulated airports; passenger-screening  canine teams; and explosives trace detection (ETD) devices to help  ensure that passengers selected for expedited screening have not  handled explosive material.
section_title: Aviation Workers Program
paragraphs: TSA also shares responsibility with airports to vet airport workers to  ensure they do not pose a security threat. Pursuant to TSA’s Aviation  Workers program, TSA, in collaboration with airport operators and FBI, is  to complete applicant background checks—known as security threat  assessments—for airport facility workers, retail employees, and airline  employees who apply for or are issued a credential for unescorted access  to secure areas in U.S. airports.
section_title: TSA Has Taken Steps to Improve Oversight of Secure Flight, but Could Take Further Action to Measure Program Performance and Address Screening Errors
paragraphs: In September 2014, we reported on three issues affecting the  effectiveness of TSA’s Secure Flight program—(1) the need for additional  performance measures to capture progress toward Secure Flight program  goals, (2) Secure Flight system matching errors, and (3) mistakes  screening personnel have made in implementing Secure Flight at the  screening checkpoint. TSA has taken steps to address these issues but  additional action would improve the agency’s oversight of the Secure  Flight program.
Need for additional performance measures: In September 2014, we  found that Secure Flight had established program goals that reflect new  program functions since 2009 to identify additional types of high-risk and  also low-risk passengers; however, the program performance measures  in place at that time did not allow TSA to fully assess its progress toward  achieving all of its goals. For example, one program goal was to  accurately identify passengers on various watch lists. To assess  performance toward this goal, Secure Flight collected various types of  data, including the number of passengers TSA identifies as matches to  high- and low-risk lists, but did not have measures to assess the extent of  system matching errors—for example, the extent to which Secure Flight is  missing passengers who are actual matches to these lists. We concluded  that additional measures that address key performance aspects related to  program goals, and that clearly identify the activities necessary to achieve  goals, in accordance with the Government Performance and Results Act,  would allow TSA to more fully assess progress toward its goals.  Therefore, we recommended that TSA develop such measures, and  ensure these measures clearly identify the activities necessary to achieve  progress toward the goal. DHS concurred with our recommendation and,  according to TSA officials, as of April 2015, TSA’s Office of Intelligence  and Analysis was evaluating its current Secure Flight performance goals  and measures and determining what new performance measures should  be established to fully measure progress against program goals.
Secure Flight system matching errors: In September 2014, we found  that TSA lacked timely and reliable information on all known cases of  Secure Flight system matching errors, meaning instances where Secure  Flight did not identify passengers who were actual matches to these lists.  TSA officials told us at the time of our review that when TSA receives  information related to matching errors of the Secure Flight system, the  Secure Flight Match Review Board reviews this information to determine  if any actions could be taken to prevent similar errors from happening  again. We identified instances in which the Match Review Board  discussed system matching errors, investigated possible actions to  address these errors, and implemented changes to strengthen system  performance. However, we also found that TSA did not have readily  available or complete information on the extent and causes of system  matching errors. We recommended that TSA develop a mechanism to  systematically document the number and causes of the Secure Flight  system’s matching errors, in accordance with federal internal control  standards. DHS concurred with our recommendation, and as of April  2015, TSA had developed such a mechanism. However, TSA has not yet  demonstrated how it will use the information to improve the performance  of the Secure Flight system.
Mistakes at screening checkpoint: We also found in September 2014  that TSA had processes in place to implement Secure Flight screening  determinations at airport checkpoints, but could take steps to enhance  these processes. Screening personnel at passenger screening  checkpoints are primarily responsible for ensuring that passengers  receive a level of screening that corresponds to the level of risk  determined by Secure Flight by verifying passengers’ identities and  identifying passengers’ screening designations. To carry out this  responsibility, among other steps, screening personnel are to confirm that  the data included on the passenger’s boarding pass and in his or her  identity document (such as a driver’s license) match one another, and  review the passenger’s boarding pass to identify his or her Secure Flight  passenger screening determination. TSA information from May 2012  through February 2014 that we assessed indicates that screening  personnel made errors at the checkpoint in screening passengers  consistent with their Secure Flight determinations. TSA officials at five of  the nine airports where we conducted interviews stated they conducted  after-action reviews of such screening errors and used these reviews to  take action to address the root causes of those errors. However, we  found that TSA did not have a systematic process for evaluating the root  causes of these screening errors across airports, which could allow TSA  to identify trends across airports and target nationwide efforts to address  these issues.
Officials with TSA’s Office of Security Operations told us in the course of  our September 2014 review that evaluating the root causes of screening  errors would be helpful and stated they were in the early stages of  forming a group to discuss these errors. However, TSA was not able to  provide documentation of the group’s membership, purpose, goals, time  frames, or methodology. Therefore, we recommended in September 2014  that TSA develop a process for evaluating the root causes of screening  errors at the checkpoint and then implement corrective measures to  address those causes. DHS concurred with our recommendations and  has developed a process for collecting and evaluating data on the root  causes of screening errors. However, as of April 2015, TSA had not yet  shown that the agency has implemented corrective measures to address  the root causes.
section_title: TSA Performance Assessments of AIT- ATR Did Not Account for All Factors Affecting the System
paragraphs: In March 2014, we reported that, according to TSA officials, checkpoint  security is a function of technology, people, and the processes that  govern them, however we found that TSA did not include each of those  factors in determining overall AIT-ATR system performance.  Specifically, we found that TSA evaluated the technology’s performance  in the laboratory to determine system effectiveness. However, laboratory  test results provide important insights but do not accurately reflect how  well the technology will perform in the field with actual human operators.  Additionally, we found that TSA did not assess how alarms are resolved  by considering how the technology, people, and processes function  collectively as an entire system when determining AIT-ATR system  performance. AIT-ATR system effectiveness relies on both the  technology’s capability to identify threat items and its operators to resolve  those threat items.
At the time of our review, TSA officials agreed that it is important to  analyze performance by including an evaluation of the technology,  operators, and processes, and stated that TSA was planning to assess  the performance of all layers of security. According to TSA, the agency  conducted operational tests on the AIT-ATR system, as well as follow-on  operational tests as requested by DHS’s Director of Operational Test and  Evaluation, but those tests were not ultimately used to assess  effectiveness of the operators’ ability to resolve alarms, as stated in  DHS’s Director of Operational Test and Evaluation’s letter of assessment  on the technology. Transportation Security Laboratory officials also  agreed that qualification testing conducted in a laboratory setting is not  always predictive of actual performance at detecting threat items. Further,  laboratory testing does not evaluate the performance of screening officers  in resolving anomalies identified by the AIT-ATR system or TSA’s current  processes or deployment strategies.
Given that TSA was seeking to procure the second generation of AIT  systems, known as AIT-2, we reported that DHS and TSA would be  hampered in their ability to ensure that future AIT systems meet mission  needs and perform as intended at airports unless TSA evaluated system  effectiveness based on both the performance of the AIT-2 technology and  screening officers who operate the technology. We recommended that  TSA measure system effectiveness based on the performance of the AIT- 2 technology and screening officers who operate the technology while  taking into account current processes and deployment strategies. TSA  concurred and reported taking steps to address this recommendation.  Specifically, in January 2015, DHS stated that TSA’s Office of Security  Capabilities evaluated the AIT-2 technology and screening officer as a  system during an operational evaluation. However, TSA has not yet  provided sufficient documentation showing that this recommendation has  been fully addressed.
section_title: TSA Has Not Tested the Overall Effectiveness of Its Managed Inclusion Process, But Plans to Conduct Such Testing
paragraphs: In December 2014, we reported that, according to TSA officials, TSA  tested the security effectiveness of the individual components of the  Managed Inclusion process—such as BDOs and ETD devices—before  implementing Managed Inclusion, and TSA determined that each layer  alone provides an effective level of security. However, in our prior body  of work, we identified challenges in several of the layers used in the  Managed Inclusion process, raising questions regarding their  effectiveness. For example, in our November 2013 report on TSA’s  behavior detection and analysis program, we found that although TSA  had taken several positive steps to validate the scientific basis and  strengthen program management of its behavior detection and analysis  program, TSA had not demonstrated that behavioral indicators can be  used to reliably and effectively identify passengers who may pose a threat  to aviation security.
Further, TSA officials stated that they had not yet tested the security  effectiveness of the Managed Inclusion process as it functions as a  whole, as TSA had been planning for such testing over the course of the  last year. TSA documentation showed that the Office of Security  Capabilities recommended in January 2013 that TSA test the security  effectiveness of Managed Inclusion as a system. We reported in  December 2014 that according to officials, TSA anticipated that testing  would begin in October 2014 and estimated that testing could take 12 to  18 months to complete.
We have also previously reported on challenges TSA has faced in  designing studies and protocols to test the effectiveness of security  systems and programs in accordance with established methodological  practices, such as in the case of the AIT systems discussed previously  and in our evaluation of BDO effectiveness. In our December 2014  report, we concluded that ensuring the planned effectiveness testing of  the Managed Inclusion process adheres to established evaluation design  practices would help TSA provide reasonable assurance that the  effectiveness testing will yield reliable results. In general, evaluations  are most likely to be successful when key steps are addressed during  design, including defining research questions appropriate to the scope of  the evaluation, and selecting appropriate measures and study  approaches that will permit valid conclusions. As a result, we  recommended that to ensure TSA’s planned testing yields reliable results,  the TSA Administrator take steps to ensure that TSA’s planned  effectiveness testing of the Managed Inclusion process adheres to  established evaluation design practices. DHS concurred with our  recommendation and began taking steps toward this goal. Specifically,  DHS stated that TSA plans to use a test and evaluation process—which  calls for the preparation of test and evaluation framework documents  including plans, analyses, and a final report describing the test results— for its planned effectiveness testing of Managed Inclusion.
section_title: TSA and the FBI Have Addressed a Weakness in TSA’s Oversight of Credentials for Airport Workers
paragraphs: In December 2011, we found that, according to TSA, limitations in its  criminal history checks increased the risk that the agency was not  detecting potentially disqualifying criminal offenses as part of its Aviation  Workers security threat assessments for airport workers. Specifically,  we reported that TSA’s level of access to criminal history record  information in the FBI’s Interstate Identification Index excluded access to  many state records such as information regarding sentencing, release  dates, and probation or parole violations, among others. As a result,  TSA reported that its ability to look into applicant criminal history records  was often incomplete.
We recommended that the TSA and the FBI jointly assess the extent to  which this limitation may pose a security risk, identify alternatives to  address any risks, and assess the costs and benefits of pursuing each  alternative. TSA and the FBI have since taken steps to address this  recommendation. For example, in 2014, the agencies evaluated the  extent of any risk and, according to TSA and FBI officials, concluded that  the risk of incomplete information did exist and could be mitigated through  expanded access to state-supplied records. TSA officials reported that  the FBI has since taken steps to expand the criminal history record  information available to TSA when conducting its security threat  assessments for airport workers and others.
Chairman Johnson, Ranking Member Carper, and members of the  committee, this completes my prepared statement. I would be pleased to  respond to any questions that you may have at this time.
For questions about this statement, please contact Jennifer Grover at  (202) 512-7141 or groverj@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this statement. Individuals making key contributions to this statement  include Maria Strudwick (Assistant Director), Claudia Becker, Michele  Fejfar, and Tom Lombardi. Key contributors for the previous work that this  testimony is based on are listed in each product.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.