Aviation Security: TSA Has Taken Steps to Improve Oversight of Key Programs, but Additional Actions Are Needed
Released: 2015-06-09T00:00:00

Letter

Background

The Aviation and Transportation Security Act (ATSA) established TSA as the primary federal agency with responsibility for securing the nation’s civil aviation system. This responsibility includes the screening of all passengers and property transported from and within the United States by commercial passenger aircraft. In accordance with ATSA, all passengers, their accessible property, and their checked baggage are screened pursuant to TSA-established procedures at the more than 450 airports at which TSA performs, or oversees the performance of, security screening operations. These procedures generally provide, among other things, that passengers pass through security checkpoints where their person, identification documents, and accessible property, are checked by screening personnel.

Secure Flight

Since its implementation, in 2009, Secure Flight has changed from a program that identifies passengers as high risk solely by matching them against federal government watch lists—primarily the No Fly List, comprised of individuals who should be precluded from boarding an aircraft, and the Selectee List, composed of individuals who should receive enhanced screening at the passenger security checkpoint—to one that uses additional lists and risk-based criteria to assign passengers to a risk category: high risk, low risk, or unknown risk. In 2010, following the December 2009 attempted attack on a U.S.-bound flight, which exposed gaps in how agencies used watch lists to screen individuals, TSA began using risk-based criteria to create additional lists for Secure Flight screening. These lists are composed of high-risk passengers who may not be in the Terrorist Screening Database (TSDB), but who TSA has determined should be subject to enhanced screening procedures.Further, in 2011, TSA began screening passengers against additional identities in the TSDB that are not included on the No Fly or Selectee Lists. In addition, as part of TSA Pre✓™, a 2011 program through which TSA designates passengers as low risk for expedited screening, TSA began screening against several new lists of preapproved low-risk travelers. TSA also began conducting TSA Pre✓™ risk assessments, an activity distinct from matching against lists that uses the Secure Flight system to assign passengers scores based upon their travel-related data, for the purpose of identifying them as low risk for a specific flight.

AIT Systems

According to TSA officials, AIT systems, also referred to as full-body scanners, provide enhanced security benefits compared with those of walk-through metal detectors by identifying nonmetallic objects and liquids. Following the deployment of AIT, the public and others raised privacy concerns because AIT systems produced images of passengers’ bodies that image operators analyzed to identify objects or anomalies that could pose a threat to an aircraft or to the traveling public. To mitigate those concerns, TSA began installing automated target recognition (ATR) software on deployed AIT systems in July 2011.with ATR (AIT-ATR) automatically interpret the image and display anomalies on a generic outline of a passenger instead of displaying images of actual passenger bodies. Screening officers use the generic image of a passenger to identify and resolve anomalies on-site in the presence of the passenger.

TSA’s Managed Inclusion Process

TSA Pre✓TM is intended to allow TSA to devote more time and resources at the airport to screening the passengers TSA determined to be higher or unknown risk, while providing expedited screening to those passengers determined to pose a lower risk to the aviation system. To assess whether a passenger is eligible for expedited screening, TSA considers, in general, (1) inclusion on an approved TSA Pre✓TM list of known travelers; (2) results from the automated TSA Pre✓TM risk assessments of all passengers; and (3) real-time threat assessments of passengers, known as Managed Inclusion, conducted at airport checkpoints. Managed Inclusion uses several layers of security, including procedures that randomly select passengers for expedited screening and a combination of behavior detection officers (BDO), who observe passengers to identify high-risk behaviors at TSA-regulated airports; passenger-screening canine teams; and explosives trace detection (ETD) devices to help ensure that passengers selected for expedited screening have not handled explosive material.

Aviation Workers Program

TSA also shares responsibility with airports to vet airport workers to ensure they do not pose a security threat. Pursuant to TSA’s Aviation Workers program, TSA, in collaboration with airport operators and FBI, is to complete applicant background checks—known as security threat assessments—for airport facility workers, retail employees, and airline employees who apply for or are issued a credential for unescorted access to secure areas in U.S. airports.

TSA Has Taken Steps to Improve Oversight of Secure Flight, but Could Take Further Action to Measure Program Performance and Address Screening Errors

In September 2014, we reported on three issues affecting the effectiveness of TSA’s Secure Flight program—(1) the need for additional performance measures to capture progress toward Secure Flight program goals, (2) Secure Flight system matching errors, and (3) mistakes screening personnel have made in implementing Secure Flight at the screening checkpoint. TSA has taken steps to address these issues but additional action would improve the agency’s oversight of the Secure Flight program.

Need for additional performance measures: In September 2014, we found that Secure Flight had established program goals that reflect new program functions since 2009 to identify additional types of high-risk and also low-risk passengers; however, the program performance measures in place at that time did not allow TSA to fully assess its progress toward achieving all of its goals. For example, one program goal was to accurately identify passengers on various watch lists. To assess performance toward this goal, Secure Flight collected various types of data, including the number of passengers TSA identifies as matches to high- and low-risk lists, but did not have measures to assess the extent of system matching errors—for example, the extent to which Secure Flight is missing passengers who are actual matches to these lists. We concluded that additional measures that address key performance aspects related to program goals, and that clearly identify the activities necessary to achieve goals, in accordance with the Government Performance and Results Act, would allow TSA to more fully assess progress toward its goals. Therefore, we recommended that TSA develop such measures, and ensure these measures clearly identify the activities necessary to achieve progress toward the goal. DHS concurred with our recommendation and, according to TSA officials, as of April 2015, TSA’s Office of Intelligence and Analysis was evaluating its current Secure Flight performance goals and measures and determining what new performance measures should be established to fully measure progress against program goals.

Secure Flight system matching errors: In September 2014, we found that TSA lacked timely and reliable information on all known cases of Secure Flight system matching errors, meaning instances where Secure Flight did not identify passengers who were actual matches to these lists. TSA officials told us at the time of our review that when TSA receives information related to matching errors of the Secure Flight system, the Secure Flight Match Review Board reviews this information to determine if any actions could be taken to prevent similar errors from happening again. We identified instances in which the Match Review Board discussed system matching errors, investigated possible actions to address these errors, and implemented changes to strengthen system performance. However, we also found that TSA did not have readily available or complete information on the extent and causes of system matching errors. We recommended that TSA develop a mechanism to systematically document the number and causes of the Secure Flight system’s matching errors, in accordance with federal internal control standards. DHS concurred with our recommendation, and as of April 2015, TSA had developed such a mechanism. However, TSA has not yet demonstrated how it will use the information to improve the performance of the Secure Flight system.

Mistakes at screening checkpoint: We also found in September 2014 that TSA had processes in place to implement Secure Flight screening determinations at airport checkpoints, but could take steps to enhance these processes. Screening personnel at passenger screening checkpoints are primarily responsible for ensuring that passengers receive a level of screening that corresponds to the level of risk determined by Secure Flight by verifying passengers’ identities and identifying passengers’ screening designations. To carry out this responsibility, among other steps, screening personnel are to confirm that the data included on the passenger’s boarding pass and in his or her identity document (such as a driver’s license) match one another, and review the passenger’s boarding pass to identify his or her Secure Flight passenger screening determination. TSA information from May 2012 through February 2014 that we assessed indicates that screening personnel made errors at the checkpoint in screening passengers consistent with their Secure Flight determinations. TSA officials at five of the nine airports where we conducted interviews stated they conducted after-action reviews of such screening errors and used these reviews to take action to address the root causes of those errors. However, we found that TSA did not have a systematic process for evaluating the root causes of these screening errors across airports, which could allow TSA to identify trends across airports and target nationwide efforts to address these issues.

Officials with TSA’s Office of Security Operations told us in the course of our September 2014 review that evaluating the root causes of screening errors would be helpful and stated they were in the early stages of forming a group to discuss these errors. However, TSA was not able to provide documentation of the group’s membership, purpose, goals, time frames, or methodology. Therefore, we recommended in September 2014 that TSA develop a process for evaluating the root causes of screening errors at the checkpoint and then implement corrective measures to address those causes. DHS concurred with our recommendations and has developed a process for collecting and evaluating data on the root causes of screening errors. However, as of April 2015, TSA had not yet shown that the agency has implemented corrective measures to address the root causes.

TSA Performance Assessments of AIT- ATR Did Not Account for All Factors Affecting the System

In March 2014, we reported that, according to TSA officials, checkpoint security is a function of technology, people, and the processes that govern them, however we found that TSA did not include each of those factors in determining overall AIT-ATR system performance. Specifically, we found that TSA evaluated the technology’s performance in the laboratory to determine system effectiveness. However, laboratory test results provide important insights but do not accurately reflect how well the technology will perform in the field with actual human operators. Additionally, we found that TSA did not assess how alarms are resolved by considering how the technology, people, and processes function collectively as an entire system when determining AIT-ATR system performance. AIT-ATR system effectiveness relies on both the technology’s capability to identify threat items and its operators to resolve those threat items.

At the time of our review, TSA officials agreed that it is important to analyze performance by including an evaluation of the technology, operators, and processes, and stated that TSA was planning to assess the performance of all layers of security. According to TSA, the agency conducted operational tests on the AIT-ATR system, as well as follow-on operational tests as requested by DHS’s Director of Operational Test and Evaluation, but those tests were not ultimately used to assess effectiveness of the operators’ ability to resolve alarms, as stated in DHS’s Director of Operational Test and Evaluation’s letter of assessment on the technology. Transportation Security Laboratory officials also agreed that qualification testing conducted in a laboratory setting is not always predictive of actual performance at detecting threat items. Further, laboratory testing does not evaluate the performance of screening officers in resolving anomalies identified by the AIT-ATR system or TSA’s current processes or deployment strategies.

Given that TSA was seeking to procure the second generation of AIT systems, known as AIT-2, we reported that DHS and TSA would be hampered in their ability to ensure that future AIT systems meet mission needs and perform as intended at airports unless TSA evaluated system effectiveness based on both the performance of the AIT-2 technology and screening officers who operate the technology. We recommended that TSA measure system effectiveness based on the performance of the AIT- 2 technology and screening officers who operate the technology while taking into account current processes and deployment strategies. TSA concurred and reported taking steps to address this recommendation. Specifically, in January 2015, DHS stated that TSA’s Office of Security Capabilities evaluated the AIT-2 technology and screening officer as a system during an operational evaluation. However, TSA has not yet provided sufficient documentation showing that this recommendation has been fully addressed.

TSA Has Not Tested the Overall Effectiveness of Its Managed Inclusion Process, But Plans to Conduct Such Testing

In December 2014, we reported that, according to TSA officials, TSA tested the security effectiveness of the individual components of the Managed Inclusion process—such as BDOs and ETD devices—before implementing Managed Inclusion, and TSA determined that each layer alone provides an effective level of security. However, in our prior body of work, we identified challenges in several of the layers used in the Managed Inclusion process, raising questions regarding their effectiveness. For example, in our November 2013 report on TSA’s behavior detection and analysis program, we found that although TSA had taken several positive steps to validate the scientific basis and strengthen program management of its behavior detection and analysis program, TSA had not demonstrated that behavioral indicators can be used to reliably and effectively identify passengers who may pose a threat to aviation security.

Further, TSA officials stated that they had not yet tested the security effectiveness of the Managed Inclusion process as it functions as a whole, as TSA had been planning for such testing over the course of the last year. TSA documentation showed that the Office of Security Capabilities recommended in January 2013 that TSA test the security effectiveness of Managed Inclusion as a system. We reported in December 2014 that according to officials, TSA anticipated that testing would begin in October 2014 and estimated that testing could take 12 to 18 months to complete.

We have also previously reported on challenges TSA has faced in designing studies and protocols to test the effectiveness of security systems and programs in accordance with established methodological practices, such as in the case of the AIT systems discussed previously and in our evaluation of BDO effectiveness. In our December 2014 report, we concluded that ensuring the planned effectiveness testing of the Managed Inclusion process adheres to established evaluation design practices would help TSA provide reasonable assurance that the effectiveness testing will yield reliable results. In general, evaluations are most likely to be successful when key steps are addressed during design, including defining research questions appropriate to the scope of the evaluation, and selecting appropriate measures and study approaches that will permit valid conclusions. As a result, we recommended that to ensure TSA’s planned testing yields reliable results, the TSA Administrator take steps to ensure that TSA’s planned effectiveness testing of the Managed Inclusion process adheres to established evaluation design practices. DHS concurred with our recommendation and began taking steps toward this goal. Specifically, DHS stated that TSA plans to use a test and evaluation process—which calls for the preparation of test and evaluation framework documents including plans, analyses, and a final report describing the test results— for its planned effectiveness testing of Managed Inclusion.

TSA and the FBI Have Addressed a Weakness in TSA’s Oversight of Credentials for Airport Workers

In December 2011, we found that, according to TSA, limitations in its criminal history checks increased the risk that the agency was not detecting potentially disqualifying criminal offenses as part of its Aviation Workers security threat assessments for airport workers. Specifically, we reported that TSA’s level of access to criminal history record information in the FBI’s Interstate Identification Index excluded access to many state records such as information regarding sentencing, release dates, and probation or parole violations, among others. As a result, TSA reported that its ability to look into applicant criminal history records was often incomplete.

We recommended that the TSA and the FBI jointly assess the extent to which this limitation may pose a security risk, identify alternatives to address any risks, and assess the costs and benefits of pursuing each alternative. TSA and the FBI have since taken steps to address this recommendation. For example, in 2014, the agencies evaluated the extent of any risk and, according to TSA and FBI officials, concluded that the risk of incomplete information did exist and could be mitigated through expanded access to state-supplied records. TSA officials reported that the FBI has since taken steps to expand the criminal history record information available to TSA when conducting its security threat assessments for airport workers and others.

Chairman Johnson, Ranking Member Carper, and members of the committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time.

For questions about this statement, please contact Jennifer Grover at (202) 512-7141 or groverj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this statement include Maria Strudwick (Assistant Director), Claudia Becker, Michele Fejfar, and Tom Lombardi. Key contributors for the previous work that this testimony is based on are listed in each product.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.