Aviation Security: TSA Is Taking Steps to Improve Expedited Screening Effectiveness, but Improvements in Screener Oversight Are Needed
Released: 2016-06-07T00:00:00

Letter

Background

Expedited Screening

In 2011, TSA began developing new expedited security procedures intended to strengthen security and improve the passenger experience by shortening lines and wait times, and in October 2011, implemented its expedited screening program—known as TSA Pre®. According to TSA, expedited screening involves a relatively more efficient and convenient screening process for individuals from whom TSA has obtained sufficient information to determine them to be lower risk, compared with the standard screening process for a traveler for whom TSA does not have such information. For example, passengers eligible for expedited screening may no longer have to remove their shoes; may leave their permitted liquids, gels, and laptops in carry-on baggage; and are not required to divest light outerwear, jackets, or belts when passing through screening checkpoints unless the screening technology alarms, in which case these items must be removed.

TSA uses the following methods to assess whether a passenger is low risk and therefore eligible for expedited screening.

Approved TSA Pre ® lists of known travelers—These lists are comprised of individuals whom TSA has determined to be low risk by virtue of their membership in a specific group, such as active duty military members, or based on group vetting requirements, or if approved through the TSA Pre® Application Program.

Automated TSA Pre ® risk assessments of all passengers— Using these assessments, TSA assigns passengers scores based upon information available to TSA to identify low risk passengers eligible for expedited screening for a specific flight prior to the passengers’ arrival at the airport.

Real-time threat assessments through Managed Inclusion— These assessments use several layers of security, including procedures that randomly select passengers for expedited screening, behavior detection officers who observe passengers to identify high- risk behaviors, and passenger screening canine teams to help ensure that passengers selected for expedited screening have not handled explosive material. TSA developed Managed Inclusion as a tool to improve the efficiency of dedicated TSA Pre ® screening lanes.

When TSA began offering expedited screening at airports in the summer of 2011, TSOs initially provided such screenings in standard lanes to passengers aged 12 and younger, and subsequently extended expedited screening to certain flight crew members and then to passengers aged 75 and older. However, in October 2011, TSA began to expand the concept of expedited airport screening to more of the flying public by piloting the TSA Pre ® program. This pilot program allowed certain frequent fliers of two air carriers to experience expedited screening at four airports. These frequent fliers became eligible for screening in dedicated expedited screening lanes, called TSA Pre ® lanes, because they had opted into the TSA Pre ® program through the air carrier with which they had attained frequent flier status.

Since October 2011, TSA has further expanded the known traveler populations eligible for expedited screening. TSA established separate TSA Pre ® lists for additional low-risk passenger populations, including members of the U.S. armed forces, Congressional Medal of Honor Society Members, members of the Homeland Security Advisory Council, and Members of Congress, among others. In March 2015, TSA officials stated that the Army, Navy, Marine Corps, Air Force, and Coast Guard branches of the U.S. armed forces, as well as Reserve and National Guard personnel, were eligible to participate. TSA also created its own TSA Pre ® list composed of individuals who apply to be preapproved as low-risk travelers through the TSA Pre ® Application Program, an initiative launched in December 2013. To apply, individuals must visit an enrollment center where they provide biographic information (i.e., name, date of birth, and address), valid identity and citizenship documentation, and fingerprints to undergo a TSA Security Threat Assessment. Applicants must be U.S. citizens, U.S. nationals, or lawful permanent residents, and cannot have been convicted of certain crimes. As of December 2015, about 8.8 million individuals were eligible, through TSA Pre ® lists, for expedited screening. Figure 1 shows the populations for each TSA Pre ® list.

Assessing TSO Performance

To carry out passenger and checked baggage screening operations, TSA employs TSOs at the vast majority of the nation’s commercial airports. TSOs must complete the New Hire Training Program (NHTP), which includes at least 40 hours of classroom training focused on their duties as a screener, a minimum of 60 hours of on-the-job training, and certification tests for the functions they will be performing. In addition, TSOs are required to take recurrent training throughout the year to maintain proficiency with skills learned during the NHTP, and to remain up-to-date with changes in screening standard operating procedures (SOP), as well as emerging threats. Also, TSOs who are absent from their screening duties for a period of time must undergo some level of “return-to-duty” training based on the amount of time they were absent. Lastly, if TSOs fail an operational test, they are required to take remedial training customized to fit the specific screener’s performance improvement needs. Furthermore, each year, TSA conducts certification testing for its airport security screeners, and in an effort to measure the performance of aviation security screening, both TSA and the DHS-OIG conduct regular covert testing of TSA screening operations.

Recent covert tests conducted by the DHS-OIG highlighted the following areas of concern: (1) the effectiveness of the passenger screening process, (2) TSA’s Advanced Imaging Technology (AIT) screening equipment, (3) related automated target recognition software used by the AIT systems, and (4) checkpoint screener performance in identifying and resolving potential security threats at airport checkpoints. In response to the results of the covert testing, TSA updated its screening SOPs, retrained TSOs to address the Inspector General’s findings, and provided additional classroom training nationwide to all TSOs. In addition, TSA developed new measures of effectiveness that it expects will better emphasize the agency’s goals for improving security effectiveness by focusing the measures on both the screening system and workforce in the areas of readiness and performance.

To measure TSO performance, TSA uses the following performance measurement tests:

Annual Proficiency Reviews (APR) evaluate TSOs’ ability to identify prohibited items on an X-ray machine, ability to resolve explosives detection system machine alarms using the appropriate tools, and whether TSOs can perform various practical skills such as pat downs, bag searches, and use of explosive trace detection technology. If a screener does not pass one of the components of the APR after two; or, in some cases, three attempts, they are subject to removal from their position.

Threat Image Projection (TIP) monitors TSOs’ ability to identify prohibited items in x-ray images of carry-on baggage at the passenger checkpoint by projecting fictional threat items onto the bags. TIP also aides in keeping TSOs focused and attentive, and in keeping their skills sharp in identifying items they do not routinely see. According to TSA policy, FSDs must monitor TIP results monthly and, if one of their TSOs identifies less than a target percentage of TIP images accurately in a month, then the TSO is required to attend remedial training.

Aviation Screening Assessment Program (ASAP) is a form of covert testing to measure, at a national level, TSO screening performance against screening SOPs. TSA’s Office of Security Operations utilizes local role players to take prohibited items such as knives, guns, or simulated improvised explosive devices, through the screening checkpoints to test TSOs performance in accurately identifying those items. ASAP tests are conducted by TSA at both screening checkpoints and checked baggage screening areas. The tests are designed to assess the operational effectiveness of screeners. TSA implemented a series of improvements to ASAP in 2010 and 2012 that introduced (1) specific testing scenarios to improve the level of standardization, (2) a formalized debriefing process, (3) training scenarios by which airports can tailor lessons learned to their operations, and (4) a strategy for allowing the reporting of comparable testing results, over time, from the airports. After these improvements, TSA renamed the program ASAP Advantage. TSA implements ASAP Advantage according to a 6- month testing schedule, and at the completion of each 6-month cycle, generates a report identifying trends in screening performance.

TSA Is Taking Steps to Improve the Security Effectiveness of Expedited Screening

TSA has taken steps to improve the security effectiveness of expedited screening since we issued our December 2014 report. Specifically, TSA has begun planning for the testing of the security effectiveness of the Managed Inclusion process as an overall system–ensuring that the testing adheres to established design practices. In addition, TSA has adjusted the TSA Pre ® Risk Assessment program algorithm used to assign passengers scores and identify low risk passengers because the DHS-OIG found that the algorithm allowed a high-risk individual access to expedited screening. Also, according to TSA documentation, TSA reduced the number of passengers screened by the Managed Inclusion process by limiting its use to airports that have canine teams to detect explosives.

Our December 2014 report found that TSA has tested the effectiveness of the individual Managed Inclusion security layers, but that TSA had not yet tested the Managed Inclusion process as an overall system. We stated that our previous work identified challenges in several of the layers used in the Managed Inclusion process, raising concerns regarding their overall effectiveness. For example, in November 2013, we found that TSA had not demonstrated that behavioral indicators can be used to reliably and effectively identify passengers who may pose a threat to aviation security. While TSA is taking steps to revise and test the behavior detection program, such as working to provide scientifically validated evidence that demonstrates that behavioral indicators can be used to identify passengers who may pose a risk to aviation security, the issue remains open. As of May 2016, TSA told us that it is taking actions to optimize the effectiveness of its behavior detection program and plans to begin an operational test of these efforts in September 2016. In our December 2014 report, we noted that TSA has previously faced challenges designing studies to test the security effectiveness of programs in accordance with established methodological practices such as ensuring an adequate sample size or randomly selecting items in a study to ensure the results can be generalizable—key features of established evaluation design practices. As a result, we recommended that TSA take steps to ensure and document that its planned testing of the Managed Inclusion process as a system adheres to established evaluation design practices. DHS concurred with our recommendation, and according to TSA officials, TSA has developed a data collection and analysis plan to be used for the testing of the Managed Inclusion system. As of May 2016, TSA is reviewing and finalizing the plan and intends to test it at ten airports in late summer or early fall 2016 according to TSA officials. We will continue to monitor TSA’s progress in addressing this recommendation.

In addition, according to a TSA memorandum dated November 2015, TSA made changes to the TSA Pre ® Risk Assessment program and Managed Inclusion process to enhance aviation security as a result of the findings and recommendations included in three prior DHS-OIG audit reports. Specifically, TSA made changes to the TSA Pre ® Risk Assessment program algorithm used to assign passenger scores because the DHS-OIG found that the program created a potential aviation security vulnerability in at least one instance by identifying a convicted felon as low risk and eligible for expedited screening. As a result, TSA recognized the increased level of uncertainty surrounding a potential threat posed by individuals who obtain expediting screening eligibility through the Risk Assessment program as compared to individuals who have been vetted and are included on one of the TSA Pre ® lists. Following the public release of the DHS-OIG’s covert testing results, TSA officials stated that TSA began a thorough review of checkpoint operations, and as a part of that review, evaluated all methods in which individuals without background checks became eligible for expedited screening. As a result of this evaluation and based on a recommendation from another DHS-OIG audit, TSA documentation shows that TSA discontinued the use of Explosives Trace Detection (ETD) devices as a method used to conduct real time threat assessments and is now limiting the use of Managed Inclusion to airports that employed canine team to detect explosives. According to the TSA administrator, these changes have resulted in a 20 percent decrease in the number of individuals who receive expedited screening.

To address this decrease in expedited screening and its likely effect on passenger wait times, TSA plans to undertake efforts to increase the number of individuals included on the TSA Pre ® lists of known travelers from the nearly 8.8 million individuals currently enrolled to 25 million individuals. In order to achieve this increase, TSA plans to change the enrollment process, increase marketing and communication efforts, and expand the number of contractors that provide enrollment services. TSA estimates that the TSA Pre ® lists of known travelers will total 25 million individuals in 3 to 4 years.

While TSA Uses TSO Screening Performance Data, It Is Constrained by Incomplete and Unreliable Data and a Lack of Data Analysis and Assessment Follow-Up

TSA utilizes data on TSO performance obtained from its various testing programs to help to ensure that individual TSOs are (1) qualified to conduct passenger and checked baggage screening based on Annual Proficiency Reviews and resulting recertifications, and (2) demonstrate proficiency, during live screening operations, in their adherence to screening standard operating procedures and other TSA guidance for detecting prohibited items. However, incomplete and unreliable data and limited analysis constrains TSA’s ability to determine the true level of TSO performance in screening passengers and baggage for prohibited items. Without this knowledge, TSA cannot fully identify and make necessary improvements to screening operations.

TSA Uses TSO Performance Data to Inform TSO Training

TSA has several programs in place to yield data for oversight and analysis of TSO screening performance. As noted previously, TSA relies on Annual Proficiency Reviews (APR) to recertify TSOs. TSA’s Office of Training and Workforce Engagement examined the results of specific APR component tests administered in 2013 to inform their development of related courses for the annual training curriculum for TSOs, known as the National Training Plan (NTP). Specifically, TSA officials stated they reviewed the results of these component tests—screening of individuals with disabilities, bag searches, and standard pat downs—and added training to the fiscal year 2015 NTP to specifically address the deficiencies they identified.

In addition, during live screening operations, TSA also monitors individual TSO performance through (1) Threat Image Projection (TIP) testing by local TSA officials which assesses the TSOs’ proficiency at identifying prohibited items in X-ray images of passengers’ carry-on baggage, and (2) Aviation Screening Assessment Program (ASAP) covert tests which assess the TSOs’ ability to properly adhere to screening standard operating procedures and prevent the passage of prohibited items through passenger and baggage checkpoints. TSA monitors the results of these testing programs to determine whether individual TSOs need remedial training based on the results.

Incomplete and Unreliable Data and A Lack of National Analysis Limit TSA’s Ability To Assess TSO Performance

Incomplete Data

TSA policy requires airport personnel to manually download TIP testing results from their individual X-ray machines and upload the monthly data into TSA’s national database repository for TSA results. According to TSA headquarters personnel responsible for overseeing the TIP program, they use these uploaded results to determine if any adjustments are needed to the quality or usefulness of the library of images maintained in the TIP system nationwide. However, as we found in May 2016, some airports had failed to submit TIP data as required. As shown in figure 2, some airports in all five airport risk categories did not report any TIP results nationally over the course of a year from fiscal year 2010 through fiscal year 2013. During the fiscal year 2009 through 2014 time frame, fiscal year 2013 had the highest percentage of airports failing to report any TIP data at nearly 14 percent. For category X and I airports, these results had generally improved by fiscal year 2014 with all of these airports reporting TIP data that year. However, the percentage of category III and IV airports that did not report TIP data generally increased during fiscal years 2013 and 2014 compared to prior years.

TSA officials attributed the missing TIP data to a transition to new X-ray screening equipment at certain airports from fiscal year 2009 through fiscal year 2012. Officials stated that, due to software compatibility issues with the new machines, TIP image capability was turned off for an extended period of time, meaning that TIP testing was not occurring on these machines and, therefore, TIP data were neither collected nor reported for these airports. TSA officials also told us that their older X-ray machines do not have the capability to automatically upload TIP data results to headquarters. As a result, some airports relying on these older X-ray machines were not able to submit TIP data automatically by electronic means and did not submit it manually. TSA officials reported that they do not have a process for determining whether TIP data have been submitted by all airports, on a regular basis, as required. TSA officials told us they are making efforts to install automatic uploading capabilities to all new machines that they expect will help ensure that TIP data reporting is complete and timely. However, TSA has placed these efforts on hold pending security concerns that must first be addressed stemming from the recent cybersecurity breaches at the Office of Personnel Management that have led to TSA reviewing its own cybersecurity efforts before moving forward with installation of automatic uploading capabilities on its X-ray machines.

TSA officials also acknowledged that, in addition to the airports discussed above that did not report any TIP data for a year or more at a time, other airports may have reported only partial TIP results data during this same time frame. TSA officials stated that, in the nationwide results data provided to GAO, it would be difficult to ascertain how much data might be missing from individual airports (during the time period covered by our data) since the number and type of machines in use at those airports at any particular point in time could vary.

Based on our observation of the incomplete TIP data, we recommended in May 2016 that DHS ensure that TSA officials at individual airports submit complete TIP results to the national database as required by TSA policy. In addition, we noted that Standards for Internal Control in the Federal Government states that the information requirements needed to achieve the agency’s objectives should be identified and communicated to management in a timely manner in order that they may carry out their internal control and other responsibilities. Further, we stated that, unless TSA takes steps to ensure that all airports submit complete, nationwide TIP data, TSA lacks assurance that (1) the decisions it makes on the content of the TIP image library are fully informed, and (2) TSOs are receiving remedial training from the TIP program which has been developed to aid their ability to identify prohibited items. In addition, we noted that, by not ensuring the collection of available TIP data as required, the effectiveness of any potential further use of TIP testing results to inform TSO training or testing (as described below) programs would be limited.

DHS concurred with our recommendation on ensuring the completeness of TIP data and is taking steps to address it. Specifically, DHS reported in April 2016 that TSA is working to establish a tracking system that will automatically identify and highlight specific airports that may be missing from the database, which will allow TSA managers to follow up with the FSDs responsible for those airports. TSA expects to pilot an information technology tool that is key to this system by May 2017. In the interim, TSA will reinforce the policy for reporting TIP results in weekly conference calls with field staff.

A Lack of National Analysis to Inform Screening Efforts

Once complete TIP data are available, TSA could use those data to more accurately monitor the effectiveness of its TSO training. TSA headquarters officials stated that they had previously not systematically analyzed TIP results data to determine any national trends for the purposes of informing future training programs or changes to screening processes or procedures. TSA officials reported that they had not used TIP data in this manner due to the agency’s expectation that TIP is a tool primarily for the benefit of local FSDs to use in monitoring the training needs, and determining areas of focus, for their individual TSOs locally. Specifically, we found that, without this complete picture that would be afforded by analysis of nationwide TIP results, TSA could not use the results to fully inform TSO training for screening passenger carry-on baggage for prohibited items that would help ensure continuous improvement in screening operations.

As a result of our examination of TSA’s use of TIP data, we recommended in May 2016 that, after complete TIP data were available, DHS ensure that TSA conduct analysis of national TIP data for trends that could inform training needs and improve future training and TSO performance assessments. We noted that using this trend analysis to inform TSO training and enhance TSO performance would satisfy provisions of the Standards for Internal Control in the Federal Government which state that an agency’s management should perform ongoing monitoring of its internal control system and associated operations, evaluate the results of those monitoring activities, and take corrective actions when warranted to achieve objectives and address risks. Further, we noted that by not including analyses of TIP results data in nationwide efforts to inform either TSO training or other image- based testing outside of TIP, TSA is missing an opportunity to utilize this extensive, nationwide TSO performance data for enhancing screening operations in addition to lacking assurance that remedial training is occurring, as required, at all airports. DHS concurred with this recommendation and is taking steps to address it. Specifically, DHS reported in April 2016 that TSA is (1) examining airports with the best TIP scores to develop best practices that can be shared with other airports, (2) examining airports with low TIP scores to better understand challenges and options for improving TIP performance, (3) planning to analyze data nationwide to determine what training best improves TIP scores, and (4) developing a process to examine which categories of images most often present challenges to the screening workforce which will inform training efforts. TSA also plans to assess TIP training and assessments over a one-year period ending in May 2017 to determine if performance improvements have been realized and what contributed to the improvement. TSA’s plan for analysis is commendable, but until the TIP data is largely complete, any nationwide review will be limited.

ASAP Covert Test Results are Unreliable

As we also reported in May 2016, TSA determined that ASAP pass rate results data were unreliable, which caused them to question the extent to which ASAP tests accurately measure TSO performance. According to TSA officials, they hired a contractor to perform independent ASAP testing at 40 airports in fiscal year 2015 to verify the reliability of the results of testing previously performed by TSA personnel at those airports. TSA found differences in the test results for most of the 40 airports when compared to the contractor’s results. Specifically, TSA officials found that TSOs at these 40 airports performed more poorly in the ASAP tests conducted by the contractor personnel as compared to the prior ASAP testing done by the local TSA personnel—indicating that these prior-year pass rates were likely showing a higher level of TSO performance in screening passengers and baggage for prohibited items than was actually the case. While TSA officials are still in the process of determining root causes for the variances of the testing results between the contractor and TSA personnel at the airports, they acknowledged that initial results from the contractor appeared to confirm their prior concerns that problems existed with maintaining the covert nature of the tests.

These prior concerns had been based on higher detection rates at some airports when compared to other airports on the same tests performed.

In order to address the concerns stemming from the contractor’s test results, TSA initiated the following actions after reviewing results of the contractor’s initial round of testing in fiscal year 2015:

Conducted briefings with FSDs on the contractor’s findings and ongoing ASAP testing which included expectations that the FSDs use the information as input in overseeing their local ASAP testing programs. According to TSA officials, they are engaging in more frequent and improved communication with FSDs and staff responsible for the ASAP testing and are including discussions of potential corrective actions when warranted.

Extended the work of the contractor by 6 months in order to determine if the previously-identified variances in results are continuing.

Engaged in efforts to better identify root causes of ASAP testing failures, including the development of a data collection tool to facilitate these efforts.

Added an ASAP headquarters testing program that will supplement the ASAP testing conducted by TSA field personnel. These headquarters testing teams will perform, on a permanent basis, the quality assurance and validation activities for ASAP that are currently being performed by the contract test teams. However, field personnel will continue to conduct the majority of ASAP testing.

TSA officials stated that, through these measures, they believe they are enhancing the accountability of the local FSDs and their staff for ensuring the quality and reliability of the local ASAP testing programs moving forward. The officials added that partial results during the 6-month extension period of contract testing indicated that the previously-identified variances in contractor and local ASAP testing had been reduced.

TSA Does Not Follow Up On Implementation of Recommendations Stemming from ASAP Testing at Airports

As we reported in May 2016, TSA does not track whether recommendations from their summary reports on ASAP results have been implemented or reasons for not implementing them. These recommendations may include, among other things, additional training for certain points in the screening process and further testing in certain areas. TSA officials stated that the various recommendations in the reports are strictly for the consideration of FSDs in the field and implementation is not mandatory. Such tracking would be consistent with Standards for Internal Control in the Federal Government which requires that internal controls be designed to ensure that ongoing monitoring occurs during the course of normal operations. This tracking would also help ensure that airports nationwide are taking corrective actions to improve TSA performance, which the agency has identified as an area of concern. Moreover, we reported that tracking the implementation of its recommendations, including the extent to which identified corrective actions are improving subsequent TSO performance and test results, will help TSA better determine the extent to which its implemented recommendations are leading to improvements in screening operations and appropriately addressing identified root causes for previous test failures. Further, without the assurance that recommendations for corrective actions based on the root causes identified in ASAP testing will be fully implemented—where appropriate—nationwide, we stated that TSA would be limited in its ability to take full advantage of any findings from the program.

Based on TSA’s lack of a tracking mechanism for the implementation of its ASAP-related recommendations to the field, we recommended in May 2016 that DHS direct TSA to track implementation by airports of these recommendations to ensure that corrective actions identified through ASAP testing are being applied. DHS concurred with this recommendation and is taking steps to address it. Specifically, DHS reported that TSA is taking actions to formalize ASAP reporting including the development of a standard format for corrective action plans that will help TSA track corrective actions and their effectiveness in addressing findings from ASAP tests. TSA expected to complete these actions by May 31, 2016.

Chairman Johnson, Ranking Member Carper and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time.

GAO Contacts and Staff Acknowledgments

For questions about this statement, please contact Jennifer Grover at (202) 512-7141 or groverj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this statement include Chris Ferencik (Assistant Director), Mike Harmond, Michelle Vaughn, Ellen Wolfe, Amanda Miller, Thomas Lombardi, and Dominick Dale. Key contributors for the previous work that this testimony is based on are listed in each product.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.