Aviation Security: TSA Is Taking Steps to Improve Expedited Screening Effectiveness, but Improvements in Screener Oversight Are Needed
Published: 2016-06-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In 2015, TSA screened or oversaw the screening of more than 708 million passengers at more than 450 U.S airports. In carrying out the screening process, TSA is responsible for ensuring the security of civil aviation while also managing the efficient flow of passengers. TSA employs screening personnel, called TSOs, to carry out passenger and baggage screening operations. Each year, TSA tests TSO performance as part of its efforts to monitor the effectiveness of aviation security screening. In 2011, TSA began providing expedited screening procedures to selected passengers, intended to strengthen security and improve the passenger experience by shortening lines and wait times.
This testimony addresses the extent to which TSA (1) has taken steps to improve the security effectiveness of expedited screening and (2) uses TSO performance testing data to enhance TSO performance in screening for prohibited items. This statement is based on reports GAO issued in May 2016 and December 2014, and selected updates. Among other things, GAO analyzed TSA documentation on expedited screening and TSO testing data.
section_title: What GAO Found
paragraphs: The Transportation Security Administration (TSA) has taken steps intended to improve the security effectiveness of expedited passenger screening since GAO reported on it in December 2014. These steps include
Adjusting the TSA Pre✓® Risk Assessment program algorithm used to assign passengers scores and identify low risk passengers;
Limiting the use of Managed Inclusion to airports that employ canine teams to detect explosives; and,
Developing plans to test the security effectiveness of the Managed Inclusion process as an overall system–ensuring that the testing adheres to established design practices.
According to a TSA memorandum dated November 2015, TSA made changes to TSA Pre✓® Risk Assessment program and Managed Inclusion process as a result of the findings and recommendations included in three prior Department of Homeland Security Office of Inspector General audit reports. According to TSA, these changes were necessary to ensure security and resulted in a 20 percent decrease in the number of individuals receiving expedited screening. Previously, in December 2014, GAO found that TSA had not tested the overall effectiveness of the Managed Inclusion process, and recommended that TSA ensure that its planned testing adhere to established evaluation design practices to yield reliable test results. DHS concurred with the recommendation and plans to begin testing the effectiveness of the Managed Inclusion process as a system during fiscal year 2016.
TSA uses data on Transportation Security Officer (TSO) performance obtained from its various testing programs to ensure that individual TSOs are (1) demonstrating through annual proficiency reviews and resulting recertification that they are qualified to continue conducting passenger and checked baggage screening, and (2) demonstrating proficiency during live screening operations in adhering to screening procedures. However, in a report containing sensitive security information completed in May 2016, GAO found that TSA's ability to fully evaluate TSO performance in screening passengers and baggage for prohibited items is constrained by incomplete and unreliable testing data and a lack of data analysis. For example, some airports did not report testing data on TSOs' ability to identify prohibited items over fiscal years 2009 through 2014 as required by TSA policy. TSA officials also stated they do not systematically analyze test results to determine any national trends for informing future TSO training. In addition, TSA determined that pass rate data for one of its covert testing programs that uses role players at airports to assess TSO performance was unreliable. Specifically, testing by an independent contractor indicated that TSA's covert testing data likely overstated TSO performance. TSA is taking action to determine the root cause of the variance in the testing results and is implementing corrective actions. Further, GAO found that TSA does not track the implementation, where appropriate, of recommendations made based on the covert testing results. DHS concurred with GAO's recommendations made in its May 2016 report and is planning actions to address them.
section_title: What GAO Recommends
paragraphs: In its May 2016 report, GAO recommended that TSA ensure that (1) airports submit complete TSO performance data, (2) the data are analyzed nationally, and (3) implementation of covert testing recommendations are tracked. DHS concurred and is taking actions to address the recommendations.

section_title: Letter
paragraphs: I am pleased to be here today to discuss our past work on the  Transportation Security Administration’s (TSA) expedited screening  process and its use of Transportation Security Officer (TSO) performance  data to improve screening operations. TSA, an agency within the  Department of Homeland Security (DHS), is the primary federal agency  responsible for the security of the nation’s aviation system. As part of this  responsibility, TSA screened or oversaw the screening of more than 708  million passengers and more than 1.6 billion carry-on bags at about 450  U.S. airports in 2015. TSA-employed screening personnel (i.e., TSOs)  carry out passenger and checked baggage screening operations to  identify prohibited items that could pose a threat to the aircraft and  passengers. These screening operations may include pat downs, search  of property, and operating metal detectors and explosives detection  equipment, among other things. While TSA’s primary aviation  responsibility is to ensure security, it also strives to balance the safety  and security of the traveling public with the efficient flow of passengers  through the screening process. In an effort to strengthen and improve  these screening operations, TSA began providing expedited screening to  selected passengers through its TSA Pre® program in October 2011.  The TSA Pre® program uses risk-based, intelligence-driven screening  concepts and technology to determine passenger risk prior to travel. The  use of expedited screening procedures is intended to allow TSA to devote  more time and resources at the airport to screening the passengers TSA  determines to be of higher risk or unknown risk while providing expedited  screening to those passengers determined to pose a lower risk. To further  expedite passenger travel for selected passengers not approved through  TSA Pre®, TSA implemented the Managed Inclusion process in 2012.  Managed Inclusion assesses passenger risk in real time at the airport  using randomization procedures, behavior detection officers (BDOs), and  passenger screening canine teams.
Each year, TSA also conducts certification testing of its TSOs, and in an  effort to measure the performance of aviation security screening, both  TSA and the Department of Homeland Security Office of Inspector  General (DHS-OIG) conduct regular covert testing of TSA screening  operations. In response to the failure rates stemming from recent covert  testing conducted by the DHS-OIG, the Secretary of the Department of  Homeland Security (DHS) directed TSA in June 2015 to take a number of  actions to address the vulnerabilities identified in the testing. Specifically,  the Secretary directed TSA to revise its standard operating procedures  (SOP) for screening, brief all Federal Security Directors (FSD) across the  country on the Inspector General’s findings, and to conduct further  training for all screening personnel and supervisors, among other things.  In October 2015, the TSA Administrator testified before Congress on the  steps TSA was taking to respond to the Secretary’s directive, including  delivering further training to every TSO and supervisor across the  country.
My testimony today addresses the extent to which TSA (1) has taken  steps to improve the security effectiveness of expedited screening and (2)  uses TSO performance testing data to enhance TSO performance in  screening for prohibited items. This statement is based on reports we  issued in May 2016 and December 2014, and selected updates.  Specifically, for our past work we analyzed TSA documentation including  expedited screening and Managed Inclusion procedures, memorandums  of agreement, and decision memorandums, TSA’s risk assessment  methodologies, and TSA’s security assessment of the Managed Inclusion  process, among other documents, to gain an understanding of how  expedited screening and Managed Inclusion operate. Moreover, we  reviewed data (ranging from 2009 to 2015) on TSA’s performance  evaluation testing programs, compared the results by airport security  category, and also assessed the reliability of the data. We found that  some testing programs had incomplete or unreliable data for the years we  analyzed and therefore were not sufficiently reliable for describing  national trends. We also reviewed TSA’s processes and actions for using  screener performance testing results to inform its operations and future  TSO training, and assessed these processes against standards in  Standards for Internal Control in the Federal Government. Further, we  interviewed program officials at TSA headquarters and at select airports  about how they analyze performance test data and how, if at all, they use  the results to adjust training or take other actions. Further details on the  scope and methodology for the previously issued reports are available  within each of the published products. We conducted this work in  accordance with generally accepted government auditing standards.  These standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions, based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Expedited Screening
paragraphs: In 2011, TSA began developing new expedited security procedures  intended to strengthen security and improve the passenger experience by  shortening lines and wait times, and in October 2011, implemented its  expedited screening program—known as TSA Pre®. According to TSA,  expedited screening involves a relatively more efficient and convenient  screening process for individuals from whom TSA has obtained sufficient  information to determine them to be lower risk, compared with the  standard screening process for a traveler for whom TSA does not have  such information. For example, passengers eligible for expedited  screening may no longer have to remove their shoes; may leave their  permitted liquids, gels, and laptops in carry-on baggage; and are not  required to divest light outerwear, jackets, or belts when passing through  screening checkpoints unless the screening technology alarms, in which  case these items must be removed.
TSA uses the following methods to assess whether a passenger is low  risk and therefore eligible for expedited screening.
Approved TSA Pre ® lists of known travelers—These lists are comprised of individuals whom TSA has determined to be low risk by virtue of their membership in a specific group, such as active duty military members, or based on group vetting requirements, or if approved through the TSA Pre® Application Program.
Automated TSA Pre ® risk assessments of all passengers— Using these assessments, TSA assigns passengers scores based upon information available to TSA to identify low risk passengers eligible for expedited screening for a specific flight prior to the passengers’ arrival at the airport.
Real-time threat assessments through Managed Inclusion— These assessments use several layers of security, including procedures that randomly select passengers for expedited screening, behavior detection officers who observe passengers to identify high- risk behaviors, and passenger screening canine teams to help ensure that passengers selected for expedited screening have not handled explosive material. TSA developed Managed Inclusion as a tool to improve the efficiency of dedicated TSA Pre ® screening lanes.
When TSA began offering expedited screening at airports in the summer  of 2011, TSOs initially provided such screenings in standard lanes to  passengers aged 12 and younger, and subsequently extended expedited  screening to certain flight crew members and then to passengers aged 75  and older. However, in October 2011, TSA began to expand the concept  of expedited airport screening to more of the flying public by piloting the  TSA Pre ® program. This pilot program allowed certain frequent fliers of  two air carriers to experience expedited screening at four airports. These  frequent fliers became eligible for screening in dedicated expedited  screening lanes, called TSA Pre ® lanes, because they had opted into  the TSA Pre ® program through the air carrier with which they had  attained frequent flier status.
Since October 2011, TSA has further expanded the known traveler  populations eligible for expedited screening. TSA established separate  TSA Pre ® lists for additional low-risk passenger populations, including  members of the U.S. armed forces, Congressional Medal of Honor  Society Members, members of the Homeland Security Advisory Council,  and Members of Congress, among others. In March 2015, TSA officials  stated that the Army, Navy, Marine Corps, Air Force, and Coast Guard  branches of the U.S. armed forces, as well as Reserve and National  Guard personnel, were eligible to participate. TSA also created its own  TSA Pre ® list composed of individuals who apply to be preapproved as  low-risk travelers through the TSA Pre ® Application Program, an  initiative launched in December 2013. To apply, individuals must visit an  enrollment center where they provide biographic information (i.e., name,  date of birth, and address), valid identity and citizenship documentation,  and fingerprints to undergo a TSA Security Threat Assessment.  Applicants must be U.S. citizens, U.S. nationals, or lawful permanent  residents, and cannot have been convicted of certain crimes. As of  December 2015, about 8.8 million individuals were eligible, through TSA  Pre ® lists, for expedited screening. Figure 1 shows the populations for  each TSA Pre ® list.
section_title: Assessing TSO Performance
paragraphs: To carry out passenger and checked baggage screening operations, TSA  employs TSOs at the vast majority of the nation’s commercial airports.  TSOs must complete the New Hire Training Program (NHTP), which  includes at least 40 hours of classroom training focused on their duties as  a screener, a minimum of 60 hours of on-the-job training, and certification  tests for the functions they will be performing. In addition, TSOs are  required to take recurrent training throughout the year to maintain  proficiency with skills learned during the NHTP, and to remain up-to-date  with changes in screening standard operating procedures (SOP), as well  as emerging threats. Also, TSOs who are absent from their screening  duties for a period of time must undergo some level of “return-to-duty”  training based on the amount of time they were absent. Lastly, if TSOs  fail an operational test, they are required to take remedial training  customized to fit the specific screener’s performance improvement  needs. Furthermore, each year, TSA conducts certification testing for its  airport security screeners, and in an effort to measure the performance of  aviation security screening, both TSA and the DHS-OIG conduct regular  covert testing of TSA screening operations.
Recent covert tests conducted by the DHS-OIG highlighted the following  areas of concern: (1) the effectiveness of the passenger screening  process, (2) TSA’s Advanced Imaging Technology (AIT) screening  equipment, (3) related automated target recognition software used by the  AIT systems, and (4) checkpoint screener performance in identifying and  resolving potential security threats at airport checkpoints. In response to  the results of the covert testing, TSA updated its screening SOPs,  retrained TSOs to address the Inspector General’s findings, and provided  additional classroom training nationwide to all TSOs. In addition, TSA  developed new measures of effectiveness that it expects will better  emphasize the agency’s goals for improving security effectiveness by  focusing the measures on both the screening system and workforce in the  areas of readiness and performance.
To measure TSO performance, TSA uses the following performance  measurement tests:
Annual Proficiency Reviews (APR) evaluate TSOs’ ability to identify  prohibited items on an X-ray machine, ability to resolve explosives  detection system machine alarms using the appropriate tools, and  whether TSOs can perform various practical skills such as pat downs,  bag searches, and use of explosive trace detection technology. If a  screener does not pass one of the components of the APR after two;  or, in some cases, three attempts, they are subject to removal from  their position.
Threat Image Projection (TIP) monitors TSOs’ ability to identify  prohibited items in x-ray images of carry-on baggage at the passenger  checkpoint by projecting fictional threat items onto the bags. TIP also  aides in keeping TSOs focused and attentive, and in keeping their  skills sharp in identifying items they do not routinely see. According to  TSA policy, FSDs must monitor TIP results monthly and, if one of their  TSOs identifies less than a target percentage of TIP images  accurately in a month, then the TSO is required to attend remedial  training.
Aviation Screening Assessment Program (ASAP) is a form of  covert testing to measure, at a national level, TSO screening  performance against screening SOPs. TSA’s Office of Security  Operations utilizes local role players to take prohibited items such as  knives, guns, or simulated improvised explosive devices, through the  screening checkpoints to test TSOs performance in accurately  identifying those items. ASAP tests are conducted by TSA at both  screening checkpoints and checked baggage screening areas. The  tests are designed to assess the operational effectiveness of  screeners. TSA implemented a series of improvements to ASAP in  2010 and 2012 that introduced (1) specific testing scenarios to  improve the level of standardization, (2) a formalized debriefing  process, (3) training scenarios by which airports can tailor lessons  learned to their operations, and (4) a strategy for allowing the  reporting of comparable testing results, over time, from the airports.  After these improvements, TSA renamed the program ASAP  Advantage. TSA implements ASAP Advantage according to a 6- month testing schedule, and at the completion of each 6-month cycle,  generates a report identifying trends in screening performance.
section_title: TSA Is Taking Steps to Improve the Security Effectiveness of Expedited Screening
paragraphs: TSA has taken steps to improve the security effectiveness of expedited  screening since we issued our December 2014 report. Specifically, TSA  has begun planning for the testing of the security effectiveness of the  Managed Inclusion process as an overall system–ensuring that the  testing adheres to established design practices. In addition, TSA has  adjusted the TSA Pre ® Risk Assessment program algorithm used to  assign passengers scores and identify low risk passengers because the  DHS-OIG found that the algorithm allowed a high-risk individual access to  expedited screening. Also, according to TSA documentation, TSA  reduced the number of passengers screened by the Managed Inclusion  process by limiting its use to airports that have canine teams to detect  explosives.
Our December 2014 report found that TSA has tested the effectiveness of  the individual Managed Inclusion security layers, but that TSA had not yet  tested the Managed Inclusion process as an overall system. We stated  that our previous work identified challenges in several of the layers used  in the Managed Inclusion process, raising concerns regarding their overall  effectiveness. For example, in November 2013, we found that TSA had  not demonstrated that behavioral indicators can be used to reliably and  effectively identify passengers who may pose a threat to aviation  security. While TSA is taking steps to revise and test the behavior  detection program, such as working to provide scientifically validated  evidence that demonstrates that behavioral indicators can be used to  identify passengers who may pose a risk to aviation security, the issue  remains open. As of May 2016, TSA told us that it is taking actions to  optimize the effectiveness of its behavior detection program and plans to  begin an operational test of these efforts in September 2016. In our  December 2014 report, we noted that TSA has previously faced  challenges designing studies to test the security effectiveness of  programs in accordance with established methodological practices such  as ensuring an adequate sample size or randomly selecting items in a  study to ensure the results can be generalizable—key features of  established evaluation design practices. As a result, we recommended  that TSA take steps to ensure and document that its planned testing of  the Managed Inclusion process as a system adheres to established  evaluation design practices. DHS concurred with our recommendation,  and according to TSA officials, TSA has developed a data collection and  analysis plan to be used for the testing of the Managed Inclusion system.  As of May 2016, TSA is reviewing and finalizing the plan and intends to  test it at ten airports in late summer or early fall 2016 according to TSA  officials. We will continue to monitor TSA’s progress in addressing this  recommendation.
In addition, according to a TSA memorandum dated November 2015,  TSA made changes to the TSA Pre ® Risk Assessment program and  Managed Inclusion process to enhance aviation security as a result of the  findings and recommendations included in three prior DHS-OIG audit  reports. Specifically, TSA made changes to the TSA Pre ® Risk  Assessment program algorithm used to assign passenger scores  because the DHS-OIG found that the program created a potential  aviation security vulnerability in at least one instance by identifying a  convicted felon as low risk and eligible for expedited screening. As a  result, TSA recognized the increased level of uncertainty surrounding a  potential threat posed by individuals who obtain expediting screening  eligibility  through the Risk Assessment program as compared to individuals who  have been vetted and are included on one of the TSA Pre ® lists.  Following the public release of the DHS-OIG’s covert testing results,  TSA officials stated that TSA began a thorough review of checkpoint  operations, and as a part of that review, evaluated all methods in which  individuals without background checks became eligible for expedited  screening. As a result of this evaluation and based on a recommendation  from another DHS-OIG audit, TSA documentation shows that TSA  discontinued the use of Explosives Trace Detection (ETD) devices as a  method used to conduct real time threat assessments and is now limiting  the use of Managed Inclusion to airports that employed canine team to  detect explosives. According to the TSA administrator, these changes  have resulted in a 20 percent decrease in the number of individuals who  receive expedited screening.
To address this decrease in expedited screening and its likely effect on  passenger wait times, TSA plans to undertake efforts to increase the  number of individuals included on the TSA Pre ® lists of known  travelers from the nearly 8.8 million individuals currently enrolled to 25  million individuals. In order to achieve this increase, TSA plans to change  the  enrollment  process,  increase  marketing  and  communication  efforts,  and expand the number of contractors that provide enrollment services.  TSA estimates that the TSA Pre ® lists of known travelers will total 25  million individuals in 3 to 4 years.
section_title: While TSA Uses TSO Screening Performance Data, It Is Constrained by Incomplete and Unreliable Data and a Lack of Data Analysis and Assessment Follow-Up
paragraphs: TSA utilizes data on TSO performance obtained from its various testing  programs to help to ensure that individual TSOs are (1) qualified to  conduct passenger and checked baggage screening based on Annual  Proficiency Reviews and resulting recertifications, and (2) demonstrate  proficiency, during live screening operations, in their adherence to  screening standard operating procedures and other TSA guidance for  detecting prohibited items. However, incomplete and unreliable data and  limited analysis constrains TSA’s ability to determine the true level of TSO  performance in screening passengers and baggage for prohibited items.  Without this knowledge, TSA cannot fully identify and make necessary  improvements to screening operations.
subsections: section_title: TSA Uses TSO Performance Data to Inform TSO Training
paragraphs: TSA has several programs in place to yield data for oversight and  analysis of TSO screening performance. As noted previously, TSA relies  on Annual Proficiency Reviews (APR) to recertify TSOs. TSA’s Office of  Training and Workforce Engagement examined the results of specific  APR component tests administered in 2013 to inform their development  of related courses for the annual training curriculum for TSOs, known as  the National Training Plan (NTP). Specifically, TSA officials stated they  reviewed the results of these component tests—screening of individuals  with disabilities, bag searches, and standard pat downs—and added  training to the fiscal year 2015 NTP to specifically address the  deficiencies they identified.
In addition, during live screening operations, TSA also monitors individual  TSO performance through (1) Threat Image Projection (TIP) testing by  local TSA officials which assesses the TSOs’ proficiency at identifying  prohibited items in X-ray images of passengers’ carry-on baggage, and  (2) Aviation Screening Assessment Program (ASAP) covert tests which  assess the TSOs’ ability to properly adhere to screening standard  operating procedures and prevent the passage of prohibited items  through passenger and baggage checkpoints. TSA monitors the results  of these testing programs to determine whether individual TSOs need  remedial training based on the results.
section_title: Incomplete and Unreliable Data and A Lack of National Analysis Limit TSA’s Ability To Assess TSO Performance
subsections: section_title: Incomplete Data
paragraphs: TSA policy requires airport personnel to manually download TIP testing  results from their individual X-ray machines and upload the monthly data  into TSA’s national database repository for TSA results. According to TSA  headquarters personnel responsible for overseeing the TIP program, they  use these uploaded results to determine if any adjustments are needed to  the quality or usefulness of the library of images maintained in the TIP  system nationwide. However, as we found in May 2016, some airports  had failed to submit TIP data as required. As shown in figure 2, some  airports in all five airport risk categories did not report any TIP results  nationally over the course of a year from fiscal year 2010 through fiscal  year 2013. During the fiscal year 2009 through 2014 time frame, fiscal  year 2013 had the highest percentage of airports failing to report any TIP  data at nearly 14 percent. For category X and I airports, these results had  generally improved by fiscal year 2014 with all of these airports reporting  TIP data that year. However, the percentage of category III and IV  airports that did not report TIP data generally increased during fiscal  years 2013 and 2014 compared to prior years.
TSA officials attributed the missing TIP data to a transition to new X-ray  screening equipment at certain airports from fiscal year 2009 through  fiscal year 2012. Officials stated that, due to software compatibility issues  with the new machines, TIP image capability was turned off for an  extended period of time, meaning that TIP testing was not occurring on  these machines and, therefore, TIP data were neither collected nor  reported for these airports. TSA officials also told us that their older X-ray  machines do not have the capability to automatically upload TIP data  results to headquarters. As a result, some airports relying on these older  X-ray machines were not able to submit TIP data automatically by  electronic means and did not submit it manually. TSA officials reported  that they do not have a process for determining whether TIP data have  been submitted by all airports, on a regular basis, as required. TSA  officials told us they are making efforts to install automatic uploading  capabilities to all new machines that they expect will help ensure that TIP  data reporting is complete and timely. However, TSA has placed these  efforts on hold pending security concerns that must first be addressed  stemming from the recent cybersecurity breaches at the Office of  Personnel Management that have led to TSA reviewing its own  cybersecurity efforts before moving forward with installation of automatic  uploading capabilities on its X-ray machines.
TSA officials also acknowledged that, in addition to the airports discussed  above that did not report any TIP data for a year or more at a time, other  airports may have reported only partial TIP results data during this same  time frame. TSA officials stated that, in the nationwide results data  provided to GAO, it would be difficult to ascertain how much data might  be missing from individual airports (during the time period covered by our  data) since the number and type of machines in use at those airports at  any particular point in time could vary.
Based on our observation of the incomplete TIP data, we recommended  in May 2016 that DHS ensure that TSA officials at individual airports  submit complete TIP results to the national database as required by TSA  policy. In addition, we noted that Standards for Internal Control in the  Federal Government states that the information requirements needed to  achieve the agency’s objectives should be identified and communicated  to management in a timely manner in order that they may carry out their  internal control and other responsibilities. Further, we stated that, unless  TSA takes steps to ensure that all airports submit complete, nationwide  TIP data, TSA lacks assurance that (1) the decisions it makes on the  content of the TIP image library are fully informed, and (2) TSOs are  receiving remedial training from the TIP program which has been  developed to aid their ability to identify prohibited items. In addition, we  noted that, by not ensuring the collection of available TIP data as  required, the effectiveness of any potential further use of TIP testing  results to inform TSO training or testing (as described below) programs  would be limited.
DHS concurred with our recommendation on ensuring the completeness  of TIP data and is taking steps to address it. Specifically, DHS reported in  April 2016 that TSA is working to establish a tracking system that will  automatically identify and highlight specific airports that may be missing  from the database, which will allow TSA managers to follow up with the  FSDs responsible for those airports. TSA expects to pilot an information  technology tool that is key to this system by May 2017. In the interim,  TSA will reinforce the policy for reporting TIP results in weekly conference  calls with field staff.
section_title: A Lack of National Analysis to Inform Screening Efforts
paragraphs: Once complete TIP data are available, TSA could use those data to more  accurately monitor the effectiveness of its TSO training. TSA  headquarters officials stated that they had previously not systematically  analyzed TIP results data to determine any national trends for the  purposes of informing future training programs or changes to screening  processes or procedures. TSA officials reported that they had not used  TIP data in this manner due to the agency’s expectation that TIP is a tool  primarily for the benefit of local FSDs to use in monitoring the training  needs, and determining areas of focus, for their individual TSOs locally.  Specifically, we found that, without this complete picture that would be  afforded by analysis of nationwide TIP results, TSA could not use the  results to fully inform TSO training for screening passenger carry-on  baggage for prohibited items that would help ensure continuous  improvement in screening operations.
As a result of our examination of TSA’s use of TIP data, we  recommended in May 2016 that, after complete TIP data were available,  DHS ensure that TSA conduct analysis of national TIP data for trends that  could inform training needs and improve future training and TSO  performance assessments. We noted that using this trend analysis to  inform TSO training and enhance TSO performance would satisfy  provisions of the Standards for Internal Control in the Federal  Government which state that an agency’s management should perform  ongoing monitoring of its internal control system and associated  operations, evaluate the results of those monitoring activities, and take  corrective actions when warranted to achieve objectives and address  risks. Further, we noted that by not including analyses of TIP results  data in nationwide efforts to inform either TSO training or other image- based testing outside of TIP, TSA is missing an opportunity to utilize this  extensive, nationwide TSO performance data for enhancing screening  operations in addition to lacking assurance that remedial training is  occurring, as required, at all airports. DHS concurred with this  recommendation and is taking steps to address it. Specifically, DHS  reported in April 2016 that TSA is (1) examining airports with the best TIP  scores to develop best practices that can be shared with other airports,  (2) examining airports with low TIP scores to better understand  challenges and options for improving TIP performance, (3) planning to  analyze data nationwide to determine what training best improves TIP  scores, and (4) developing a process to examine which categories of  images most often present challenges to the screening workforce which  will inform training efforts. TSA also plans to assess TIP training and  assessments over a one-year period ending in May 2017 to determine if  performance improvements have been realized and what contributed to  the improvement. TSA’s plan for analysis is commendable, but until the  TIP data is largely complete, any nationwide review will be limited.
section_title: ASAP Covert Test Results are Unreliable
paragraphs: As we also reported in May 2016, TSA determined that ASAP pass rate  results data were unreliable, which caused them to question the extent to  which ASAP tests accurately measure TSO performance. According to  TSA officials, they hired a contractor to perform independent ASAP  testing at 40 airports in fiscal year 2015 to verify the reliability of the  results of testing previously performed by TSA personnel at those  airports. TSA found differences in the test results for most of the 40  airports when compared to the contractor’s results. Specifically, TSA  officials found that TSOs at these 40 airports performed more poorly in  the ASAP tests conducted by the contractor personnel as compared to  the prior ASAP testing done by the local TSA personnel—indicating that  these prior-year pass rates were likely showing a higher level of TSO  performance in screening passengers and baggage for prohibited items  than was actually the case. While TSA officials are still in the process of  determining root causes for the variances of the testing results between  the contractor and TSA personnel at the airports, they acknowledged that  initial results from the contractor appeared to confirm their prior concerns  that problems existed with maintaining the covert nature of the tests.
These prior concerns had been based on higher detection rates at some  airports when compared to other airports on the same tests performed.
In order to address the concerns stemming from the contractor’s test  results, TSA initiated the following actions after reviewing results of the  contractor’s initial round of testing in fiscal year 2015:
Conducted briefings with FSDs on the contractor’s findings and ongoing ASAP testing which included expectations that the FSDs use the information as input in overseeing their local ASAP testing programs. According to TSA officials, they are engaging in more frequent and improved communication with FSDs and staff responsible for the ASAP testing and are including discussions of potential corrective actions when warranted.
Extended the work of the contractor by 6 months in order to determine if the previously-identified variances in results are continuing.
Engaged in efforts to better identify root causes of ASAP testing failures, including the development of a data collection tool to facilitate these efforts.
Added an ASAP headquarters testing program that will supplement the ASAP testing conducted by TSA field personnel. These headquarters testing teams will perform, on a permanent basis, the quality assurance and validation activities for ASAP that are currently being performed by the contract test teams. However, field personnel will continue to conduct the majority of ASAP testing.
TSA officials stated that, through these measures, they believe they are  enhancing the accountability of the local FSDs and their staff for ensuring  the quality and reliability of the local ASAP testing programs moving  forward. The officials added that partial results during the 6-month  extension period of contract testing indicated that the previously-identified  variances in contractor and local ASAP testing had been reduced.
section_title: TSA Does Not Follow Up On Implementation of Recommendations Stemming from ASAP Testing at Airports
paragraphs: As we reported in May 2016, TSA does not track whether  recommendations from their summary reports on ASAP results have  been implemented or reasons for not implementing them. These  recommendations may include, among other things, additional training for  certain points in the screening process and further testing in certain  areas. TSA officials stated that the various recommendations in the  reports are strictly for the consideration of FSDs in the field and  implementation is not mandatory. Such tracking would be consistent with  Standards for Internal Control in the Federal Government which requires  that internal controls be designed to ensure that ongoing monitoring  occurs during the course of normal operations. This tracking would also  help ensure that airports nationwide are taking corrective actions to  improve TSA performance, which the agency has identified as an area of  concern. Moreover, we reported that tracking the implementation of its  recommendations, including the extent to which identified corrective  actions are improving subsequent TSO performance and test results, will  help TSA better determine the extent to which its implemented  recommendations are leading to improvements in screening operations  and appropriately addressing identified root causes for previous test  failures. Further, without the assurance that recommendations for  corrective actions based on the root causes identified in ASAP testing will  be fully implemented—where appropriate—nationwide, we stated that  TSA would be limited in its ability to take full advantage of any findings  from the program.
Based on TSA’s lack of a tracking mechanism for the implementation of  its ASAP-related recommendations to the field, we recommended in May  2016 that DHS direct TSA to track implementation by airports of these  recommendations to ensure that corrective actions identified through  ASAP testing are being applied. DHS concurred with this  recommendation and is taking steps to address it. Specifically, DHS  reported that TSA is taking actions to formalize ASAP reporting including  the development of a standard format for corrective action plans that will  help TSA track corrective actions and their effectiveness in addressing  findings from ASAP tests. TSA expected to complete these actions by  May 31, 2016.
Chairman Johnson, Ranking Member Carper and Members of the  Committee, this completes my prepared statement. I would be pleased to  respond to any questions that you may have at this time.
section_title: GAO Contacts and Staff Acknowledgments
paragraphs: For questions about this statement, please contact Jennifer Grover at  (202) 512-7141 or groverj@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this statement. Individuals making key contributions to this statement  include Chris Ferencik (Assistant Director), Mike Harmond, Michelle  Vaughn, Ellen Wolfe, Amanda Miller, Thomas Lombardi, and Dominick  Dale. Key contributors for the previous work that this testimony is based  on are listed in each product.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.