Aviation Security: TSA Should Ensure Testing Data Are Complete and Fully Used to Improve Screener Training and Operations
Published: 2016-09-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: TSA trains TSOs to screen passengers and baggage for items that could pose a threat at nearly 440 airports across the country. One way TSA and the Department of Homeland Security (DHS) Office of Inspector General (OIG) measure TSO performance is through covert testing of TSA screening operations. In response to the findings from recent DHS OIG covert testing, the Secretary of DHS directed TSA in June 2015 to conduct further training for all TSOs and supervisors. GAO was asked to review TSA's efforts to train and test TSOs. This report examines (1) how TSA trains TSOs and evaluates the training; (2) how TSA measures TSO performance and what the data show; and (3) to what extent TSA uses TSO performance data to enhance TSO performance. GAO analyzed TSO performance data from 2009 through 2015, reviewed documents regarding TSA training and testing, and interviewed TSA officials at headquarters and 10 airports. GAO selected these airports based on airport risk categories, among other things. Information from these airports was not generalizable, but provided insights into TSO training and testing. This is a public version of a sensitive report that GAO issued in May 2016.
section_title: What GAO Found
paragraphs: The Transportation Security Administration (TSA) uses a variety of programs to train and evaluate Transportation Security Officers (TSO) who are responsible for screening passengers and baggage for threats to aviation security. For example, by law, TSOs must complete 40 hours of classroom training, 60 hours of on-the-job training, and certification tests before performing screening. Once certified, TSA requires TSOs to complete annual training under the National Training Plan. Since 2013, TSA has been phasing in a program to evaluate its training to inform use of training resources. TSA expects that this evaluation program should help the agency determine how well training meets TSOs' needs, provides them with needed knowledge and skill, and has an impact on their performance.
TSA measures TSO performance in various ways, including (1) annual proficiency reviews, which certify TSOs by evaluating their ability to carry out screening standard operating procedures; (2) assessments of X-ray machine operators' ability to identify prohibited items by displaying fictional threat items, such as guns or explosives, onto X-ray images of actual baggage; and (3) covert testing programs that use role players to take prohibited items through screening checkpoints to test TSOs or determine how TSOs interact with the public, among other things. Over the time periods GAO reviewed, TSA data on the results of annual proficiency reviews and covert testing on how TSOs interact with the public show that TSOs' scores (pass rates) varied by airport security risk category. GAO is not providing TSOs' scores for annual proficiency reviews, X-ray machine operator assessments, or covert testing for prohibited items at checkpoints in this report due to the sensitive or classified nature of the data or the data reliability concerns discussed below.
TSA has made use of annual proficiency review data to enhance TSO training, but its use of other testing data is constrained by incomplete and unreliable data. Specifically, due to software compatibility issues and a lack of automatic uploading capability, airport reporting on assessments of X-ray machine operators was not complete, as required by TSA policy, for each year of data GAO examined (fiscal years 2009 through 2014), limiting their reliability and use to enhance TSO training. In addition, for the data it does collect on these assessments, TSA has not taken steps to analyze these data nationwide, which could help the agency identify potential trends or opportunities to improve TSO performance. Furthermore, in 2015, TSA determined that prior year results of one of its two covert testing programs to assess TSOs' ability to identify prohibited items at checkpoints were unreliable, resulting in pass rates that were likely higher than actual TSO performance. TSA has since taken steps to enhance reliability by hiring a contractor to perform independent validation testing, among other things. Finally, TSA does not require or track implementation by field personnel of national recommendations related to these covert tests, thereby limiting the agency's ability to take advantage of the corrective actions identified from the tests.
section_title: What GAO Recommends
paragraphs: GAO recommends that TSA (1) collect complete data on assessments of X-ray machine operators, (2) analyze these data nationally for opportunities to enhance TSO performance, and (3) track the implementation of covert testing recommendations. TSA concurred with the recommendations.

section_title: Letter
paragraphs: The screening of airport passengers and their checked baggage is a  critical component in securing our nation’s commercial aviation system.  Since the terrorist attacks of September 11, 2001, the Transportation  Security Administration (TSA) has been tasked with screening airline  passengers and their accessible and checked baggage for prohibited and  other potentially dangerous items that could pose a threat to the aircraft  and passengers. In fiscal year 2014 alone, approximately 660 million  passengers and 2 billion bags were screened at nearly 440 TSA- regulated airports across the nation. Each year, TSA conducts  certification testing for its airport security screeners, and in an effort to  measure the performance of aviation security screening, both TSA and  the Department of Homeland Security Office of Inspector General (DHS- OIG) conduct regular covert testing of TSA screening operations. In  response to the failure rates stemming from recent covert testing  conducted by the DHS-OIG, the Secretary of the Department of  Homeland Security (DHS) directed TSA in June 2015 to take a number of  actions to address the vulnerabilities identified in the testing. Specifically,  the Secretary directed TSA to revise its standard operating procedures  (SOP) for screening, brief all Federal Security Directors (FSD) across the  country on the Inspector General’s findings, and to conduct further  training for all Transportation Security Officers (TSO) and supervisors,  among other things. In October 2015, the TSA Administrator testified  before Congress on the steps TSA was taking to respond to the  Secretary’s directive, including delivering further training to every TSO  and supervisor across the country.
In 2005, we reviewed actions TSA had taken to enhance training for  TSOs, how TSA ensured all required TSO training was completed, and  what actions TSA had taken to measure and enhance TSO performance.  We found that TSA lacked adequate internal controls to ensure screeners  received legally mandated remedial training and to monitor its recurrent  training program. Specifically, we found that TSA policy did not specify  the responsible party for ensuring screeners completed training and that  TSA could not document that screeners received training. We made  recommendations to TSA to close these gaps, which TSA concurred with,  and has since taken steps to address. More recently, you asked us to  review issues related to TSA’s training and testing of airport security  screeners. This report addresses the following objectives: (1) How does  TSA train TSOs and to what extent does TSA evaluate the training; (2)  How does TSA measure the performance of TSOs and what do the  performance data show; and (3) To what extent does TSA use TSO  performance data to enhance TSO performance.
This report is a public version of a prior sensitive report that we provided  to you. TSA deemed some of the information in the prior report Sensitive  Security Information, which must be protected from public disclosure.  Therefore, this report omits sensitive information regarding specific details  and results of TSA training and testing programs for its TSOs in addition  to the names of airports visited during our review. The information  provided in this report is more limited in scope, in that it excludes such  sensitive information, but it addresses the same questions as the  sensitive report and the methodology used for both reports is the same.
To address the first objective on how TSA trains TSOs and to what extent  TSA evaluates the training, we reviewed relevant TSA policies and  procedures for training, including management directives and the National  Training Plan (NTP), which prescribes the annual training curriculum for  TSOs. We also reviewed documentation on training requirements,  including those contained in the Aviation and Transportation Security Act  (ATSA), as well as documents on TSA’s training development and  completion. We interviewed TSA headquarters officials from the Office of  Training and Development (OTD), the Office of Human Capital (OHC),  and the Office of Security Operations (OSO), who are responsible for  developing and monitoring TSO training, and staff from a total of 10  airports—including Federal Security Directors (FSD), transportation  security managers, instructors, training managers, TSOs, and others to  determine how training is carried out in the field. Specifically, we  conducted site visits to 6 airports of different sizes, including 3 airports in  category X, and one airport each in categories I, II, and III. Further, we  conducted phone interviews with officials at 1 airport each in categories I,  II, III, and IV to obtain additional perspectives on how airport officials carry  out training requirements locally—particularly at airports with smaller  numbers of flights and passenger boardings. We selected the airports to  visit in person based on factors such as airport risk category, geographic  proximity to one another, and our analysis of the airports’ TSO  performance on annual screening certification tests from 2009 through  2014. We selected at least one airport from the high, low, and middle of  the performance distribution. Our visits to airports provide insights about  TSA training, but observations from these airports are not generalizable  to all airports across the country.
To assess the extent to which TSA evaluates TSO training, we reviewed  TSA documents used for evaluating training courses, including end-of- course surveys administered to participants. Further, we reviewed draft  documents on TSA’s training evaluation plan, which TSA is currently  developing, including a draft management directive and draft SOPs for  evaluating training courses. We compared the training evaluation  documentation to the Kirkpatrick model for training evaluation, which TSA  uses as guidance for its evaluations of TSO training. We also  interviewed TSA headquarters officials responsible for evaluating TSO  training and for developing and implementing the TSA training evaluation  plan. Further, we interviewed management officials at each of the 10  airports to further understand how, if at all, training at individual airports is  evaluated locally.
To address the second objective on how TSA measures the performance  of TSOs and what the performance data show, we analyzed data from the  following performance evaluation programs:
Annual Proficiency Review (APR), which is an annual certification test  TSOs must pass to remain employed as a screener. We analyzed  APR pass rates from calendar year 2009 (the first year for which data  were available) through 2015 (the last year for which data were  available at the time of our review).
Threat Image Projection (TIP) system data from fiscal Year 2009  through 2014, the last year the data were available at the time of our  data request. The TIP system helps TSA determine whether operators  correctly identify threat items that are electronically superimposed on  the X-ray monitor during the screening of passenger property at the  checkpoint.
Presence, Advisement, Communication, and Execution (PACE)  testing program, which TSA uses to measure whether TSOs are  adhering to standard operating procedures while screening at the  passenger checkpoint. We analyzed PACE data from calendar  2011, the year the program was started, through 2014, the last year  for which the data were available.
Aviation Screening Assessment Program (ASAP), a covert testing  program designed to assess the operational effectiveness of  screeners by evaluating screeners’ ability to properly follow TSA’s  standard operating procedures for screening and keep prohibited  items from being taken through the checkpoint. We analyzed ASAP  data from calendar year 2013 through 2015 because TSA made  adjustments to the ASAP testing program in 2013, and therefore the  pre-2013 testing data are not comparable to the 2013 through 2015  data. Results of ASAP testing are classified at the secret level and are  not included in this report.
We assessed the reliability of the APR, TIP, PACE, and ASAP data by (1)  interviewing agency officials regarding data collection practices and (2)  testing the data for missing data and duplicates, among other things. We  found the APR and PACE data sufficiently reliable to present average  pass rates for TSOs during calendar years 2009 through 2015. We found  the TIP data to be incomplete, and thus unreliable for describing national  trends, because TSA could not provide TIP scores for every airport for the  period in which we conducted our analysis. Therefore, we do not present  TIP data in this report. See appendix I for additional details.
To analyze what the TSO performance data show, we examined trends in  APR, PACE, and ASAP results across time, and analyzed the results by  airport categories (X, I, II, III, IV) to identify any trends among airports in  different risk categories. See appendix I for further details on our data  analysis.
To address the third objective on the extent to which TSA uses TSO  performance data to enhance screening performance, we reviewed TSA’s  processes and actions for using screener testing results to inform its  operations and training, and assessed these processes against standards  in Standards for Internal Control in the Federal Government. Further, we  interviewed program officials at TSA headquarters and at each of the  airports we visited about how they analyze performance data such as  APR, TIP, ASAP, and PACE data, and how, if at all, they use the results  to adjust training or take any other actions.
We conducted this performance audit from February 2015 to September  2016, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: After the terrorist attacks of September 11, 2001, the President signed the  Aviation and Transportation Security Act (ATSA) into law on November  19, 2001, with the primary goal of strengthening the security of the  nation’s civil aviation system. ATSA created TSA as the agency with  responsibility for securing all modes of transportation, including civil  aviation. As part of this responsibility, TSA performs or oversees the  performance of security operations at the nation’s nearly 440 commercial  (i.e., TSA-regulated) airports, including passenger and checked baggage  screening operations. FSDs are TSA officials responsible for overseeing  TSA security activities, including passenger and checked baggage  screening, at one or more commercial airports. TSA classifies  commercial airports in the United States into one of five security risk  categories (X, I, II, III, and IV) based on various factors, such as the total  number of takeoffs and landings annually, and other special security  considerations. In general, category X airports have the largest number of  passenger boardings and category IV airports have the smallest. TSA  periodically reviews airports in each category and, if appropriate, updates  airport categorizations to reflect current operations. Figure 1 shows the  number of commercial airports by airport security category as of July  2015.
subsections: section_title: Passenger Screening
paragraphs: TSA uses a multilayered security strategy aimed to enhance aviation  security. Within those layers of security, TSA’s airport passenger  checkpoint screening system includes, among other things, (1) screening  personnel (i.e., TSOs); (2) SOPs that guide screening processes  conducted by TSOs; and (3) technology, such as advanced imaging  technology systems (often referred to as body scanners) or walk-through  metal detectors, used to conduct screening of passengers. To carry out  passenger and checked baggage screening operations, TSA employs  TSOs at the vast majority of the nation’s commercial airports. There are  several levels of screening officers deployed at the passenger checkpoint:
Transportation Security Officer (TSO): Performs the majority of  security functions to screen people and property to mitigate threats.  Screening may include pat downs, search of property, and operating  technology including walk-through metal detectors, X-ray machines,  and explosives detection equipment, among other things.
Lead Transportation Security Officer (LTSO): Leads a staff of TSOs,  including distributing and adjusting workload and tasks among  employees and oversees the security screening team on a daily basis.  Implements security procedures and provides coaching and guidance  to TSOs in performing screening duties, among other things. LTSOs  also perform screening functions along with added responsibilities,  such as resolving alarms and supervising screening locations when a  supervisor is not available.
Supervisory Transportation Security Officer (STSO): Oversees  screening checkpoints and/or baggage screening, supervises LTSOs  and TSOs in performance of security screening and ensures all  required screening is performed in accordance with SOPs. Reviews  and evaluates work and performance of LTSOs and TSOs, approves  leave, and recommends corrective or disciplinary actions, among  other things. STSOs also perform screening functions and resolve  passenger alarms.
Transportation Security Manager (TSM): Coordinates and facilitates  TSA security activities and manage one or more programs as  assigned by the Federal Security Director. A TSM assigned to  oversee screening checkpoints manages security activities, including  recognizing and correcting improper use or application of equipment  or screening procedures, monitors screening operations, and  implements changes to enhance security and efficiency at screening  locations.
TSOs inspect individuals and property as part of the passenger screening  process to deter and prevent the carriage of any unauthorized explosive,  incendiary, weapon, or other prohibited items on board an aircraft or into  the airport sterile area—in general, an area of an airport that provides  passengers access to boarding aircraft and to which access is controlled  through the screening of persons and property. Ordinarily, screening of  accessible property at the screening checkpoint begins when an  individual places accessible property on the x-ray conveyor belt or hands  accessible property to TSA personnel. As shown in figure 2, TSOs then  review images of the property running through the X-ray machine and  look for signs of prohibited items. The passengers themselves are  typically screened via a walk-through metal detector or an advanced  imaging technology machine, and passengers generally have the option  to request screening by a pat down if they do not wish to be screened by  these technologies. Passengers will also be subject to a pat down if they  are screened by the walk-through metal detector or advanced imaging  technology system and the equipment alarms (in order to resolve the  alarm).
section_title: Checked Baggage Screening
paragraphs: TSOs also inspect checked baggage to deter, detect, and prevent the  carriage of any unauthorized explosive, incendiary, or weapon onboard  an aircraft. Figure 3 shows the general process used to screen checked  bags. Checked baggage screening is accomplished through the use of  explosives detection systems or explosives trace detection systems, and  through the use of alternative means, such as manual searches and  canine teams when the explosives detection systems are unavailable.
section_title: Screener Training Requirements
paragraphs: In accordance with ATSA, screeners must complete a minimum of 40  hours of classroom instruction, 60 hours of on-the-job training, and  successfully complete an on-the-job training examination before they are  certified as security screeners. Screeners can be certified to conduct  passenger screening or checked baggage screening, or they may be  certified as dual function and can then conduct passenger and checked  baggage screening. ATSA also requires that TSA provide operational  testing of screening personnel, and any individual who fails an operational  test must successfully complete remedial training on that specified  security function before returning to duty. In addition, screeners must  also undergo an annual proficiency review to ensure they continue to  meet all qualifications and standards required to perform a screening  function. TSA also requires remedial training for TSOs who fail an  annual proficiency review.
Covert tests recently conducted by the DHS-OIG highlighted areas of  concern for TSA regarding the effectiveness of the passenger screening  process. Specifically, the DHS-OIG conducted covert testing to  determine the effectiveness of TSA’s Advanced Imaging Technology  screening equipment, its related automated target recognition software,  and checkpoint screener performance in identifying and resolving  potential security threats at airport checkpoints. TSA has responded to  the DHS Secretary’s direction regarding the results of the DHS-OIG  covert testing, in part, by updating its screening SOPs and retraining  TSOs to address the Inspector General’s findings. Also in response to  the DHS-OIG findings, TSA has developed new measures of  effectiveness that it expects will better emphasize the agency’s goals for  improving security effectiveness by focusing the measures on both the  screening system and workforce in the areas of readiness and  performance. For example, improved workforce measures, now being  reported monthly, include those to track TSOs’ progress against training  requirements, absences due to injuries or other reasons, and whether  they are meeting performance thresholds on various tests of performance  and job proficiency.
section_title: TSA Uses a Variety of Programs to Train TSOs and Is Developing a Plan to Expand Evaluations of TSO Training
subsections: section_title: TSA Uses New Hire, Recurrent, Remedial, and Return-to-Duty Training Programs for TSOs
paragraphs: TSO training is comprised of a compendium of courses that includes  basic training for initial hires, recurrent training, remedial training, and  return-to-duty training. For example, all new hires receive a combination  of classroom, hands-on, and web-based training. After TSOs finish their  initial new hire training, they receive recurrent and specialized training  courses throughout the year that are provided either via classroom  instruction or through the TSA Online Learning Center. Recurrent training  typically focuses on core screening skills and policies such as X-ray  image interpretation, detection techniques, and screening SOPs. TSOs  receive remedial training when they have failed an operational or  certification test, or if a supervisor identifies a need for further training,  among other things. Further, according to TSA, TSOs who are absent  from their screening duties for a period of time must undergo some level  of “return-to-duty” training based on the amount of time they were absent.  For example, TSOs certified in a screening function but who have not  performed that function for a period of 15 consecutive days or more are  required to complete a return-to-duty training program before being  allowed to perform that function independently. Table 1 describes the  various types of training TSOs receive.
The Office of Training and Development (OTD), within TSA headquarters,  oversees the development, delivery, and evaluation of training programs  for TSA employees. The National Training Plan (NTP), developed jointly  by OTD and the Office of Security Operations, contains the core  curriculum for TSOs to meet their annual training, including the classes  and hours required for TSOs to complete for the year. TSA headquarters  officials implement the NTP to provide ongoing training throughout the  year aimed at continually improving screeners’ knowledge, skills, and  abilities. However, the responsibility for managing the individual training  of TSOs is largely decentralized and it primarily falls on Security Training  Instructors at individual airports to train TSOs on parts of the NTP by  certain dates throughout the year. Managers in the field track the  percentage of the NTP curriculum that TSOs have completed on a  monthly basis using the Online Learning Center database. In addition,  TSA officials at all 10 airports we contacted stated that they monitor  various testing results for their TSOs and observe screening operations at  their airports’ checkpoints, to determine any local, specialized training  needs their screening force may need—over and above that included in  the NTP issued by TSA headquarters. TSA headquarters can also add  training requirements throughout the year as needed, such as the  recently completed “Mission Essentials—Threat Mitigation” training  discussed later.
TSA officials we spoke with at airports noted challenges associated with  completing not only the required training under the NTP, but also training  associated with frequent changes to the screening SOPs for how  particular screening practices are to be conducted at the checkpoint. For  example, TSA personnel at 8 of the 10 airports stated that it was  sometimes difficult to meet the training requirements in the NTP because  they did not have the TSO personnel to both staff the checkpoints and get  all the required training accomplished. Specifically, TSA officials at larger  airports with more passenger throughput, such as category X and  category I airports, reported having ongoing challenges balancing training  with the operational needs at the checkpoint. In contrast, TSA officials at  smaller airports, such as category III and IV airports did not report having  this challenge frequently. TSA officials at all three of the category X  airports stated they addressed the challenge of meeting training  requirements by scheduling large amounts of training during slower travel  seasons for their airport so they would not have to spend time training  TSOs during peak travel periods. Starting in fiscal year 2014, TSA  headquarters began sending the majority of the NTP training  requirements for the entire fiscal year out to the field at the beginning of  the year—rather than at quarterly intervals throughout the year—allowing  airports the flexibility to train TSOs at different rates depending on the  operational needs of the airport.
TSA training officials at 6 of the 10 airports stated that it is challenging for  them to keep the TSOs trained on the frequent changes to screening  SOPs. For example, TSA officials from two airports stated that during the  delivery of a recent NHTP class, screening SOPs were updated to require  an officer to use a handheld metal detector to resolve an alarm arising  from a passenger going through an advanced imaging technology  scanner. Due to the change happening while the new officers were in the  middle of their introductory training, the steps for using the handheld  metal detector were not integrated into the NHTP curriculum. As a  consequence, after the NHTP course was completed, TSA instructors  separately trained the new hires on how to conduct this type of alarm  resolution. In addition, TSA officials at 9 airports we spoke with stated that  the TSOs used “read and sign” binders to train on some SOP changes,  where the officers sign a document stating they read the change to the  screening SOPs. However, officers reported that this type of training did  not ensure they understood how to implement the change at the  screening checkpoint. According to TSA headquarters officials, they plan  to conduct more hands-on training to teach screening SOP changes  moving forward. Further, TSA personnel at 7 of the 10 airports added that  many of the screening SOPs can have room for interpretation, which also  prompted officials at 2 of these airports to create new airport-level training  to address whether to let particular items through the checkpoint such as  bowling balls and other heavy, blunt objects.
section_title: TSA Recently Implemented a Retraining Program for its TSOs in Response to Findings of the DHS OIG
paragraphs: TSA implemented a TSO re-training program in fiscal year 2015 to retrain  its screening workforce in response to findings of the DHS-OIG, which  conducted its own covert testing of TSA’s checkpoint operations and  technology in the spring of 2015. Specifically, in response to the DHS- OIG findings, TSA provided additional training nationwide to all TSOs— referred to as “Mission Essentials—Threat Mitigation” training. According  to TSA documentation, the purpose of this 8-hour classroom training was  to provide the opportunity for the workforce to become familiar with the  intelligence and threat information that underlies TSA’s use of checkpoint  technologies, operational procedures, and the TSO workforce to mitigate  threats. TSA officials described the training as covering the “why”  behind the equipment and procedures TSA uses to screen passengers  and baggage. For example, the training included:  instruction on how social engineering techniques may be used in an  attempt to defeat TSA risk mitigation procedures, updates on SOP changes for screening certain types of passengers, demonstrations on Improvised Explosive Devices (IED) and how pat  downs are used to mitigate the threat, and an overview of checkpoint equipment capabilities and limitations and  the role of using screening SOPs and best practices to mitigate gaps  caused by equipment limitations.
In addition to the 8-hour course provided for screening officers,  supervisors were provided additional training on their responsibilities for  ensuring the correct implementation of the checkpoint SOPs and how to  provide on-the-spot corrections and constructive feedback to officers.  TSA officials added that, in order to ensure enhanced mission focus, the  agency will begin sending all new-hire TSOs to the TSA Academy at the  Federal Law Enforcement Training Center in Glynco, Georgia rather than  conducting the classroom portion of the NHTP at individual airports. The  officials stated this would help standardize the new hire training and  provide a sense to the new hires that they are part of something larger  than just their local airport. TSA officials stated the first new-hire classes  started at the TSA Academy in January 2016.
section_title: TSA Evaluates Select Training Courses Using Elements of the Kirkpatrick Model
paragraphs: To evaluate its training of TSOs, TSA generally follows the Kirkpatrick  model, which is a commonly accepted training evaluation model endorsed  by the Office of Personnel Management (OPM) and used throughout the  federal government. Currently, using the model, TSA implements  training evaluation surveys and conducts analysis of the responses for a  select number of training courses. TSA’s goal for conducting Kirkpatrick- style training evaluation is to answer questions such as how well a  training course met a learner’s needs; what knowledge and skill a course  imparted to learners; what impact the training had on learner  performance; and what the benefits of the training were.
The Kirkpatrick model consists of a four-level approach for soliciting  feedback from training course participants and evaluating the impact the  training had on individual development, among other things. Table 2  provides a description of what each level within the Kirkpatrick model is to  accomplish and TSA’s progress in implementing the levels.
section_title: TSA Is Developing a Program to Expand Training Evaluations and Support Management Decisions on Training Resources
paragraphs: According to TSA officials, the agency is developing a training evaluation  program that will allow it to standardize and expand training evaluation  efforts. In 2013, TSA assessed its training evaluation practices and found  that existing training evaluation efforts did not meet TSA’s needs because  they lacked a formal, comprehensive approach to training evaluation. As  a result, TSA identified the need to establish a formal training evaluation  program, based on the Kirkpatrick model, to standardize its policy,  processes, and procedures for evaluating training and has been working  to establish the program since December of 2013. TSA’s Standards and  Integration Office, within the Office of Training and Development, has  developed a plan for implementing the new evaluation program, which is  intended to support agency leadership in making decisions on how to use  training resources. In addition, TSA expects to approve a Management  Directive and Standard Operating Procedures for the training evaluation  program by May 2016 to define the roles and responsibilities for TSA  offices running the training evaluation program as well as lay out the  steps for analyzing and reporting data collected from the training  evaluations. TSA officials stated the training evaluation plan will be  subject to annual revision, and OTD will continue to update and review  the plan. Standards and Integration Office officials are responsible for  developing the training evaluations and collecting the evaluation data  while TSA’s Training Operations Division will administer the training  evaluations.
TSA’s training evaluation plan describes the types of stakeholders  involved in training evaluation, the communications strategy for sharing  information on training across the agency, and the reporting requirements  for training evaluation. For example, the plan identifies, in broad terms,  which Kirkpatrick Level evaluation reports the Standards and Integration  Office Evaluations Team will generate, who will receive the reports, and  how they will be used. In one example, the reporting plan shows that  Levels 1, 2, and 3 reports should be sent to program managers to help  them allocate screening resources and modify training. This program  strategy, if followed, should position TSA to make data-based strategic  decisions on the effectiveness of training courses once the training  evaluation plan is fully implemented. For example, TSA plans to use  training evaluation data to conduct curriculum reviews to improve training  courses and programs.
TSA is planning to implement its new training evaluation program in four  phases. During the first phase, TSA plans to implement Level 1 and Level  3 training evaluations for their TSO Basic Training Program and for core  operational courses, and to collect and analyze the data from these  evaluations. In phase two TSA plans to expand Level 1 and 3 training  evaluations to key courses in their National Training Plan. Phase two is  scheduled to begin in late 2016. Once TSA has implemented these  training evaluations for TSO Basic Training and for courses in the NTP,  TSA plans to add selected Online Learning Center courses to their  training evaluation program which constitutes phase three. Finally, in  phase four, TSA plans to evaluate whether they need new training  courses, and if so, all newly approved training courses would be required  to develop an evaluation plan.
section_title: TSA Uses a Variety of Methods to Measure TSO Performance, and Results Vary by Type of Task Tested and Airport Risk Category TSA Measures TSO Performance through Tests Conducted in a Non- Operational Setting and at Active Checkpoints
paragraphs: TSA uses a variety of methods to measure the performance of its TSOs,  including the Annual Proficiency Review (APR)—an annual certification  program to evaluate TSOs’ skill in performing the various screening  functions. Portions of the APR are computer-based X-ray image tests  done in a non-operational setting away from the active checkpoints while  the remaining tests are skills demonstrations performed in a realistic, but  inactive, screening environment such as an unused screening lane.  Which components of the APR an individual TSO must take are  dependent on whether that TSO is certified to perform passenger  screening, baggage screening, or has dual certification to perform both  functions.
TSA has other testing programs that take place during active operations  at the checkpoints to assess TSOs’ level of adherence to screening SOPs  and associated management directives. These include the Threat Image  Projection (TIP) image testing; the Aviation Screening Assessment  Program (ASAP); and Presence, Advisement, Communication, and  Execution (PACE) covert testing. Table 3 provides a summary of TSO  performance measurement tests.
In addition to the ASAP covert testing and other tests detailed in Table 3  for assessing the effectiveness of TSOs in carrying out screening  functions, the TSA Office of Inspection (OOI) Special Operations Division  (SOD) regularly conducts independent covert “red team” testing to  measure the effectiveness of TSA security systems and identify  vulnerabilities in transportation security as a whole. TSA develops and  deploys red team tests based upon current intelligence of threats against  transportation systems. In addition to assessing TSOs’ ability to detect  threat items similar to ASAP testing, OOI’s red team covert testing also  assesses the effectiveness of other aspects of the screening operation— including screening procedures followed by the TSOs and the technology  they use at the checkpoints.
TSA policy requires FSDs to provide remedial training to TSOs who either  fail components of the APR (before being allowed to retake those  portions) or do not maintain a minimum score on TIP image tests.  Similarly, TSOs who fail ASAP or red team covert tests—that is,  operational tests—must take, in accordance with ATSA, remedial training  before returning to their screening duties. TSA policy has not specifically  required remedial training for any TSOs who failed PACE tests. Instead,  each airport’s FSD was expected to make their own determination  regarding any necessary retraining based on the PACE testing results.
subsections: section_title: Results of APR and PACE Screener Performance Tests Varied by Specific Task Tested and Airport Risk Category
paragraphs: TSA data on the results of APR and PACE testing show that TSOs’ pass  rates on both of these tests varied by airport risk category over the time  periods we reviewed. Specifically, from calendar year 2009 through 2015,  the percentage of TSOs that passed their APR certification tests on the  first attempt remained relatively constant, with a dip occurring in calendar  year 2010 followed by an increase by a similar percentage in 2015.  According to TSA officials, this performance dip occurred because TSA  ended the practice of using an outside contractor to evaluate TSOs during  the APR tests. TSA officials explained that the TSA personnel who took  over the evaluation function displayed less flexibility than was previously  allowed in scoring of the various APR component tests in that first year  after the transition (2010).
According to TSA officials, the aforementioned changes to the APR  testing program for 2015 (including practice runs prior to grading the  practical skills evaluation portions of the test and dividing the testing by  quarters) have led to improvement in the overall APR pass rates for 2015  compared to prior years. TSA officials explained that they decided to re- examine how they conducted APR testing and implemented the resulting  changes in response to feedback from TSOs that certain aspects of the  testing created unnecessary anxiety which affected their performance.
As described earlier, APR consists of several component tests that  evaluate specific TSO functions. As shown in table 4, these component  tests include X-ray image testing and passenger pat downs, which cover  actions taken by TSOs in routine screening operations at the passenger  and baggage screening checkpoints.
In addition to the overall APR pass rates varying by airport security  category, the results of these individual component tests also varied by  the type of test administered during the 2009 to 2015 timeframe.  Scores  for specific APR components tests are Sensitive Security Information and  not included in this report. In addition, due to issues with both the  reliability and sensitivity of TIP and ASAP testing, we are not discussing  results of those testing programs in this report. The specific data reliability  concerns related to these two testing programs are discussed later in this  report.
TSA also conducted PACE tests at category X, I, and II airports to  determine TSOs’ adherence to TSA management directives and SOPs in  areas such as overall appearance and demeanor, properly  communicating and providing instruction to passengers, and following  proper procedures. TSOs’ scores on PACE tests generally remained  above 80 percent from fiscal years 2009 through 2014. Also, based on  our review of PACE test results from fiscal years 2012 through 2014, we  determined that TSOs scored higher at smaller airports than larger  airports during this period with the difference being most pronounced  between category X airports and category II airports.
section_title: TSA Uses TSO Performance Data but Is Constrained by Incomplete and Unreliable Data and Lack of National Analysis and Recommendation Follow-Up
subsections: section_title: In 2014, TSA Reviewed Selected Annual Proficiency Review Results to Develop Courses for the 2015 National Training Plan
paragraphs: As noted previously, TSA normally uses APR testing results primarily to  assess individual TSOs’ skills for performing screening functions in order  to annually re-certify them to continue participating in screening  operations. According to TSA officials responsible for developing the  annual NTP, in fiscal year 2014, TSA’s Office of Training and Workforce  Engagement (OTWE) also examined results of specific component APR  tests to inform their development of related courses for the NTP.  Specifically, the officials stated that they reviewed the results of selected  2013 APR component tests—screening of individuals with disabilities  (IWD), bag searches, and standard pat downs. In response, the TSA  training officials said they added training to the fiscal year 2015 NTP to  specifically address the deficiencies they identified in their review of the  2013 APR component tests.
section_title: TSA Collects and Analyzes TIP Results to Ensure Quality of Test Images, but Lacks Complete Data and Does Not Analyze Results Nationwide
paragraphs: TSA policy requires airport personnel to manually download TIP testing  results from their individual X-ray machines and upload the monthly data  into TSA’s national database repository for TSA results. According to TSA  headquarters personnel responsible for overseeing TIP, they use these  uploaded results to determine if any adjustments are needed to the  quality or usefulness of the library of images maintained in the TIP system  nationwide. For example, an image for which TSOs have a high degree of  accuracy in identifying might be removed and replaced with an image that  presents more of a challenge. Conversely, an image that is frequently  missed might be reassessed to determine if the image is unrealistically  difficult and an adjustment needs to be made. However, TSA’s database  of TIP results is missing data for some airports for some years.  Additionally, TSA does not analyze the TIP data it collects on a  nationwide basis to identify potential trends in TIP test scores or  opportunities for improving screener performance.
subsections: section_title: Incomplete Data
paragraphs: While TSA uses data submitted by the airports to update its TIP image  library, it is doing so with incomplete data. As shown in figure 4, some  airports in all five airport risk categories did not report any TIP results  nationally over the course of a year from fiscal year 2010 through fiscal  year 2013. During the fiscal year 2009 through 2014 time frame, fiscal  year 2013 had the highest percentage of airports failing to report any TIP  data at nearly 14 percent. For category X and I airports, these results had  generally improved by fiscal year 2014 with all of these airports reporting  TIP data that year. However, the percentage of category III and IV  airports that did not report TIP data generally increased during fiscal  years 2013 and 2014 compared to prior years.
TSA attributed this incomplete data to a transition to new X-ray screening  equipment at certain airports from fiscal year 2009 through fiscal year  2012. Officials stated that, due to software compatibility issues with the  new machines, TIP image capability was turned off for an extended  period of time, meaning that TIP testing was not occurring on these  machines and, therefore, TIP data were neither collected nor reported for  these airports. TSA officials also told us that their older X-ray machines  do not have the capability to automatically upload TIP data results to  headquarters. As a result, some airports relying on these older X-ray  machines were not able to submit TIP data automatically by electronic  means and did not submit it manually. TSA officials reported that they do  not have a process for determining whether TIP data have been  submitted by all airports, on a regular basis, as required. TSA officials told  us they are making efforts to install automatic uploading capabilities to all  new machines that they expect will help ensure that TIP data reporting is  complete and timely. However, TSA has placed these efforts on hold  pending security concerns that must first be addressed stemming from  the recent cybersecurity breaches at the Office of Personnel Management  that have led to TSA reviewing its own cybersecurity efforts before  moving forward with installation of automatic uploading capabilities on its  X-ray machines.
TSA officials also acknowledged that, in addition to the airports discussed  above that did not report any TIP data for a year or more at a time, other  airports may have reported only partial TIP results data during this same  time frame. TSA officials stated that, in the nationwide results data  provided to GAO, it would be difficult to ascertain how much data might  be missing from individual airports (during the time period covered by our  data) since the number and type of machines in use at those airports at  any particular point in time could vary.
TSA policy requires TSA officials at airports to report all of their TIP  results data, on a monthly basis, to a national database. Further, FSDs  must monitor TIP results monthly and require TSOs to attend remedial  training if their threat identification rate falls below a target percentage.  Standards for Internal Control in the Federal Government states that the  information requirements needed to achieve the agency’s objectives  should be identified and communicated to management. Specifically,  management should obtain relevant data from reliable internal and  external sources in a timely manner based on the identified information  requirements that allows them to carry out their internal control and other  responsibilities. We acknowledge that because the full universe of X-ray  machines, and their uploading capabilities, is difficult to determine on a  daily basis, it is unlikely that TSA can fully confirm whether all of the TIP  data across the nation are being submitted. However, our review of TIP  data from fiscal year 2009 through 2014 found that up to 14 percent of  airports did not submit any TIP data in one of the years reviewed (2013).  Unless TSA takes steps to ensure that all airports submit complete,  nationwide TIP data, TSA lacks assurance that the decisions it makes on  the content of the TIP image library are fully informed, and also lacks  assurances that TSOs are receiving remedial training from the TIP  program which has been developed to aid their ability to identify  prohibited items. For example, while TSA is working to install automatic  uploading capabilities on all X-ray machines, enforcing the requirement  for airport officials to manually submit their TIP data would help ensure  more complete data by which to assess and address TIP results. In  addition, by not ensuring the collection of available TIP data, as required,  the effectiveness of any potential further use of TIP testing results to  inform TSO training or testing (as described below) programs is limited.
section_title: No National Analysis to Inform Screening Efforts
paragraphs: With regard to any potential further use of the TIP results, TSA  headquarters officials told us that, to date, they have not systematically  used the TIP results data to analyze national trends for purposes of  informing future training programs or changes to screening processes or  procedures. TSA officials said that they have not used national TIP data  in this manner due to the agency’s expectation that TIP is a tool primarily  for the benefit of local FSDs to use in monitoring the training needs, and  determining areas of focus, for their individual TSOs locally. TSA officials  at all 10 airports we contacted stated that their FSDs monitored TIP  results and used TIP data to inform their decisions on remedial or other  training needs of their TSOs. According to the TSA headquarters official  responsible for overseeing the TIP program, TSA formed an Integrated  Project Team in fiscal year 2015 specifically tasked with studying,  developing, and implementing an effective nationwide strategy and  process for using TIP testing to enhance TSOs’ threat detection skills. In  developing the planned strategy, this team is examining six focus areas— including the improvement of TIP capabilities for enhancing TSO  effectiveness through improved remedial training and updating the TIP  image library to be responsive to emerging threats. Since the team is  newly formed, it has yet to complete its work. Due to the fact that the bulk  of the team’s work is yet to be done, it is unclear how or whether these six  focus areas include plans to monitor, on a national basis, trends in the  results of TIP testing that could help highlight areas for improvement to  future image-based screening tests (such as the Image Mastery  Assessment component of APR testing) or TSO training. Standards for  Internal Control in the Federal Government states that an agency’s  management should perform ongoing monitoring of its internal control  system and associated operations, evaluate the results of those  monitoring activities, and take corrective actions when warranted to  achieve objectives and address risks. By not including analyses of TIP  results data in nationwide efforts to inform either TSO training or other  image-based testing outside of TIP, TSA is missing an opportunity to  utilize this extensive, nationwide TSO performance data for enhancing  screening operations in addition to lacking assurance that remedial  training is occurring, as required, at all airports.
section_title: ASAP Test Results are Unreliable, and TSA Does Not Ensure that Recommendations from Nationwide Analysis of ASAP Scores are Implemented
subsections: section_title: ASAP Pass Rate Results Are Unreliable
paragraphs: In an effort to assess the quality of ASAP testing conducted by TSA field  officials at commercial airports, TSA headquarters officials brought in a  contractor in fiscal year 2015 to independently perform ASAP covert  testing at 40 airports and thereby verify the validity of the testing results at  the airports. The contractor personnel performed the same type of  ASAP testing that had previously been performed by local TSA personnel  at the airports. The contractor’s initial round of covert testing was  completed in October 2015, and TSA has analyzed the results of the  contractor’s tests and compared them to ASAP tests performed  previously at the 40 airports. In doing this analysis, TSA found differences  in the test results for most of the 40 airports when comparing the  contractor’s results versus the local TSA testers’ results for the same  airports. According to TSA officials, TSOs at these 40 airports  performed more poorly in the ASAP tests conducted by the contractor  personnel as compared to the prior ASAP testing done by the local TSA  personnel—indicating that these prior-year pass rates were likely showing  a higher level of performance than was actually the case. Also, according  to the officials, these differences in test results have led them to question  the extent to which the ASAP tests accurately measure TSO  performance.
TSA is in the process of determining root causes for these variances of  testing results between the contractor and TSA personnel at the airports.  According to TSA officials, initial results from the contractor’s work seem  to confirm their prior concerns (before the contractor testing was  conducted) that problems exist with successfully maintaining the covert  nature of tests at airports. TSA officials explained that these prior  concerns were based on the high detection rates at some airports when  compared to other airports on the same tests. With respect to the difficulty  in maintaining the covert nature of the tests, TSA officials at 7 of 10 of the  airports we contacted indicated challenges with obtaining anonymous role  players to ensure that the ASAP tests remain covert. For example, TSA  officials at one airport we visited reported having to rely on the availability  of state and local government employees and U.S. Customs and Border  Protection personnel to perform as role players. Another smaller airport  we visited reported challenges finding role players among local TSA  personnel that the TSOs working the screening lanes would not  recognize. As a result, they tend to use new hires, National Guard,  Federal Aviation Administration, and Federal Bureau of Investigation  personnel. TSA officials stated that proposed changes to the ASAP SOP  will provide FSDs greater authority to use role players that they have  vetted and accepted responsibility for, beyond state, local, and federal  government officials.
In an effort to address concerns stemming from their initial analysis of the  contractor’s test results, TSA briefed its FSDs on these results and stated  that it expects the FSDs will use this information as input in overseeing  their local ASAP testing programs. In addition, TSA has extended the  work of the contractor by 6 months in order to do further testing that it can  compare to local ASAP test results going forward. TSA stated it will  continue to analyze the contractor’s results and compare them against  the ongoing results from local ASAP testing overseen by the FSDs to  determine if the previously-identified variances in results are continuing.  TSA officials stated that the findings of the contractor during the 6-month  extension period indicated that the variances previously identified in  results for the contractor testing versus the local ASAP testing at the  airports have been reduced. TSA headquarters officials attributed the  reduction in variance to more frequent and improved communication with  the FSDs and those responsible for conducting the local ASAP tests— specifically with regard to the contractor’s test findings and potential  corrective actions they should undertake to improve the local ASAP  testing programs. TSA headquarters officials added that it is through  these measures that they are improving the accountability of the local  FSDs and their staff for ensuring the quality and reliability of the local  ASAP testing going forward.
TSA officials added that, after the start of the contractor’s work, they had  initiated an effort to improve aspects of the ASAP testing program that will  include better identification of root causes for ASAP testing failures, which  they expect will improve the development of associated corrective actions  moving forward. This effort is still ongoing and also includes merging  aspects of PACE testing into the ASAP program to help identify instances  where a lack of standardization in the application of specific screening  SOPs (which PACE testing is designed to measure) may negatively  impact the screening process. Regarding TSA’s efforts to better identify  root causes of ASAP failures to improve the program, TSA has developed  a data collection tool that TSA officials said would support these efforts by  gathering critical data from test failures that they will analyze to determine  root causes of the failure. According to TSA officials, the tool has recently  been developed and field tested and is pending initial roll out.
In addition to the ASAP/PACE merger, program enhancements related to  the identification of root causes, and ongoing contractor ASAP testing,  TSA officials are adding ASAP headquarters testing to supplement the  ASAP testing that will continue to be performed by TSA personnel in the  field (referred to as the Field Evaluation Team or FET). TSA stated this  new headquarters-based testing effort will be referred to as the  Headquarters Evaluation Team (HET) and would be formed from the  former PACE evaluation teams. According to TSA, these headquarters- based covert testing teams will perform quality assurance and validation  activities for ASAP that are currently being performed by the contract test  teams. In addition, TSA expects that the contractor and new headquarters  ASAP testing program will provide assurance that the ASAP testing still  being conducted by TSA personnel at the airports is accurate. However,  field ASAP testing will still account for the majority of TSA’s ASAP covert  tests. TSA officials stated they expect once the HET program is initiated,  the contractor testing will be discontinued. Also, according to TSA, a  newly-developed data collection tool will be used by all of the ASAP  testing groups moving forward (i.e., FET, HET, and the contract test  teams) to determine the root causes of test failures that will better inform  TSA’s corrective actions.
section_title: TSA Does Not Follow Up On Implementation of Recommendations at Airports
paragraphs: TSA conducts ASAP testing in 6-month increments and produces a  summary report of results across all airports, complete with  recommendations, at the end of each 6-month cycle. In these reports,  TSA details the analysis it has performed on the nationwide results of the  ASAP testing that shows how TSOs have performed in their duties at the  various decision points on the passenger and checked baggage  screening lanes. This analysis includes failure rates at these various  points, reasons for the failures, and related recommendations where  appropriate to improve TSO performance. These recommendations may  include, among other things, additional training for certain points in the  screening process and further testing in certain areas. According to TSA  officials, they have recently moved to more frequent weekly and monthly  reporting of ASAP results to the field as part of the aforementioned effort  to improve communication with FSDs and staff with regard to findings and  trends coming from the ASAP testing results—including those results  from the ASAP contractor.
TSA headquarters does not require FSDs to implement recommendations  from the six-month cycle reports nor does it track whether the  recommendations have been implemented, or conversely, reasons for not  implementing them. TSA officials stated that the various  recommendations cited in the cycle reports are strictly for the  consideration of FSDs in the field and implementation is not mandatory.  TSA officials also stated that the ASAP cycle reports are intended to  analyze nationwide trends in TSO performance and identify causes of  potential deficiencies. TSA invests time and resources to produce these  reports—which include test results and corrective actions—on a routine  basis and disseminates the information to airport FSDs. Given this  investment, tracking implementation of the recommendations detailed in  those reports, in addition to any recommendations that may be present in  the more frequently-implemented weekly or monthly reporting, would help  TSA ensure that corrective actions are being taken at airports nationwide  to improve TSO performance, which the agency has identified as an area  of concern based on the nationwide trend analysis. Moreover, tracking  the implementation of its recommendations, including the extent to which  identified corrective actions are improving future TSO performance and  test results, will help TSA better determine the extent to which its  implemented recommendations are leading to improvements in screening  operations and appropriately addressing identified root causes for  previous test failures.
Standards for Internal Control in the Federal Government requires that  internal controls be designed to ensure that ongoing monitoring occurs  during the course of normal operations. Specifically, internal controls  direct managers to (1) promptly evaluate and resolve findings from audits  and other reviews, including those showing deficiencies and  recommendations reported by auditors and others who evaluate  agencies’ operations; (2) determine proper actions in response to findings  and recommendations from audits and reviews; and (3) complete, within  established time frames, all actions that correct or otherwise resolve the  matters brought to management’s attention. We recognize the efforts  TSA has recently initiated to improve the accuracy and reliability of ASAP  testing. However, without the assurance that recommendations for  corrective actions based on the root causes identified in ASAP testing will  be fully implemented—where appropriate—nationwide, TSA will be  limited in its ability to take full advantage of any findings from the  program.
section_title: Conclusions
paragraphs: Training TSOs and obtaining an accurate understanding of their  effectiveness in detecting prohibited items on passengers, and in their  baggage, can have a critical impact on the security of millions of air  travelers each year. TSA has put an extensive program in place to train  its TSOs to perform these critical screening functions and responded to  recent covert test findings of the DHS OIG by implementing a retraining  program for all its screening officers to address issues identified in the  testing. TSA has also begun implementing a plan to expand evaluations  of its TSO training efforts in order to better inform future management  decisions. In addition to its training and evaluation efforts, TSA conducts  wide-ranging covert testing and annual certification testing of its TSOs.  While we commend TSA’s recent efforts to re-examine its testing  programs, such as steps to improve the accuracy and reliability of ASAP  testing, the agency could further enhance its testing programs to more  accurately gauge the true level of TSO performance and ensure  continuing improvement in screening operations. For example, enforcing  its requirement that all airports submit TIP results data would help TSA  continually improve the test. Further, the agency could use these data on  a nationwide level to inform and potentially improve training of TSOs in  screening passenger carry-on baggage for prohibited items. In addition,  given that TSA uses ASAP covert testing results to assess whether TSOs  follow proper screening procedures and successfully detect prohibited  items, ensuring that any recommendations stemming from the ASAP  testing failures are tracked and implemented, where appropriate, would  further support the program’s objective to improve the performance and  quality of security screening.
section_title: Recommendations for Executive Action
paragraphs: To improve TSA’s ability to take full advantage of testing results to inform  and potentially improve screening operations, we recommend that the  Secretary of the Department of Homeland Security direct the  Administrator of TSA to take the following three actions:
Ensure that TSA officials at individual airports submit complete TIP  results to the TSA national database as required, including manually  submitting data when automated uploading is not available.
Conduct analysis of national TIP data for trends that could inform  training needs and improve future training and TSO performance  assessments.
Track implementation by airports of ASAP recommendations to  ensure that corrective actions identified through ASAP testing are  being applied.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of the sensitive version of this report to DHS for their  review and comment. DHS provided written comments, which are noted  below and reproduced in full in appendix II, and technical comments,  which we incorporated as appropriate.
DHS concurred with all three recommendations in the report and  described actions underway or planned to address them. With regard to  the first recommendation that TSA ensure TIP data is submitted to the  TSA national database as required, DHS concurred and stated that TSA  is working to establish a tracking system that will automatically identify  and highlight specific airports that may be missing from the database. The  automated system will allow TSA to establish an internal webpage that  will automatically generate a list of airports that have not submitted TIP  data as required, and which managers will be able to use to follow-up with  Federal Security Directors to ensure TIP data is submitted. The agency  stated that the automated process is dependent on the development of an  information technology (IT) tool which they anticipate will be piloted by  May 31, 2017. In the interim, while this IT tool is being developed, TSA  officials will monitor compliance with TIP reporting requirements and  follow up with those airports missing TIP data, including identifying  reasons for the airport’s non-compliance. TSA is also drafting a revised  TIP Operations Directive that is intended to provide further guidance and  direction to the field on TIP requirements. TSA estimates they will  complete these actions to address the first recommendation by  September 30, 2016.
With regard to the second recommendation to conduct analysis of  national TIP data for trends that could inform training needs and improve  future TSO performance, DHS concurred and detailed the following  actions to address this recommendation:
TSA’s Office of Training and Development (OTD) has begun to  update TIP remediation requirements and work with airports that have  achieved the highest TIP scores to identify any best practices that  could be shared with other airports.
OTD plans to work with airports that struggle with TIP to identify  information about their oversight and remediation program with the  goal of using the highest and lowest scoring airports to assess the  effect of oversight and remediation on performance. TSA plans to  analyze data across the network to determine what remediation  training best supports improvements in TIP scores.
TSA is developing a process to analyze specific data connected to  threat categories of TIP images which will allow officials to identify the  specific types of threats that are presenting challenges to the  workforce. OTD will then be able to identify what additional training  should be developed to improve performance for that particular threat  category.
TSA plans to assess TIP training and assessments over the next 12  months to determine if performance improvement has been realized,  and if so, what contributed to the improvement.
OTD is working with a contractor to design a report that is intended to  capture officer performance results connected to specific types of TIP  images to better drive training content and improve performance.
TSA’s Office of Security Capabilities is working with both OTD and the  Office of Security Operations (OSO) to capture TIP data for the  development of threat categories to assess individual TSO’s  performance and asking TSA’s Office of Acquisitions for a contract  modification that will provide for more frequent report updates.
TSA estimates they will complete these actions to address the second  recommendation by May 31, 2017.
With regard to the third recommendation to track implementation by  airports of ASAP recommendations to ensure that corrective actions are  being applied, DHS concurred and stated that TSA has taken actions to  formalize ASAP reporting. For example, TSA has reported developing a  standard format for Corrective Action Plans, which are submitted and  implemented after an ASAP failure. This should help TSA track corrective  actions and their effectiveness in addressing findings from ASAP tests.  Further, TSA plans to conduct reassessments within 30-60 days after a  Corrective Action Plan has been submitted to ensure corrective actions  have been implemented. TSA also reported that the standard format for  CAPs deliberately maps corrective actions to their identified issues.  According to TSA, as of August 2016, OSO has conducted more than 55  post-Headquarters Evaluation Team testing calls and more than 50  effectiveness calls to review CAPs. OSO has extracted common themes  from high performing airports and distributed this “best practice”  information to all its regional directors and federal security directors. TSA  also stated that OSO is reassessing those previously-tested airports to  ensure that corrective actions are implemented and detection  performance is improving at or above the national average. These efforts  by TSA to ensure that corrective actions identified through ASAP testing  are being applied, if continued in future testing cycles, should address the  intent of this recommendation. These completed actions for the third  recommendation along with the planned actions for the first and second  recommendations, if fully implemented, should address the intent of the  three recommendations contained in this report.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Homeland Security, the Attorney General of  the United States, and other interested parties. In addition, the report is  available at no charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7141 or groverj@gao.gov. Key contributors to this report  are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report answers the following questions:  1.  How does the Transportation Security Administration (TSA) train  Transportation Security Officers (TSO), and to what extent does TSA  evaluate the training?  2.  How does TSA measure the performance of TSOs, and what do the  performance data show?  3.  To what extent does TSA use TSO performance data to enhance  TSO performance?
To address our first objective regarding how TSA trains TSOs and to what  extent TSA evaluates the training, we reviewed relevant TSA policies and  procedures for training, including management directives and the National  Training Plan (NTP), which prescribes the annual training curriculum for  TSOs. We also reviewed documentation on training requirements,  including those contained in the Aviation and Transportation Security Act,  as well as documents on TSA’s training development and completion.  We interviewed TSA headquarters officials responsible for developing  and monitoring TSO training, including officials from TSA’s Office of  Training and Development (OTD), Office of Human Capital (OHC), and  the Office of Security Operations (OSO). Further, we interviewed staff  from a total of 10 airports—including Federal Security Directors (FSD),  transportation security managers, instructors, training managers, TSOs,  and other TSA staff, such as explosives experts, to determine how  training is carried out in the field and to learn what TSA employees in the  field thought about training. Specifically, we conducted site visits to six  airports, including three airports in category X, and one airport each in  Categories I, II, and III. Further, we conducted phone interviews with  officials at one airport each in categories I, II, III, and IV to obtain  additional perspectives on how airport officials carry out training  requirements locally—particularly at airports with smaller numbers of  flights and passenger boardings. We selected the airports to visit in  person based on factors such as airport category, geographic proximity to  one another, and our analysis of the airports’ TSO performance on annual  screening certification tests from calendar years 2009 through 2014. For  example, we calculated the average first time pass rates for screeners  taking their Annual Proficiency Review (APR) exams for each airport in  each calendar year from 2009 to 2014 and sorted the scores by airport  risk category. APR assessments are annual certification tests TSOs must  pass to remain employed as a screener. We then selected at least one  airport from the high, low, and middle of the performance distribution and  made sure to cover at least one airport in every risk category.
To assess the extent to which TSA evaluates TSO training, we reviewed  TSA documents used for evaluating training courses, including end-of- course surveys administered to learners. Further, we reviewed draft  documents on TSA’s training evaluation plan, including a draft  management directive and draft standard operating procedures for  evaluating training courses. We compared the training evaluation  documentation to the Kirkpatrick model for training evaluation, which is  the model TSA uses as guidance for its evaluations of TSO training. We  also interviewed TSA headquarters officials responsible for evaluating  TSO training and for developing and implementing the TSA training  evaluation plan. For example, we interviewed TSA officials from OTD,  OSO, and OHC to determine the extent to which they evaluated training  courses and used this information to refine future training. Further, we  interviewed management officials at each of the airports we visited to  further understand how, if at all, training at individual airports is evaluated  locally.
For our second objective, to determine how TSA measures the  performance of TSOs and what the performance data show, we analyzed  data from four different performance evaluation programs and we  interviewed TSA officials responsible for collecting and analyzing the  data. First, we reviewed and analyzed data on APRs, including analyzing  APR pass rates from calendar year 2009 (the first year for which data  were available) through 2015. For example, we calculated the average  first time pass rate for screeners taking the APR assessments for each  airport and sorted the results by year, airport category, and by each  individual APR assessment. See Table 4 for a description of the APR  assessments we analyzed. We then conducted a trend analysis to  observe overall APR first-time pass rates over time, and we compared  APR first-time pass rates for screeners across airport risk categories to  determine whether there were any differences in pass rates across airport  categories. In addition, we interviewed officials in charge of the APR  testing process, including officials from OSO, OHC, OTD from TSA  headquarters, as well as local airport officials in charge of overseeing the  tests.
Second, we reviewed Threat Image Projection (TIP) system data from  fiscal year 2009 to fiscal year 2014, the last year available at the time of  our data request. The TIP system is intended to help TSA measure  whether operators correctly identify threat items that are electronically  superimposed on the X-ray monitor during the screening of passenger  property at the checkpoint. Specifically, we analyzed the average  percentage of TIP images correctly identified during screening by  screeners at different airport categories over time to determine whether  there were differences in average TIP scores between airport categories.  Further, we interviewed TSA officials in charge of the TIP image library  from the Office of Security Capabilities to understand how TIP data are  recorded and collected, and how the TIP images are selected for use.
Third, we reviewed data from TSA’s Presence, Advisement,  Communication, and Execution (PACE) testing program, which TSA uses  to measure whether TSOs are adhering to standard operating procedures  while screening at the passenger checkpoint. We reviewed PACE data  from calendar year 2011, the year the program was started, until 2014  and charted PACE scores by airport category across time. We also  interviewed appropriate TSA officials regarding the PACE program to  understand how the program worked and how the scores were  calculated.
Finally, we analyzed data from the Aviation Screening Assessment  Program (ASAP), a covert testing program used to evaluate screeners’  ability to properly follow TSA’s standard operating procedures for  screening and keep prohibited items from being taken through the  checkpoint.  We analyzed ASAP data from fiscal years 2013 through  2015 because TSA made adjustments to the ASAP testing program in  2013, and therefore the pre-2013 testing data are not comparable to the  2013 through 2015 data. Results of ASAP testing are classified at the  secret level and are not included in this report. Additionally we  interviewed TSA officials from OSO responsible for the ASAP program to  gain their perspectives on the program. We also interviewed officials  responsible for conducting ASAP tests at each of the airports we visited  to understand how the tests worked in practice, what happened after a  test was passed or failed, and to learn about any challenges officials  faced in running the tests.
We assessed the reliability of the APR, TIP, PACE, and ASAP data by (1)  interviewing agency officials responsible for maintaining the data about  how the data were collected and entered into the respective databases,  how the data were used, and what procedures were in place to ensure  the data were complete; and (2) testing the data for missing data,  duplicates, or entries that otherwise appeared to be unusual. We found  the APR and PACE data to be sufficiently reliable to present in this  report. However, we found that the TIP data were incomplete for the  years we were analyzing and therefore not sufficiently reliable to include  in this report.
Specifically, TSA officials in charge of the TIP data stated they were  uncertain how complete the TIP data were nationwide at any point in  time, but added that it is likely never fully complete. Officials stated that  this was due to two reasons. First, when TSA first deployed new X-ray  machines between 2009 and 2012, the TIP software was not activated on  them due to technical issues. As a result, no TIP data were reported for  those machines over this period. Second, the newer X-ray machines  coming online are equipped to upload TIP data to TSA headquarters  automatically over a network. However, not all machines in the field are  equipped to do this, and TSA temporarily stopped implementation of the  automatic upload capability on the newer machines in 2015 because of  network security concerns. Instead, TSA personnel must manually  download the TIP data for these machines on a monthly basis as it does  for older machines without this automatic upload capability. As a result,  TSA headquarters has not received TIP data from every airport for every  month over the time period of our review resulting in the database being  incomplete. TSA could not provide us with information on the extent of the  missing data and we were not able to determine based on the data  provided how many X-ray machines were unaccounted for between 2009  and 2014.
For our third objective, to determine the extent to which TSA uses TSO  performance data to enhance screening performance, we reviewed TSA’s  processes and actions for using screener testing results to inform its  operations and training, and assessed these processes against standards  in Standards for Internal Control in the Federal Government. Further, we  interviewed program officials from several offices at TSA headquarters  about how they analyze performance data such as APR, TIP, and PACE  data, and how, if at all, they use the results to adjust training or take any  other actions. Similarly, we interviewed officials from each of the airports  we visited about how the collected, reported, monitored, and used the  performance data they collected.
We conducted this performance audit from February 2015 to September  2016, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Homeland Security
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Jennifer Grover (202) 512-7141 or groverj@gao.gov.
section_title: Acknowledgments
paragraphs: In addition to the contact named above, Christopher E. Ferencik,  Assistant Director; Mike Harmond, Analyst in Charge; and Brendan  Kretzschmar made key contributions to this report. Also contributing to  the report were, Eric D. Hauswirth, Susan Hsu, Thomas F. Lombardi,  Heidi Nielson, Ying Long, Amanda Miller, Ruben Montes de Oca, Dae  Park, and Christine San.