Aviation Security: TSA Should Ensure Testing Data Are Complete and Fully Used to Improve Screener Training and Operations
Released: 2016-09-07T00:00:00

Letter

Background

After the terrorist attacks of September 11, 2001, the President signed the Aviation and Transportation Security Act (ATSA) into law on November 19, 2001, with the primary goal of strengthening the security of the nation’s civil aviation system. ATSA created TSA as the agency with responsibility for securing all modes of transportation, including civil aviation. As part of this responsibility, TSA performs or oversees the performance of security operations at the nation’s nearly 440 commercial (i.e., TSA-regulated) airports, including passenger and checked baggage screening operations. FSDs are TSA officials responsible for overseeing TSA security activities, including passenger and checked baggage screening, at one or more commercial airports. TSA classifies commercial airports in the United States into one of five security risk categories (X, I, II, III, and IV) based on various factors, such as the total number of takeoffs and landings annually, and other special security considerations. In general, category X airports have the largest number of passenger boardings and category IV airports have the smallest. TSA periodically reviews airports in each category and, if appropriate, updates airport categorizations to reflect current operations. Figure 1 shows the number of commercial airports by airport security category as of July 2015.

Passenger Screening

TSA uses a multilayered security strategy aimed to enhance aviation security. Within those layers of security, TSA’s airport passenger checkpoint screening system includes, among other things, (1) screening personnel (i.e., TSOs); (2) SOPs that guide screening processes conducted by TSOs; and (3) technology, such as advanced imaging technology systems (often referred to as body scanners) or walk-through metal detectors, used to conduct screening of passengers. To carry out passenger and checked baggage screening operations, TSA employs TSOs at the vast majority of the nation’s commercial airports. There are several levels of screening officers deployed at the passenger checkpoint:

Transportation Security Officer (TSO): Performs the majority of security functions to screen people and property to mitigate threats. Screening may include pat downs, search of property, and operating technology including walk-through metal detectors, X-ray machines, and explosives detection equipment, among other things.

Lead Transportation Security Officer (LTSO): Leads a staff of TSOs, including distributing and adjusting workload and tasks among employees and oversees the security screening team on a daily basis. Implements security procedures and provides coaching and guidance to TSOs in performing screening duties, among other things. LTSOs also perform screening functions along with added responsibilities, such as resolving alarms and supervising screening locations when a supervisor is not available.

Supervisory Transportation Security Officer (STSO): Oversees screening checkpoints and/or baggage screening, supervises LTSOs and TSOs in performance of security screening and ensures all required screening is performed in accordance with SOPs. Reviews and evaluates work and performance of LTSOs and TSOs, approves leave, and recommends corrective or disciplinary actions, among other things. STSOs also perform screening functions and resolve passenger alarms.

Transportation Security Manager (TSM): Coordinates and facilitates TSA security activities and manage one or more programs as assigned by the Federal Security Director. A TSM assigned to oversee screening checkpoints manages security activities, including recognizing and correcting improper use or application of equipment or screening procedures, monitors screening operations, and implements changes to enhance security and efficiency at screening locations.

TSOs inspect individuals and property as part of the passenger screening process to deter and prevent the carriage of any unauthorized explosive, incendiary, weapon, or other prohibited items on board an aircraft or into the airport sterile area—in general, an area of an airport that provides passengers access to boarding aircraft and to which access is controlled through the screening of persons and property. Ordinarily, screening of accessible property at the screening checkpoint begins when an individual places accessible property on the x-ray conveyor belt or hands accessible property to TSA personnel. As shown in figure 2, TSOs then review images of the property running through the X-ray machine and look for signs of prohibited items. The passengers themselves are typically screened via a walk-through metal detector or an advanced imaging technology machine, and passengers generally have the option to request screening by a pat down if they do not wish to be screened by these technologies. Passengers will also be subject to a pat down if they are screened by the walk-through metal detector or advanced imaging technology system and the equipment alarms (in order to resolve the alarm).

Checked Baggage Screening

TSOs also inspect checked baggage to deter, detect, and prevent the carriage of any unauthorized explosive, incendiary, or weapon onboard an aircraft. Figure 3 shows the general process used to screen checked bags. Checked baggage screening is accomplished through the use of explosives detection systems or explosives trace detection systems, and through the use of alternative means, such as manual searches and canine teams when the explosives detection systems are unavailable.

Screener Training Requirements

In accordance with ATSA, screeners must complete a minimum of 40 hours of classroom instruction, 60 hours of on-the-job training, and successfully complete an on-the-job training examination before they are certified as security screeners. Screeners can be certified to conduct passenger screening or checked baggage screening, or they may be certified as dual function and can then conduct passenger and checked baggage screening. ATSA also requires that TSA provide operational testing of screening personnel, and any individual who fails an operational test must successfully complete remedial training on that specified security function before returning to duty. In addition, screeners must also undergo an annual proficiency review to ensure they continue to meet all qualifications and standards required to perform a screening function. TSA also requires remedial training for TSOs who fail an annual proficiency review.

Covert tests recently conducted by the DHS-OIG highlighted areas of concern for TSA regarding the effectiveness of the passenger screening process. Specifically, the DHS-OIG conducted covert testing to determine the effectiveness of TSA’s Advanced Imaging Technology screening equipment, its related automated target recognition software, and checkpoint screener performance in identifying and resolving potential security threats at airport checkpoints. TSA has responded to the DHS Secretary’s direction regarding the results of the DHS-OIG covert testing, in part, by updating its screening SOPs and retraining TSOs to address the Inspector General’s findings. Also in response to the DHS-OIG findings, TSA has developed new measures of effectiveness that it expects will better emphasize the agency’s goals for improving security effectiveness by focusing the measures on both the screening system and workforce in the areas of readiness and performance. For example, improved workforce measures, now being reported monthly, include those to track TSOs’ progress against training requirements, absences due to injuries or other reasons, and whether they are meeting performance thresholds on various tests of performance and job proficiency.

TSA Uses a Variety of Programs to Train TSOs and Is Developing a Plan to Expand Evaluations of TSO Training

TSA Uses New Hire, Recurrent, Remedial, and Return-to-Duty Training Programs for TSOs

TSO training is comprised of a compendium of courses that includes basic training for initial hires, recurrent training, remedial training, and return-to-duty training. For example, all new hires receive a combination of classroom, hands-on, and web-based training. After TSOs finish their initial new hire training, they receive recurrent and specialized training courses throughout the year that are provided either via classroom instruction or through the TSA Online Learning Center. Recurrent training typically focuses on core screening skills and policies such as X-ray image interpretation, detection techniques, and screening SOPs. TSOs receive remedial training when they have failed an operational or certification test, or if a supervisor identifies a need for further training, among other things. Further, according to TSA, TSOs who are absent from their screening duties for a period of time must undergo some level of “return-to-duty” training based on the amount of time they were absent. For example, TSOs certified in a screening function but who have not performed that function for a period of 15 consecutive days or more are required to complete a return-to-duty training program before being allowed to perform that function independently. Table 1 describes the various types of training TSOs receive.

The Office of Training and Development (OTD), within TSA headquarters, oversees the development, delivery, and evaluation of training programs for TSA employees. The National Training Plan (NTP), developed jointly by OTD and the Office of Security Operations, contains the core curriculum for TSOs to meet their annual training, including the classes and hours required for TSOs to complete for the year. TSA headquarters officials implement the NTP to provide ongoing training throughout the year aimed at continually improving screeners’ knowledge, skills, and abilities. However, the responsibility for managing the individual training of TSOs is largely decentralized and it primarily falls on Security Training Instructors at individual airports to train TSOs on parts of the NTP by certain dates throughout the year. Managers in the field track the percentage of the NTP curriculum that TSOs have completed on a monthly basis using the Online Learning Center database. In addition, TSA officials at all 10 airports we contacted stated that they monitor various testing results for their TSOs and observe screening operations at their airports’ checkpoints, to determine any local, specialized training needs their screening force may need—over and above that included in the NTP issued by TSA headquarters. TSA headquarters can also add training requirements throughout the year as needed, such as the recently completed “Mission Essentials—Threat Mitigation” training discussed later.

TSA officials we spoke with at airports noted challenges associated with completing not only the required training under the NTP, but also training associated with frequent changes to the screening SOPs for how particular screening practices are to be conducted at the checkpoint. For example, TSA personnel at 8 of the 10 airports stated that it was sometimes difficult to meet the training requirements in the NTP because they did not have the TSO personnel to both staff the checkpoints and get all the required training accomplished. Specifically, TSA officials at larger airports with more passenger throughput, such as category X and category I airports, reported having ongoing challenges balancing training with the operational needs at the checkpoint. In contrast, TSA officials at smaller airports, such as category III and IV airports did not report having this challenge frequently. TSA officials at all three of the category X airports stated they addressed the challenge of meeting training requirements by scheduling large amounts of training during slower travel seasons for their airport so they would not have to spend time training TSOs during peak travel periods. Starting in fiscal year 2014, TSA headquarters began sending the majority of the NTP training requirements for the entire fiscal year out to the field at the beginning of the year—rather than at quarterly intervals throughout the year—allowing airports the flexibility to train TSOs at different rates depending on the operational needs of the airport.

TSA training officials at 6 of the 10 airports stated that it is challenging for them to keep the TSOs trained on the frequent changes to screening SOPs. For example, TSA officials from two airports stated that during the delivery of a recent NHTP class, screening SOPs were updated to require an officer to use a handheld metal detector to resolve an alarm arising from a passenger going through an advanced imaging technology scanner. Due to the change happening while the new officers were in the middle of their introductory training, the steps for using the handheld metal detector were not integrated into the NHTP curriculum. As a consequence, after the NHTP course was completed, TSA instructors separately trained the new hires on how to conduct this type of alarm resolution. In addition, TSA officials at 9 airports we spoke with stated that the TSOs used “read and sign” binders to train on some SOP changes, where the officers sign a document stating they read the change to the screening SOPs. However, officers reported that this type of training did not ensure they understood how to implement the change at the screening checkpoint. According to TSA headquarters officials, they plan to conduct more hands-on training to teach screening SOP changes moving forward. Further, TSA personnel at 7 of the 10 airports added that many of the screening SOPs can have room for interpretation, which also prompted officials at 2 of these airports to create new airport-level training to address whether to let particular items through the checkpoint such as bowling balls and other heavy, blunt objects.

TSA Recently Implemented a Retraining Program for its TSOs in Response to Findings of the DHS OIG

TSA implemented a TSO re-training program in fiscal year 2015 to retrain its screening workforce in response to findings of the DHS-OIG, which conducted its own covert testing of TSA’s checkpoint operations and technology in the spring of 2015. Specifically, in response to the DHS- OIG findings, TSA provided additional training nationwide to all TSOs— referred to as “Mission Essentials—Threat Mitigation” training. According to TSA documentation, the purpose of this 8-hour classroom training was to provide the opportunity for the workforce to become familiar with the intelligence and threat information that underlies TSA’s use of checkpoint technologies, operational procedures, and the TSO workforce to mitigate threats. TSA officials described the training as covering the “why” behind the equipment and procedures TSA uses to screen passengers and baggage. For example, the training included: instruction on how social engineering techniques may be used in an attempt to defeat TSA risk mitigation procedures, updates on SOP changes for screening certain types of passengers, demonstrations on Improvised Explosive Devices (IED) and how pat downs are used to mitigate the threat, and an overview of checkpoint equipment capabilities and limitations and the role of using screening SOPs and best practices to mitigate gaps caused by equipment limitations.

In addition to the 8-hour course provided for screening officers, supervisors were provided additional training on their responsibilities for ensuring the correct implementation of the checkpoint SOPs and how to provide on-the-spot corrections and constructive feedback to officers. TSA officials added that, in order to ensure enhanced mission focus, the agency will begin sending all new-hire TSOs to the TSA Academy at the Federal Law Enforcement Training Center in Glynco, Georgia rather than conducting the classroom portion of the NHTP at individual airports. The officials stated this would help standardize the new hire training and provide a sense to the new hires that they are part of something larger than just their local airport. TSA officials stated the first new-hire classes started at the TSA Academy in January 2016.

TSA Evaluates Select Training Courses Using Elements of the Kirkpatrick Model

To evaluate its training of TSOs, TSA generally follows the Kirkpatrick model, which is a commonly accepted training evaluation model endorsed by the Office of Personnel Management (OPM) and used throughout the federal government. Currently, using the model, TSA implements training evaluation surveys and conducts analysis of the responses for a select number of training courses. TSA’s goal for conducting Kirkpatrick- style training evaluation is to answer questions such as how well a training course met a learner’s needs; what knowledge and skill a course imparted to learners; what impact the training had on learner performance; and what the benefits of the training were.

The Kirkpatrick model consists of a four-level approach for soliciting feedback from training course participants and evaluating the impact the training had on individual development, among other things. Table 2 provides a description of what each level within the Kirkpatrick model is to accomplish and TSA’s progress in implementing the levels.

TSA Is Developing a Program to Expand Training Evaluations and Support Management Decisions on Training Resources

According to TSA officials, the agency is developing a training evaluation program that will allow it to standardize and expand training evaluation efforts. In 2013, TSA assessed its training evaluation practices and found that existing training evaluation efforts did not meet TSA’s needs because they lacked a formal, comprehensive approach to training evaluation. As a result, TSA identified the need to establish a formal training evaluation program, based on the Kirkpatrick model, to standardize its policy, processes, and procedures for evaluating training and has been working to establish the program since December of 2013. TSA’s Standards and Integration Office, within the Office of Training and Development, has developed a plan for implementing the new evaluation program, which is intended to support agency leadership in making decisions on how to use training resources. In addition, TSA expects to approve a Management Directive and Standard Operating Procedures for the training evaluation program by May 2016 to define the roles and responsibilities for TSA offices running the training evaluation program as well as lay out the steps for analyzing and reporting data collected from the training evaluations. TSA officials stated the training evaluation plan will be subject to annual revision, and OTD will continue to update and review the plan. Standards and Integration Office officials are responsible for developing the training evaluations and collecting the evaluation data while TSA’s Training Operations Division will administer the training evaluations.

TSA’s training evaluation plan describes the types of stakeholders involved in training evaluation, the communications strategy for sharing information on training across the agency, and the reporting requirements for training evaluation. For example, the plan identifies, in broad terms, which Kirkpatrick Level evaluation reports the Standards and Integration Office Evaluations Team will generate, who will receive the reports, and how they will be used. In one example, the reporting plan shows that Levels 1, 2, and 3 reports should be sent to program managers to help them allocate screening resources and modify training. This program strategy, if followed, should position TSA to make data-based strategic decisions on the effectiveness of training courses once the training evaluation plan is fully implemented. For example, TSA plans to use training evaluation data to conduct curriculum reviews to improve training courses and programs.

TSA is planning to implement its new training evaluation program in four phases. During the first phase, TSA plans to implement Level 1 and Level 3 training evaluations for their TSO Basic Training Program and for core operational courses, and to collect and analyze the data from these evaluations. In phase two TSA plans to expand Level 1 and 3 training evaluations to key courses in their National Training Plan. Phase two is scheduled to begin in late 2016. Once TSA has implemented these training evaluations for TSO Basic Training and for courses in the NTP, TSA plans to add selected Online Learning Center courses to their training evaluation program which constitutes phase three. Finally, in phase four, TSA plans to evaluate whether they need new training courses, and if so, all newly approved training courses would be required to develop an evaluation plan.

TSA Uses a Variety of Methods to Measure TSO Performance, and Results Vary by Type of Task Tested and Airport Risk Category TSA Measures TSO Performance through Tests Conducted in a Non- Operational Setting and at Active Checkpoints

TSA uses a variety of methods to measure the performance of its TSOs, including the Annual Proficiency Review (APR)—an annual certification program to evaluate TSOs’ skill in performing the various screening functions. Portions of the APR are computer-based X-ray image tests done in a non-operational setting away from the active checkpoints while the remaining tests are skills demonstrations performed in a realistic, but inactive, screening environment such as an unused screening lane. Which components of the APR an individual TSO must take are dependent on whether that TSO is certified to perform passenger screening, baggage screening, or has dual certification to perform both functions.

TSA has other testing programs that take place during active operations at the checkpoints to assess TSOs’ level of adherence to screening SOPs and associated management directives. These include the Threat Image Projection (TIP) image testing; the Aviation Screening Assessment Program (ASAP); and Presence, Advisement, Communication, and Execution (PACE) covert testing. Table 3 provides a summary of TSO performance measurement tests.

In addition to the ASAP covert testing and other tests detailed in Table 3 for assessing the effectiveness of TSOs in carrying out screening functions, the TSA Office of Inspection (OOI) Special Operations Division (SOD) regularly conducts independent covert “red team” testing to measure the effectiveness of TSA security systems and identify vulnerabilities in transportation security as a whole. TSA develops and deploys red team tests based upon current intelligence of threats against transportation systems. In addition to assessing TSOs’ ability to detect threat items similar to ASAP testing, OOI’s red team covert testing also assesses the effectiveness of other aspects of the screening operation— including screening procedures followed by the TSOs and the technology they use at the checkpoints.

TSA policy requires FSDs to provide remedial training to TSOs who either fail components of the APR (before being allowed to retake those portions) or do not maintain a minimum score on TIP image tests. Similarly, TSOs who fail ASAP or red team covert tests—that is, operational tests—must take, in accordance with ATSA, remedial training before returning to their screening duties. TSA policy has not specifically required remedial training for any TSOs who failed PACE tests. Instead, each airport’s FSD was expected to make their own determination regarding any necessary retraining based on the PACE testing results.

Results of APR and PACE Screener Performance Tests Varied by Specific Task Tested and Airport Risk Category

TSA data on the results of APR and PACE testing show that TSOs’ pass rates on both of these tests varied by airport risk category over the time periods we reviewed. Specifically, from calendar year 2009 through 2015, the percentage of TSOs that passed their APR certification tests on the first attempt remained relatively constant, with a dip occurring in calendar year 2010 followed by an increase by a similar percentage in 2015. According to TSA officials, this performance dip occurred because TSA ended the practice of using an outside contractor to evaluate TSOs during the APR tests. TSA officials explained that the TSA personnel who took over the evaluation function displayed less flexibility than was previously allowed in scoring of the various APR component tests in that first year after the transition (2010).

According to TSA officials, the aforementioned changes to the APR testing program for 2015 (including practice runs prior to grading the practical skills evaluation portions of the test and dividing the testing by quarters) have led to improvement in the overall APR pass rates for 2015 compared to prior years. TSA officials explained that they decided to re- examine how they conducted APR testing and implemented the resulting changes in response to feedback from TSOs that certain aspects of the testing created unnecessary anxiety which affected their performance.

As described earlier, APR consists of several component tests that evaluate specific TSO functions. As shown in table 4, these component tests include X-ray image testing and passenger pat downs, which cover actions taken by TSOs in routine screening operations at the passenger and baggage screening checkpoints.

In addition to the overall APR pass rates varying by airport security category, the results of these individual component tests also varied by the type of test administered during the 2009 to 2015 timeframe. Scores for specific APR components tests are Sensitive Security Information and not included in this report. In addition, due to issues with both the reliability and sensitivity of TIP and ASAP testing, we are not discussing results of those testing programs in this report. The specific data reliability concerns related to these two testing programs are discussed later in this report.

TSA also conducted PACE tests at category X, I, and II airports to determine TSOs’ adherence to TSA management directives and SOPs in areas such as overall appearance and demeanor, properly communicating and providing instruction to passengers, and following proper procedures. TSOs’ scores on PACE tests generally remained above 80 percent from fiscal years 2009 through 2014. Also, based on our review of PACE test results from fiscal years 2012 through 2014, we determined that TSOs scored higher at smaller airports than larger airports during this period with the difference being most pronounced between category X airports and category II airports.

TSA Uses TSO Performance Data but Is Constrained by Incomplete and Unreliable Data and Lack of National Analysis and Recommendation Follow-Up

In 2014, TSA Reviewed Selected Annual Proficiency Review Results to Develop Courses for the 2015 National Training Plan

As noted previously, TSA normally uses APR testing results primarily to assess individual TSOs’ skills for performing screening functions in order to annually re-certify them to continue participating in screening operations. According to TSA officials responsible for developing the annual NTP, in fiscal year 2014, TSA’s Office of Training and Workforce Engagement (OTWE) also examined results of specific component APR tests to inform their development of related courses for the NTP. Specifically, the officials stated that they reviewed the results of selected 2013 APR component tests—screening of individuals with disabilities (IWD), bag searches, and standard pat downs. In response, the TSA training officials said they added training to the fiscal year 2015 NTP to specifically address the deficiencies they identified in their review of the 2013 APR component tests.

TSA Collects and Analyzes TIP Results to Ensure Quality of Test Images, but Lacks Complete Data and Does Not Analyze Results Nationwide

TSA policy requires airport personnel to manually download TIP testing results from their individual X-ray machines and upload the monthly data into TSA’s national database repository for TSA results. According to TSA headquarters personnel responsible for overseeing TIP, they use these uploaded results to determine if any adjustments are needed to the quality or usefulness of the library of images maintained in the TIP system nationwide. For example, an image for which TSOs have a high degree of accuracy in identifying might be removed and replaced with an image that presents more of a challenge. Conversely, an image that is frequently missed might be reassessed to determine if the image is unrealistically difficult and an adjustment needs to be made. However, TSA’s database of TIP results is missing data for some airports for some years. Additionally, TSA does not analyze the TIP data it collects on a nationwide basis to identify potential trends in TIP test scores or opportunities for improving screener performance.

Incomplete Data

While TSA uses data submitted by the airports to update its TIP image library, it is doing so with incomplete data. As shown in figure 4, some airports in all five airport risk categories did not report any TIP results nationally over the course of a year from fiscal year 2010 through fiscal year 2013. During the fiscal year 2009 through 2014 time frame, fiscal year 2013 had the highest percentage of airports failing to report any TIP data at nearly 14 percent. For category X and I airports, these results had generally improved by fiscal year 2014 with all of these airports reporting TIP data that year. However, the percentage of category III and IV airports that did not report TIP data generally increased during fiscal years 2013 and 2014 compared to prior years.

TSA attributed this incomplete data to a transition to new X-ray screening equipment at certain airports from fiscal year 2009 through fiscal year 2012. Officials stated that, due to software compatibility issues with the new machines, TIP image capability was turned off for an extended period of time, meaning that TIP testing was not occurring on these machines and, therefore, TIP data were neither collected nor reported for these airports. TSA officials also told us that their older X-ray machines do not have the capability to automatically upload TIP data results to headquarters. As a result, some airports relying on these older X-ray machines were not able to submit TIP data automatically by electronic means and did not submit it manually. TSA officials reported that they do not have a process for determining whether TIP data have been submitted by all airports, on a regular basis, as required. TSA officials told us they are making efforts to install automatic uploading capabilities to all new machines that they expect will help ensure that TIP data reporting is complete and timely. However, TSA has placed these efforts on hold pending security concerns that must first be addressed stemming from the recent cybersecurity breaches at the Office of Personnel Management that have led to TSA reviewing its own cybersecurity efforts before moving forward with installation of automatic uploading capabilities on its X-ray machines.

TSA officials also acknowledged that, in addition to the airports discussed above that did not report any TIP data for a year or more at a time, other airports may have reported only partial TIP results data during this same time frame. TSA officials stated that, in the nationwide results data provided to GAO, it would be difficult to ascertain how much data might be missing from individual airports (during the time period covered by our data) since the number and type of machines in use at those airports at any particular point in time could vary.

TSA policy requires TSA officials at airports to report all of their TIP results data, on a monthly basis, to a national database. Further, FSDs must monitor TIP results monthly and require TSOs to attend remedial training if their threat identification rate falls below a target percentage. Standards for Internal Control in the Federal Government states that the information requirements needed to achieve the agency’s objectives should be identified and communicated to management. Specifically, management should obtain relevant data from reliable internal and external sources in a timely manner based on the identified information requirements that allows them to carry out their internal control and other responsibilities. We acknowledge that because the full universe of X-ray machines, and their uploading capabilities, is difficult to determine on a daily basis, it is unlikely that TSA can fully confirm whether all of the TIP data across the nation are being submitted. However, our review of TIP data from fiscal year 2009 through 2014 found that up to 14 percent of airports did not submit any TIP data in one of the years reviewed (2013). Unless TSA takes steps to ensure that all airports submit complete, nationwide TIP data, TSA lacks assurance that the decisions it makes on the content of the TIP image library are fully informed, and also lacks assurances that TSOs are receiving remedial training from the TIP program which has been developed to aid their ability to identify prohibited items. For example, while TSA is working to install automatic uploading capabilities on all X-ray machines, enforcing the requirement for airport officials to manually submit their TIP data would help ensure more complete data by which to assess and address TIP results. In addition, by not ensuring the collection of available TIP data, as required, the effectiveness of any potential further use of TIP testing results to inform TSO training or testing (as described below) programs is limited.

No National Analysis to Inform Screening Efforts

With regard to any potential further use of the TIP results, TSA headquarters officials told us that, to date, they have not systematically used the TIP results data to analyze national trends for purposes of informing future training programs or changes to screening processes or procedures. TSA officials said that they have not used national TIP data in this manner due to the agency’s expectation that TIP is a tool primarily for the benefit of local FSDs to use in monitoring the training needs, and determining areas of focus, for their individual TSOs locally. TSA officials at all 10 airports we contacted stated that their FSDs monitored TIP results and used TIP data to inform their decisions on remedial or other training needs of their TSOs. According to the TSA headquarters official responsible for overseeing the TIP program, TSA formed an Integrated Project Team in fiscal year 2015 specifically tasked with studying, developing, and implementing an effective nationwide strategy and process for using TIP testing to enhance TSOs’ threat detection skills. In developing the planned strategy, this team is examining six focus areas— including the improvement of TIP capabilities for enhancing TSO effectiveness through improved remedial training and updating the TIP image library to be responsive to emerging threats. Since the team is newly formed, it has yet to complete its work. Due to the fact that the bulk of the team’s work is yet to be done, it is unclear how or whether these six focus areas include plans to monitor, on a national basis, trends in the results of TIP testing that could help highlight areas for improvement to future image-based screening tests (such as the Image Mastery Assessment component of APR testing) or TSO training. Standards for Internal Control in the Federal Government states that an agency’s management should perform ongoing monitoring of its internal control system and associated operations, evaluate the results of those monitoring activities, and take corrective actions when warranted to achieve objectives and address risks. By not including analyses of TIP results data in nationwide efforts to inform either TSO training or other image-based testing outside of TIP, TSA is missing an opportunity to utilize this extensive, nationwide TSO performance data for enhancing screening operations in addition to lacking assurance that remedial training is occurring, as required, at all airports.

ASAP Test Results are Unreliable, and TSA Does Not Ensure that Recommendations from Nationwide Analysis of ASAP Scores are Implemented

ASAP Pass Rate Results Are Unreliable

In an effort to assess the quality of ASAP testing conducted by TSA field officials at commercial airports, TSA headquarters officials brought in a contractor in fiscal year 2015 to independently perform ASAP covert testing at 40 airports and thereby verify the validity of the testing results at the airports. The contractor personnel performed the same type of ASAP testing that had previously been performed by local TSA personnel at the airports. The contractor’s initial round of covert testing was completed in October 2015, and TSA has analyzed the results of the contractor’s tests and compared them to ASAP tests performed previously at the 40 airports. In doing this analysis, TSA found differences in the test results for most of the 40 airports when comparing the contractor’s results versus the local TSA testers’ results for the same airports. According to TSA officials, TSOs at these 40 airports performed more poorly in the ASAP tests conducted by the contractor personnel as compared to the prior ASAP testing done by the local TSA personnel—indicating that these prior-year pass rates were likely showing a higher level of performance than was actually the case. Also, according to the officials, these differences in test results have led them to question the extent to which the ASAP tests accurately measure TSO performance.

TSA is in the process of determining root causes for these variances of testing results between the contractor and TSA personnel at the airports. According to TSA officials, initial results from the contractor’s work seem to confirm their prior concerns (before the contractor testing was conducted) that problems exist with successfully maintaining the covert nature of tests at airports. TSA officials explained that these prior concerns were based on the high detection rates at some airports when compared to other airports on the same tests. With respect to the difficulty in maintaining the covert nature of the tests, TSA officials at 7 of 10 of the airports we contacted indicated challenges with obtaining anonymous role players to ensure that the ASAP tests remain covert. For example, TSA officials at one airport we visited reported having to rely on the availability of state and local government employees and U.S. Customs and Border Protection personnel to perform as role players. Another smaller airport we visited reported challenges finding role players among local TSA personnel that the TSOs working the screening lanes would not recognize. As a result, they tend to use new hires, National Guard, Federal Aviation Administration, and Federal Bureau of Investigation personnel. TSA officials stated that proposed changes to the ASAP SOP will provide FSDs greater authority to use role players that they have vetted and accepted responsibility for, beyond state, local, and federal government officials.

In an effort to address concerns stemming from their initial analysis of the contractor’s test results, TSA briefed its FSDs on these results and stated that it expects the FSDs will use this information as input in overseeing their local ASAP testing programs. In addition, TSA has extended the work of the contractor by 6 months in order to do further testing that it can compare to local ASAP test results going forward. TSA stated it will continue to analyze the contractor’s results and compare them against the ongoing results from local ASAP testing overseen by the FSDs to determine if the previously-identified variances in results are continuing. TSA officials stated that the findings of the contractor during the 6-month extension period indicated that the variances previously identified in results for the contractor testing versus the local ASAP testing at the airports have been reduced. TSA headquarters officials attributed the reduction in variance to more frequent and improved communication with the FSDs and those responsible for conducting the local ASAP tests— specifically with regard to the contractor’s test findings and potential corrective actions they should undertake to improve the local ASAP testing programs. TSA headquarters officials added that it is through these measures that they are improving the accountability of the local FSDs and their staff for ensuring the quality and reliability of the local ASAP testing going forward.

TSA officials added that, after the start of the contractor’s work, they had initiated an effort to improve aspects of the ASAP testing program that will include better identification of root causes for ASAP testing failures, which they expect will improve the development of associated corrective actions moving forward. This effort is still ongoing and also includes merging aspects of PACE testing into the ASAP program to help identify instances where a lack of standardization in the application of specific screening SOPs (which PACE testing is designed to measure) may negatively impact the screening process. Regarding TSA’s efforts to better identify root causes of ASAP failures to improve the program, TSA has developed a data collection tool that TSA officials said would support these efforts by gathering critical data from test failures that they will analyze to determine root causes of the failure. According to TSA officials, the tool has recently been developed and field tested and is pending initial roll out.

In addition to the ASAP/PACE merger, program enhancements related to the identification of root causes, and ongoing contractor ASAP testing, TSA officials are adding ASAP headquarters testing to supplement the ASAP testing that will continue to be performed by TSA personnel in the field (referred to as the Field Evaluation Team or FET). TSA stated this new headquarters-based testing effort will be referred to as the Headquarters Evaluation Team (HET) and would be formed from the former PACE evaluation teams. According to TSA, these headquarters- based covert testing teams will perform quality assurance and validation activities for ASAP that are currently being performed by the contract test teams. In addition, TSA expects that the contractor and new headquarters ASAP testing program will provide assurance that the ASAP testing still being conducted by TSA personnel at the airports is accurate. However, field ASAP testing will still account for the majority of TSA’s ASAP covert tests. TSA officials stated they expect once the HET program is initiated, the contractor testing will be discontinued. Also, according to TSA, a newly-developed data collection tool will be used by all of the ASAP testing groups moving forward (i.e., FET, HET, and the contract test teams) to determine the root causes of test failures that will better inform TSA’s corrective actions.

TSA Does Not Follow Up On Implementation of Recommendations at Airports

TSA conducts ASAP testing in 6-month increments and produces a summary report of results across all airports, complete with recommendations, at the end of each 6-month cycle. In these reports, TSA details the analysis it has performed on the nationwide results of the ASAP testing that shows how TSOs have performed in their duties at the various decision points on the passenger and checked baggage screening lanes. This analysis includes failure rates at these various points, reasons for the failures, and related recommendations where appropriate to improve TSO performance. These recommendations may include, among other things, additional training for certain points in the screening process and further testing in certain areas. According to TSA officials, they have recently moved to more frequent weekly and monthly reporting of ASAP results to the field as part of the aforementioned effort to improve communication with FSDs and staff with regard to findings and trends coming from the ASAP testing results—including those results from the ASAP contractor.

TSA headquarters does not require FSDs to implement recommendations from the six-month cycle reports nor does it track whether the recommendations have been implemented, or conversely, reasons for not implementing them. TSA officials stated that the various recommendations cited in the cycle reports are strictly for the consideration of FSDs in the field and implementation is not mandatory. TSA officials also stated that the ASAP cycle reports are intended to analyze nationwide trends in TSO performance and identify causes of potential deficiencies. TSA invests time and resources to produce these reports—which include test results and corrective actions—on a routine basis and disseminates the information to airport FSDs. Given this investment, tracking implementation of the recommendations detailed in those reports, in addition to any recommendations that may be present in the more frequently-implemented weekly or monthly reporting, would help TSA ensure that corrective actions are being taken at airports nationwide to improve TSO performance, which the agency has identified as an area of concern based on the nationwide trend analysis. Moreover, tracking the implementation of its recommendations, including the extent to which identified corrective actions are improving future TSO performance and test results, will help TSA better determine the extent to which its implemented recommendations are leading to improvements in screening operations and appropriately addressing identified root causes for previous test failures.

Standards for Internal Control in the Federal Government requires that internal controls be designed to ensure that ongoing monitoring occurs during the course of normal operations. Specifically, internal controls direct managers to (1) promptly evaluate and resolve findings from audits and other reviews, including those showing deficiencies and recommendations reported by auditors and others who evaluate agencies’ operations; (2) determine proper actions in response to findings and recommendations from audits and reviews; and (3) complete, within established time frames, all actions that correct or otherwise resolve the matters brought to management’s attention. We recognize the efforts TSA has recently initiated to improve the accuracy and reliability of ASAP testing. However, without the assurance that recommendations for corrective actions based on the root causes identified in ASAP testing will be fully implemented—where appropriate—nationwide, TSA will be limited in its ability to take full advantage of any findings from the program.

Conclusions

Training TSOs and obtaining an accurate understanding of their effectiveness in detecting prohibited items on passengers, and in their baggage, can have a critical impact on the security of millions of air travelers each year. TSA has put an extensive program in place to train its TSOs to perform these critical screening functions and responded to recent covert test findings of the DHS OIG by implementing a retraining program for all its screening officers to address issues identified in the testing. TSA has also begun implementing a plan to expand evaluations of its TSO training efforts in order to better inform future management decisions. In addition to its training and evaluation efforts, TSA conducts wide-ranging covert testing and annual certification testing of its TSOs. While we commend TSA’s recent efforts to re-examine its testing programs, such as steps to improve the accuracy and reliability of ASAP testing, the agency could further enhance its testing programs to more accurately gauge the true level of TSO performance and ensure continuing improvement in screening operations. For example, enforcing its requirement that all airports submit TIP results data would help TSA continually improve the test. Further, the agency could use these data on a nationwide level to inform and potentially improve training of TSOs in screening passenger carry-on baggage for prohibited items. In addition, given that TSA uses ASAP covert testing results to assess whether TSOs follow proper screening procedures and successfully detect prohibited items, ensuring that any recommendations stemming from the ASAP testing failures are tracked and implemented, where appropriate, would further support the program’s objective to improve the performance and quality of security screening.

Recommendations for Executive Action

To improve TSA’s ability to take full advantage of testing results to inform and potentially improve screening operations, we recommend that the Secretary of the Department of Homeland Security direct the Administrator of TSA to take the following three actions:

Ensure that TSA officials at individual airports submit complete TIP results to the TSA national database as required, including manually submitting data when automated uploading is not available.

Conduct analysis of national TIP data for trends that could inform training needs and improve future training and TSO performance assessments.

Track implementation by airports of ASAP recommendations to ensure that corrective actions identified through ASAP testing are being applied.

Agency Comments and Our Evaluation

We provided a draft of the sensitive version of this report to DHS for their review and comment. DHS provided written comments, which are noted below and reproduced in full in appendix II, and technical comments, which we incorporated as appropriate.

DHS concurred with all three recommendations in the report and described actions underway or planned to address them. With regard to the first recommendation that TSA ensure TIP data is submitted to the TSA national database as required, DHS concurred and stated that TSA is working to establish a tracking system that will automatically identify and highlight specific airports that may be missing from the database. The automated system will allow TSA to establish an internal webpage that will automatically generate a list of airports that have not submitted TIP data as required, and which managers will be able to use to follow-up with Federal Security Directors to ensure TIP data is submitted. The agency stated that the automated process is dependent on the development of an information technology (IT) tool which they anticipate will be piloted by May 31, 2017. In the interim, while this IT tool is being developed, TSA officials will monitor compliance with TIP reporting requirements and follow up with those airports missing TIP data, including identifying reasons for the airport’s non-compliance. TSA is also drafting a revised TIP Operations Directive that is intended to provide further guidance and direction to the field on TIP requirements. TSA estimates they will complete these actions to address the first recommendation by September 30, 2016.

With regard to the second recommendation to conduct analysis of national TIP data for trends that could inform training needs and improve future TSO performance, DHS concurred and detailed the following actions to address this recommendation:

TSA’s Office of Training and Development (OTD) has begun to update TIP remediation requirements and work with airports that have achieved the highest TIP scores to identify any best practices that could be shared with other airports.

OTD plans to work with airports that struggle with TIP to identify information about their oversight and remediation program with the goal of using the highest and lowest scoring airports to assess the effect of oversight and remediation on performance. TSA plans to analyze data across the network to determine what remediation training best supports improvements in TIP scores.

TSA is developing a process to analyze specific data connected to threat categories of TIP images which will allow officials to identify the specific types of threats that are presenting challenges to the workforce. OTD will then be able to identify what additional training should be developed to improve performance for that particular threat category.

TSA plans to assess TIP training and assessments over the next 12 months to determine if performance improvement has been realized, and if so, what contributed to the improvement.

OTD is working with a contractor to design a report that is intended to capture officer performance results connected to specific types of TIP images to better drive training content and improve performance.

TSA’s Office of Security Capabilities is working with both OTD and the Office of Security Operations (OSO) to capture TIP data for the development of threat categories to assess individual TSO’s performance and asking TSA’s Office of Acquisitions for a contract modification that will provide for more frequent report updates.

TSA estimates they will complete these actions to address the second recommendation by May 31, 2017.

With regard to the third recommendation to track implementation by airports of ASAP recommendations to ensure that corrective actions are being applied, DHS concurred and stated that TSA has taken actions to formalize ASAP reporting. For example, TSA has reported developing a standard format for Corrective Action Plans, which are submitted and implemented after an ASAP failure. This should help TSA track corrective actions and their effectiveness in addressing findings from ASAP tests. Further, TSA plans to conduct reassessments within 30-60 days after a Corrective Action Plan has been submitted to ensure corrective actions have been implemented. TSA also reported that the standard format for CAPs deliberately maps corrective actions to their identified issues. According to TSA, as of August 2016, OSO has conducted more than 55 post-Headquarters Evaluation Team testing calls and more than 50 effectiveness calls to review CAPs. OSO has extracted common themes from high performing airports and distributed this “best practice” information to all its regional directors and federal security directors. TSA also stated that OSO is reassessing those previously-tested airports to ensure that corrective actions are implemented and detection performance is improving at or above the national average. These efforts by TSA to ensure that corrective actions identified through ASAP testing are being applied, if continued in future testing cycles, should address the intent of this recommendation. These completed actions for the third recommendation along with the planned actions for the first and second recommendations, if fully implemented, should address the intent of the three recommendations contained in this report.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, the Attorney General of the United States, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7141 or groverj@gao.gov. Key contributors to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

This report answers the following questions: 1. How does the Transportation Security Administration (TSA) train Transportation Security Officers (TSO), and to what extent does TSA evaluate the training? 2. How does TSA measure the performance of TSOs, and what do the performance data show? 3. To what extent does TSA use TSO performance data to enhance TSO performance?

To address our first objective regarding how TSA trains TSOs and to what extent TSA evaluates the training, we reviewed relevant TSA policies and procedures for training, including management directives and the National Training Plan (NTP), which prescribes the annual training curriculum for TSOs. We also reviewed documentation on training requirements, including those contained in the Aviation and Transportation Security Act, as well as documents on TSA’s training development and completion. We interviewed TSA headquarters officials responsible for developing and monitoring TSO training, including officials from TSA’s Office of Training and Development (OTD), Office of Human Capital (OHC), and the Office of Security Operations (OSO). Further, we interviewed staff from a total of 10 airports—including Federal Security Directors (FSD), transportation security managers, instructors, training managers, TSOs, and other TSA staff, such as explosives experts, to determine how training is carried out in the field and to learn what TSA employees in the field thought about training. Specifically, we conducted site visits to six airports, including three airports in category X, and one airport each in Categories I, II, and III. Further, we conducted phone interviews with officials at one airport each in categories I, II, III, and IV to obtain additional perspectives on how airport officials carry out training requirements locally—particularly at airports with smaller numbers of flights and passenger boardings. We selected the airports to visit in person based on factors such as airport category, geographic proximity to one another, and our analysis of the airports’ TSO performance on annual screening certification tests from calendar years 2009 through 2014. For example, we calculated the average first time pass rates for screeners taking their Annual Proficiency Review (APR) exams for each airport in each calendar year from 2009 to 2014 and sorted the scores by airport risk category. APR assessments are annual certification tests TSOs must pass to remain employed as a screener. We then selected at least one airport from the high, low, and middle of the performance distribution and made sure to cover at least one airport in every risk category.

To assess the extent to which TSA evaluates TSO training, we reviewed TSA documents used for evaluating training courses, including end-of- course surveys administered to learners. Further, we reviewed draft documents on TSA’s training evaluation plan, including a draft management directive and draft standard operating procedures for evaluating training courses. We compared the training evaluation documentation to the Kirkpatrick model for training evaluation, which is the model TSA uses as guidance for its evaluations of TSO training. We also interviewed TSA headquarters officials responsible for evaluating TSO training and for developing and implementing the TSA training evaluation plan. For example, we interviewed TSA officials from OTD, OSO, and OHC to determine the extent to which they evaluated training courses and used this information to refine future training. Further, we interviewed management officials at each of the airports we visited to further understand how, if at all, training at individual airports is evaluated locally.

For our second objective, to determine how TSA measures the performance of TSOs and what the performance data show, we analyzed data from four different performance evaluation programs and we interviewed TSA officials responsible for collecting and analyzing the data. First, we reviewed and analyzed data on APRs, including analyzing APR pass rates from calendar year 2009 (the first year for which data were available) through 2015. For example, we calculated the average first time pass rate for screeners taking the APR assessments for each airport and sorted the results by year, airport category, and by each individual APR assessment. See Table 4 for a description of the APR assessments we analyzed. We then conducted a trend analysis to observe overall APR first-time pass rates over time, and we compared APR first-time pass rates for screeners across airport risk categories to determine whether there were any differences in pass rates across airport categories. In addition, we interviewed officials in charge of the APR testing process, including officials from OSO, OHC, OTD from TSA headquarters, as well as local airport officials in charge of overseeing the tests.

Second, we reviewed Threat Image Projection (TIP) system data from fiscal year 2009 to fiscal year 2014, the last year available at the time of our data request. The TIP system is intended to help TSA measure whether operators correctly identify threat items that are electronically superimposed on the X-ray monitor during the screening of passenger property at the checkpoint. Specifically, we analyzed the average percentage of TIP images correctly identified during screening by screeners at different airport categories over time to determine whether there were differences in average TIP scores between airport categories. Further, we interviewed TSA officials in charge of the TIP image library from the Office of Security Capabilities to understand how TIP data are recorded and collected, and how the TIP images are selected for use.

Third, we reviewed data from TSA’s Presence, Advisement, Communication, and Execution (PACE) testing program, which TSA uses to measure whether TSOs are adhering to standard operating procedures while screening at the passenger checkpoint. We reviewed PACE data from calendar year 2011, the year the program was started, until 2014 and charted PACE scores by airport category across time. We also interviewed appropriate TSA officials regarding the PACE program to understand how the program worked and how the scores were calculated.

Finally, we analyzed data from the Aviation Screening Assessment Program (ASAP), a covert testing program used to evaluate screeners’ ability to properly follow TSA’s standard operating procedures for screening and keep prohibited items from being taken through the checkpoint. We analyzed ASAP data from fiscal years 2013 through 2015 because TSA made adjustments to the ASAP testing program in 2013, and therefore the pre-2013 testing data are not comparable to the 2013 through 2015 data. Results of ASAP testing are classified at the secret level and are not included in this report. Additionally we interviewed TSA officials from OSO responsible for the ASAP program to gain their perspectives on the program. We also interviewed officials responsible for conducting ASAP tests at each of the airports we visited to understand how the tests worked in practice, what happened after a test was passed or failed, and to learn about any challenges officials faced in running the tests.

We assessed the reliability of the APR, TIP, PACE, and ASAP data by (1) interviewing agency officials responsible for maintaining the data about how the data were collected and entered into the respective databases, how the data were used, and what procedures were in place to ensure the data were complete; and (2) testing the data for missing data, duplicates, or entries that otherwise appeared to be unusual. We found the APR and PACE data to be sufficiently reliable to present in this report. However, we found that the TIP data were incomplete for the years we were analyzing and therefore not sufficiently reliable to include in this report.

Specifically, TSA officials in charge of the TIP data stated they were uncertain how complete the TIP data were nationwide at any point in time, but added that it is likely never fully complete. Officials stated that this was due to two reasons. First, when TSA first deployed new X-ray machines between 2009 and 2012, the TIP software was not activated on them due to technical issues. As a result, no TIP data were reported for those machines over this period. Second, the newer X-ray machines coming online are equipped to upload TIP data to TSA headquarters automatically over a network. However, not all machines in the field are equipped to do this, and TSA temporarily stopped implementation of the automatic upload capability on the newer machines in 2015 because of network security concerns. Instead, TSA personnel must manually download the TIP data for these machines on a monthly basis as it does for older machines without this automatic upload capability. As a result, TSA headquarters has not received TIP data from every airport for every month over the time period of our review resulting in the database being incomplete. TSA could not provide us with information on the extent of the missing data and we were not able to determine based on the data provided how many X-ray machines were unaccounted for between 2009 and 2014.

For our third objective, to determine the extent to which TSA uses TSO performance data to enhance screening performance, we reviewed TSA’s processes and actions for using screener testing results to inform its operations and training, and assessed these processes against standards in Standards for Internal Control in the Federal Government. Further, we interviewed program officials from several offices at TSA headquarters about how they analyze performance data such as APR, TIP, and PACE data, and how, if at all, they use the results to adjust training or take any other actions. Similarly, we interviewed officials from each of the airports we visited about how the collected, reported, monitored, and used the performance data they collected.

We conducted this performance audit from February 2015 to September 2016, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Homeland Security

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Jennifer Grover (202) 512-7141 or groverj@gao.gov.

Acknowledgments

In addition to the contact named above, Christopher E. Ferencik, Assistant Director; Mike Harmond, Analyst in Charge; and Brendan Kretzschmar made key contributions to this report. Also contributing to the report were, Eric D. Hauswirth, Susan Hsu, Thomas F. Lombardi, Heidi Nielson, Ying Long, Amanda Miller, Ruben Montes de Oca, Dae Park, and Christine San.