Aviation Security: Basic Training Program for Transportation Security Officers Would Benefit from Performance Goals and Measures
Released: 2018-07-26T00:00:00

Letter

Background

After the terrorist attacks of September 11, 2001, Congress passed and the President signed the Aviation and Transportation Security Act into law on November 19, 2001, with the primary goal of strengthening the security of the nation’s civil aviation system. The act established TSA as the agency with responsibility for securing all modes of transportation, including civil aviation. As part of this responsibility, TSA performs or oversees security operations at the nation’s nearly 440 commercial airports, including managing passenger and checked baggage screening operations.

TSOs inspect individuals and property to deter and prevent passengers from bringing prohibited items on board an aircraft or into the airport sterile area—in general, an area of an airport to which access is controlled through the screening of persons and property. While working at an airport checkpoint as shown in figure 1, TSOs perform a variety of tasks, which include:

Travel document verification: a TSO checks passengers’ identification against the boarding pass and the individual presenting the identification.

Divestiture: a TSO assists passengers by informing them what items need to be placed on the x-ray conveyor belt.

X-ray interpretation: TSOs screen passengers’ carry-on baggage and personal property by interpreting x-ray images to identify any prohibited items.

Advanced imaging technology operations: Passengers are screened via advanced imaging technology (often referred to as body scanners), which identifies areas where they may be concealing prohibited items.

Walk-through metal detector operation: a TSO operates the walk- through metal detector.

Physical searches: Passengers can opt to be screened through a physical search, or TSOs may perform a physical search to resolve an alarm triggered by the AIT system or the walk-through metal detector, among other reasons.

Explosive trace detection and manual searches of property: TSOs use an explosives trace detection system by swabbing carry-on baggage and testing the sample for explosive residue or vapors. This test is usually performed in conjunction with a manual search of the carry-on baggage.

Exit lane monitoring: a TSO watches the lane through which passengers exit the sterile area to ensure that no one enters the sterile area through that passage.

Within TSA, two offices work together to manage TSOs and ensure their training is current and relevant. OSO is responsible for allocating TSO staff to airports, scheduling TSO work hours and training availability, and developing SOPs that govern how TSOs screen passengers and baggage. OTD is responsible for developing initial and ongoing training curricula for TSOs based in part on SOPs. Within OTD, a dedicated team is located at the Academy to manage updates at TSO Basic Training.

In accordance with the Aviation and Transportation Security Act, screeners must complete a minimum of 40 hours of classroom instruction, 60 hours of on-the-job training, and successfully complete an on-the-job training examination. Until 2016, new TSOs completed these training requirements at or near their home airports through the New Hire Training Program (NHTP). Since TSA centralized the TSO Basic Training program in January 2016, TSOs fulfill these training requirements through classroom training at the Academy as well as training at their home airports prior to the Academy and on-the-job training after completion of TSO Basic Training. During the 2 weeks spent at the Academy, TSOs receive 80 hours of training on standard operating procedures, threat detection, and the use of screening equipment. Prior to attending TSO Basic Training, new TSOs complete computer-based prerequisite training and may shadow experienced TSOs at a checkpoint. TSO Basic Training allows for participants to be trained at a dedicated facility with more hands-on training than was possible for NHTP (see Appendix I for a comparison of the two programs).

As shown in table 1, of the $53 million obligated from January 2016 through March 2018, TSA obligated $18.2 million for procurement and development of the modular buildings on the FLETC campus used for TSA training, as well as associated hardware and set-up obligations such as audio/video equipment and fully operational simulated checkpoints. TSA obligated an additional $12 million in fiscal year 2016 and $13.7 million in fiscal year 2017 for the delivery of TSO Basic Training, including associated student travel and related equipment. TSA officials told us that due to continuing budget resolutions that funded the government between October 2017 and March 2018, TSA was not able to fully fund the interagency contract between TSA and FLETC to support the TSO Basic Training course in fiscal year 2018 at the beginning of the year. For this reason, TSA does not yet have 2018 training obligations available for reporting through its accounting system. However, based on the average cost per student in fiscal year 2017 of about $2,300 to attend TSO Basic Training, TSA estimates total training obligations of approximately $9.1 million in the first half of fiscal year 2018.

TSA Established the TSO Basic Training Program at the Academy to Obtain Benefits from Centralized Training

According to the business case for TSO Basic Training and TSA officials, implementation of the TSO Basic Training program at the Academy was anticipated to provide a number of potential benefits. The anticipated benefits identified generally align under two distinct categories: (1) efficiencies and improvements obtained through the centralized delivery of training, and (2) enhanced professionalism and “esprit de corps” obtained through bringing newly hired screeners together for centralized training. Collectively, these benefits were also envisioned by TSA headquarters officials to have a positive impact on screening effectiveness and public perception of the TSA workforce.

Based on several analyses of training delivery options that TSA has conducted since 2008, TSA determined that a centralized training academy would have a number of potential benefits relative to the decentralized training previously administered at field airports through NHTP. Among the potential efficiencies and improvements cited by TSA are: Increased consistency and standardization. According to TSA documents and OTD headquarters officials, centralized training provides a standardized curriculum that serves as a foundation for the skills, knowledge, and equipment used across an array of different airport environments. The TSA business case and other supporting analyses note that such an approach offers greater consistency of training delivery and a better mechanism for developing, delivering, and evaluating course content.

Equipment availability and expanded course content. TSO Basic Training includes a full suite of dedicated checkpoint equipment and x-ray image simulators for students to practice learned skills, eliminating the challenge of finding available equipment and training times in a busy airport environment (see figure 2). Officials told us that being more familiar with the screening equipment increases TSOs’ readiness for on-the-job training when they return to their home airports. Initial test results also indicate that participants trained at the Academy receive higher pass rates on end-of course assessments of x-ray image interpretation skills than those who received their initial training at their home airports. Specifically, according to TSA data, of the 5,877 test-takers who received training at TSO Basic Training in 2016, 91.5 percent passed the Image Interpretation Test on their first attempt. In contrast, 83.2 percent of the 1,458 test-takers who received training at local airports in 2016 passed the test on their first attempt. In addition, the Academy curriculum incorporates new learning opportunities, including a live demonstration of improvised explosive devices and an active shooter drill, both of which would be difficult to reproduce within the airport environment, according to TSA officials.

Dedicated faculty and instructor development. TSO Basic Training offers a dedicated faculty and support staff focused exclusively on training TSOs. According to TSA officials, before TSO Basic Training, training at individual airports was often conducted by TSOs for whom instruction was a collateral duty, whereas instructors at the Academy have full-time training responsibilities and enhanced opportunities to learn from each other, increase their professional training skills, and provide feedback on the delivery of course curriculum.

Centralized facility and shared logistics. By locating the TSA Academy at FLETC, TSA is able to take advantage of the services and logistical support that FLETC provides. Specifically, FLETC services and logistics include accommodations, meals, and transportation, thereby reducing the administrative demands on TSA personnel and allowing students a focused and efficient training experience. Additional efficiencies cited by TSA officials include lower overall costs for office space, janitorial services, and other operational costs because such costs are shared by the 96 agencies that use FLETC. According to TSA officials, conducting training at FLETC can also help TSA accommodate hiring surges and better augment future training, if needed. For example, TSA officials reported that the facility has surge capacity from its current capacity of 240 students up to 300 new students if sufficient instructors are available.

According to TSA documents and training officials, another key benefit of centralized training is the opportunity to enhance professionalism and help foster camaraderie and esprit de corps. TSA anticipates that centralized, standardized training will not only provide trainees with an increased focus on the TSA mission and operational environment, but can serve to instill a common culture and sense of belonging among the broader community of TSOs nationwide. In its business case, TSA notes that centralized training of new recruits is a common model employed by the armed forces and other federal law enforcement agencies within DHS, such as U.S. Customs and Border Protection and the U.S. Coast Guard. According to the business case, by bringing together newly hired TSOs from around the country, TSA also hopes to inspire in its trainees a singular identity and unity of purpose, which previous analyses generally found lacking as part of the decentralized training approach.

The business case also associates such increases in professionalism and esprit de corps with greater employee satisfaction and the potential for reduced attrition. Analysis conducted by TSA in 2017 provides some initial support for positive trends in these areas. For example, results of a 2017 TSA employee engagement survey indicated that respondents who attended TSO Basic Training reported higher scores in categories including Organizational Commitment, Job Satisfaction, and Overall Morale versus respondents who did not attend. TSA also reported a 19 percent reduction in the attrition rate during the first 180 days of being hired for those attending TSO Basic Training at the Academy in 2016 versus those who received their initial training at field airports through the New Hire Training Program.

Factors Considered in Updating the TSO Basic Training Curriculum Include Evolving Security Threats and Input from Course Participants

OTD Uses Information from OSO to Update the TSO Basic Training Curriculum to Address Evolving Security Threats

OTD updates and modifies the TSO Basic Training curriculum based, in part, on regular communications from OSO, the office responsible for developing SOPs for screening operations and managing TSO performance. Officials from both offices told us that OSO provides information to OTD on changes to SOPs as soon as changes are made so they can update the TSO Basic Training curriculum. For instance, in 2017, when OSO began planning major changes to the SOPs, the office gave OTD information about the planned SOP revisions, as well as the airports where the new SOPs would be piloted. In response, OTD modified its curriculum and was able to provide revised training for new TSOs based at airports that were piloting the program, while providing TSOs at all other airports the prior version of training. OTD officials noted that in some cases TSA must quickly update SOPs to reflect imminent threats. According to officials, a plan is in place to make changes to TSO Basic Training curriculum in response to emerging or imminent threats, although such threats have not been experienced since the establishment of TSO Basic Training in 2016.

In addition to changes in SOPs, OSO officials indicated they may also change the timing of when TSOs employed by TSA attend TSO Basic Training. Specifically, officials told us that OSO plans to implement a new model for TSO Basic Training in which TSOs will attend TSO Basic Training 2 to 6 months after they are hired rather than as soon as is practical. According to TSA, the agency is pursuing this change to, among other things, implement a transparent career path for TSOs employed by TSA and to encourage and reward skill development. During the 2 to 6 months prior to attending TSO Basic Training, TSOs will perform checkpoint tasks that require training that can be delivered at the airport as soon as they are hired, such as checking passengers’ travel documents and helping passengers move through the checkpoint. Once TSOs are able to perform these initial tasks, they will attend TSO Basic Training at the Academy, Officials told us they are preparing for the change by modifying the TSO Basic Training curriculum to eliminate subjects that will be covered at the airports and to emphasize skills that more experienced TSOs will need, such as performing physical searches of passengers. TSA plans to implement the revised model beginning in August 2018.

Finally, OTD receives information on TSO performance and uses that information to inform TSO Basic Training curriculum. For example, two TSA offices—OSO and the Office of Inspections—perform regular effectiveness testing of airport checkpoints through covert testing and share the results with OTD. After each covert testing event, each office conducts interviews with TSOs to determine the factors that contributed to their effectiveness at identifying prohibited items. Officials told us that OSO and OTD hold regular meetings to discuss the analyses of covert testing failures and ways in which training curriculum can be modified to address the reasons for the failures, which are then incorporated into the TSO Basic Training curriculum. Office of Inspections officials noted that they participated in the development of the TSO Basic Training curriculum and provide regular reports to OTD on covert testing results.

When Making Updates to TSO Basic Training Curriculum, OTD Considers Feedback from Instructors, Course Participants, and Contractors

OTD gathers input from TSO Basic Training participants, instructors, and contractors on ways to update the curriculum. For instance, TSO Basic Training instructors told us they submit “white paper proposals” to TSO Basic Training course managers detailing their suggested changes to the course. They can also provide feedback and suggestions during “train the trainer” sessions, in which all instructors participate when TSO Basic Training is updated. Instructors told us that all sessions include an opportunity for instructors to provide feedback after reviewing the new curriculum. Officials told us that they take instructors’ feedback into account when implementing new curriculum. For instance, officials told us that at the suggestion of instructors, they added time for discussion at the end of each checkpoint lab to help capture and share lessons learned.

OTD also collects feedback from TSOs who have participated in the course, both at the end of their two weeks at the Academy and several months after their completion of the course. At the end of TSO Basic Training, OTD collects feedback from participants through a survey with both multiple choice and open-ended questions. The survey includes questions on the course curriculum and instructor performance. Officials told us that they regularly review the results of the survey and consider whether it is appropriate to address the feedback by modifying TSO Basic Training. For instance, the most often provided feedback for altering the curriculum was to increase the time in hands-on training using screening equipment in the Academy’s simulated checkpoints. In response, OTD officials told us they added nearly 5 hours of hands-on training to the 80- hour program in addition to the 6 hours that had previously been a part of the curriculum.

In addition to collecting feedback from TSO Basic Training participants and instructors, TSA officials told us that TSA regularly uses a contractor to support the design and development of training courses and to assess existing courses, including TSO Basic Training. In 2016, the contractor conducted an evaluation of the instructional integrity of the TSO Basic Training curriculum. The resulting report made a number of recommendations to improve the curriculum and structure of TSO Basic Training, many of which OTD has implemented. For instance, the contractor recommended that TSO Basic Training include more opportunities for review of the material to reinforce TSOs’ understanding. In response, OTD implemented a review session at the end of the first week of training so TSOs have an opportunity to clarify information presented over the first week.

TSA Has Made Progress in Implementing a Training Evaluation Model but Has Not Established Specific Goals and Performance Measures to Assess TSO Basic Training

TSA has implemented three of the four levels of the Kirkpatrick Model, a training evaluation model that, in part, helps TSA collect feedback from course participants and evaluate the impact on individual development. However, the agency has not developed goals for the program or related performance metrics to demonstrate progress toward goals.

TSA Has Made Progress in Implementing the Kirkpatrick Model to Evaluate Its TSO Basic Training Program

To evaluate the TSO Basic Training program, TSA uses the Kirkpatrick Model, which is a commonly accepted training evaluation model endorsed by the Office of Personnel Management and used throughout the federal government. The Kirkpatrick Model consists of a four-level approach for soliciting feedback from training course participants and evaluating the impact the training had on individual development, among other things. To date, TSA has implemented the first three levels of the model by administering (1) course surveys to participants at the end of the training program; (2) an end-of-course written exam and an x-ray image interpretation test to assess achievement of learning objectives; and (3) course surveys to participants and their supervisors several months after completing training to collect information regarding how the training affected behavior or performance on the job. OTD officials told us that they have not yet implemented Level 4 of the model because they do not believe they have enough data. Table 2 provides a description of what each level within the Kirkpatrick model is to accomplish and TSA’s progress in implementing the levels.

TSA Has Not Yet Developed Goals and Performance Measures for TSO Basic Training

While TSA reported potential benefits of TSO Basic Training in its business case and implemented the Kirkpatrick Model to assess training, it has not yet identified specific goals that the TSO Basic Training program is expected to achieve, nor has it developed performance measures to evaluate progress toward goals. The business case and the Kirkpatrick Model are positive steps and document certain benefits of TSO Basic Training, but without a set of specific training goals and associated performance measures for the program, TSA is not able to fully evaluate the program’s effectiveness and ensure accountability toward results. Such goals are important to help ensure alignment with course objectives and the end-of course examinations administered as part of Level 2 of the Kirkpatrick Model. In addition, without the development of specific goals, it is not possible to determine what types of performance measures should be used to help show progress toward such goals. For example, in its business case, TSA identified improved employee morale as one of the anticipated benefits of TSO Basic Training. However, there are no goals or metrics specifically related to this benefit. If TSA believes improved morale should be something for which TSO Basic Training aims, goals and measures could help them demonstrate the extent to which this benefit is being realized by the training program.

Leading management practices related to training evaluation guidance identifies the importance of agencies developing and using performance measures regularly to ensure accountability and assess progress toward achieving results that are aligned with the agency’s mission and goals. In addition, these practices highlight the importance of agencies having clear goals about what the training or development program is expected to achieve as a precursor to developing such measures. When designed effectively, performance measures help decision makers (1) determine the contributions that training makes to improve results, (2) identify potential gaps in performance, and (3) determine where to focus resources to improve results. In particular, incorporating valid measures of effectiveness into training programs can enable an organization to better ensure that desired changes occur in trainees’ skills, knowledge, and abilities.

According to OTD officials, the TSO Basic Training program was established on an accelerated schedule in late 2015 as one of multiple efforts to improve training delivery and help enhance screener effectiveness. Officials stated that the program is still relatively new and they plan to collect several additional years of data on system-wide screening performance before conducting efforts to further evaluate the impact of the training. They reported that the lack of performance measures is also due to the inherent difficulty of tying specific training initiatives to broader organizational results. Officials told us that once TSOs return to their home airports after TSO Basic Training, they are exposed to additional on-the-job training and differing airport cultures, which make it difficult to isolate the effects of TSO Basic Training. However, senior training officials agreed that establishing applicable goals and performance measures for the TSO Basic Training program would be helpful to support ongoing efforts and better measure program progress.

We recognize that developing metrics to assess the performance of training programs on broad organizational results can be challenging. However, there are additional opportunities to develop program goals and performance measures as part of the training evaluation efforts at the Academy to help ensure that participants can demonstrate proficiency in performing core technical skills before returning to their home airports. We believe that developing goals for a training program does not need to wait for years of data. Goals reflect desired results, connected to an agency’s mission, which a program plans to achieve. In the over 2 years of using TSO Basic Training, TSA has not stated what results the training program is to achieve.

TSOs provide a crucial function to help ensure passenger safety, and it is important to have goals aligned with this mission, as well as associated measures of effectiveness of the training they receive at TSO Basic Training to determine the extent to which they are able to fulfill their important role. As noted by leading management practices for training evaluation, agencies need credible information to demonstrate a training program is contributing to a goal and they can develop such data through a mix of quantitative and qualitative indicators. We found that options for assessing the effectiveness of TSO Basic Training could include measuring TSO performance by leveraging data from end of course examinations, such as the x-ray image interpretation test, and introducing similar additional tests or mechanisms to further evaluate trainees’ knowledge and skills in effective screening procedures. Additional options could include measuring employee morale as indicated by TSOs on their Kirkpatrick Level 1 surveys at the completion of the training program, and comparing these results against applicable program goals for employee morale that TSA could establish related to TSO Basic Training. By identifying annual goals and measures for TSO Basic Training, TSA will also be better positioned to move forward with Level 4 of the Kirkpatrick Model to evaluate the impact of training on broader organizational results. Given that over $50 million has been obligated to set up and operate the TSO Basic Training program to date, it is important that TSA incorporate annual goals and measures into the training program to be better informed when making training decisions and to help hold itself accountable for training results on a regular basis.

Conclusions

TSOs perform a critical role in securing our nation’s commercial aviation system and often represent the most visible face of TSA to the public. For this reason, new hire training is an integral function to ensure that TSOs are obtaining the foundational skills and knowledge to help prepare them to perform their jobs effectively. In 2016, TSA initiated a major change to its training approach for new hires to help ensure a consistent and standardized training experience and promote enhanced camaraderie and esprit de corps. Although TSA has implemented a framework to assess participant reactions to the training and their knowledge of course content, it has not yet established goals for the TSO Basic Training program or measures to gauge effectiveness of the training TSOs receive to determine the extent to which they can fulfill their crucial role in ensuring passenger safety. By taking these steps, TSA will be better positioned to determine if the program is improving trainees’ skills, knowledge, and abilities and whether additional skill development, or other training modifications, may be needed.

Recommendation for Executive Action

We are making one recommendation to the Administrator of TSA. Specifically, the Administrator of TSA should establish specific goals for the TSO Basic Training program and develop performance measures that can be used to assess if the program is achieving desired outcomes and help ensure accountability for training results on a regular basis. (Recommendation 1)

Agency Comments and Our Evaluation

We provided a draft of this report to DHS for review and comment. DHS provided written comments, which are reprinted in appendix II, and technical comments, which we incorporated as appropriate. DHS agreed with our recommendation that TSA establish specific goals for the TSO Basic Training program and develop performance measures that can be used to assess if the program is achieving desired outcomes. In addition, in its written comments DHS outlined steps to address this recommendation.

With regard to performance goals, TSA plans to establish broad goals that include successful screening and improved morale, among others. The stated goals are an appropriate response to our recommendation that TSA develop goals specifically for TSO Basic Training. These actions, if implemented effectively, should address the intent of our recommendation.

With regard to developing performance measures that can be used to assess program outcomes, TSA intends to leverage existing mechanisms through its Kirkpatrick Model evaluations to measure program success. As we noted in the report, implementing the first three levels of the Kirkpatrick Model are positive steps that document certain benefits of TSO Basic Training, but they do not address specific goals or performance measures. Kirkpatrick Model Level 2 evaluations include proficiency exams administered prior to TSOs’ departure from the Academy. Data from these evaluations, in conjunction with specific goals, may provide quantifiable metrics that could inform further refinement of the TSO Basic Training curriculum. However, the surveys being used by TSA for Level 3 of the Kirkpatrick Model do not include metrics that would allow TSA to measure the program’s effectiveness and ensure accountability toward results. Specifically, the surveys do not demonstrate whether TSO Basic Training is reaching goals related to successful screening or improved morale because survey results are influenced by factors outside of the training program. We will continue to monitor TSA’s efforts in this area.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (206) 287-4804 or AndersonN@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Appendix I: Comparison of New Hire Training Program and Transportation Security Officer (TSO) Basic Training

In 2016, the Transportation Security Administration (TSA) established the TSO Basic Training program at the TSA Academy, located at the Federal Law Enforcement Training Centers in Glynco, Georgia. TSO Basic Training allows new TSOs to be trained at a dedicated facility with simulated checkpoints. Previously, TSOs’ initial training was delivered through the New Hire Training Program at or near their home airports, at which they were able to practice using checkpoint equipment only when the equipment was not being used, such as after hours. For further comparison of the two programs, see Table 3.

Appendix II: Comments from the Department of Homeland Security

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgements

In addition to the contact named above, Dawn Locke, Assistant Director; Miriam Hill, Analyst in Charge; and Ryan Lambert made key contributions to this report. Also contributing to the report were Elizabeth Dretsch, Eric Hauswirth, Susan Hsu, Heidi Nielson, and Adam Vogt.