Aviation Security: TSA Updated Screener Training to Address Risks, but Should Enhance Processes to Monitor Compliance
Released: 2020-02-13T00:00:00

Letter

Background TSOs and Agency Roles for Their Training

TSA is the primary federal agency responsible for implementing and overseeing the security of the nation’s civil aviation system and, in general, is responsible for ensuring that all passengers and belongings transported by passenger aircraft to, from, within, or overflying the United States are adequately screened. Over 43,000 TSOs stationed across the nation’s approximately 440 commercial airports are responsible for inspecting individuals and belongings to deter and prevent passengers from bringing prohibited items on board an aircraft or into the airport sterile area.

Within TSA, two offices—T&D and Security Operations—are to work together to manage TSOs and ensure their training is current and relevant. T&D is responsible for developing initial and ongoing training curricula for TSOs based in part on TSA’s standard operating procedures that govern how TSOs screen passengers and baggage. Security Operations is responsible for allocating TSO staff to airports, and scheduling TSO work hours and training availability. Within Security Operations, FSDs are responsible for overseeing security operations at the nation’s commercial airports, many overseeing multiple airports within a specific geographic area. FSDs report to one of three executive directors, who in turn are responsible for annually assessing FSD performance, including oversight of TSO training.

TSO Training Requirements

TSA’s screener training is comprised of a compendium of courses that includes basic training for initial hires, recurrent training, remedial training, and return-to-duty training. The National Training Plan specifies annual training requirements and contains the core curriculum for TSOs, including the classes and hours required for TSOs to complete. In accordance with the Aviation and Transportation Security Act, screeners must complete a minimum of 40 hours of classroom instruction and 60 hours of on-the-job training, and must successfully complete an on-the- job training examination. Until 2016, new TSOs completed these training requirements at or near their home airports through the New Hire Training Program. In January 2016, TSA centralized this training under the TSO Basic Training Program at the TSA Academy in Glynco, Georgia. Further, in August 2018, TSA launched the first phases of TSO Career Progression, in which new hire screeners receive local training and gain experience in a limited number of screening functions before advancing to the next stage of training at the TSA Academy, roughly around the four-month mark.

In 2015, in response to the DHS Office of Inspector General covert test findings that highlighted areas of concern in the passenger screening process, TSA implemented a TSO re-training effort, beginning with a nationwide training called “Mission Essentials—Threat Mitigation.” According to TSA, this training provided the opportunity for the TSO workforce to become familiar with the threat information that underlies TSA’s use of checkpoint technologies and operational procedures to mitigate risks.

Federal Training Evaluation Requirements and Training Evaluation Models

In 2009, OPM developed and published regulations that require agencies to evaluate training programs annually. According to the regulations, these training evaluations are to help agencies determine how well such plans and programs contribute to mission accomplishment and meet organizational performance goals. One commonly accepted training evaluation model, endorsed by OPM and commonly used in the federal government to evaluate training, is known as the Kirkpatrick model. The Kirkpatrick model consists of a four-level approach for soliciting feedback from training course participants and evaluating the impact the training had on individual development, among other things.

The following describes what each level within the Kirkpatrick model is to accomplish:

Level 1: The first level measures the training participants’ reaction to, and satisfaction with, the training program. A level 1 evaluation could take the form of a course survey that a participant fills out immediately after completing the training.

Level 2: The second level measures the extent to which learning has occurred because of the training effort. A level 2 evaluation could take the form of a written exam that a participant takes during the course.

Level 3: The third level measures how training affects changes in behavior on the job. Such an evaluation could take the form of a survey sent to participants several months after they have completed the training to follow up on the impact of the training on the job.

Level 4: The fourth level measures the impact of the training program on the agency’s mission or organizational results. Such an evaluation could take the form of comparing operational data before, and after, a training modification.

TSA Revised Screener Training to Address Risks Identified through Covert Tests and Emerging Threats

Since 2015, TSA’s T&D has developed and updated TSO training programs in response to findings from covert tests and reporting on emerging threats that identified risks to aviation security. T&D uses an online database to track results from covert tests and reporting on emerging threats, and any changes to training that T&D makes as a result. According to T&D data from May 2015 through June 2019, T&D officials reviewed 62 risks that warranted a review for a potential change to training, and 56 of the risks led officials to make training changes across its TSO curriculum. Overall, T&D made changes affecting 40 different training courses.

Based on our review of TSO training curriculum from May 2015 through June 2019, we found that changes T&D made to its TSO training took many forms. In some cases, T&D changed training to place additional emphasis on a certain aspect of a current standard operating procedure or provide context on the importance of following it. For example, in 2019, T&D updated its instructor-led course—”Mission Essentials: Resolution Tools and Procedures”—to address covert tests where TSOs failed to detect simulated explosive devices hidden in bags or concealed on individuals at checkpoints. The training included a review of methods terrorists may use to plan and carry out attacks in order to emphasize the importance of following the standard operating procedure. The updated training also included leading practices for searching belongings and a discussion of issues that may affect a TSO’s ability to detect threat items hidden in belongings or on individuals. In fiscal year 2019, T&D also updated instructor-led courses on its explosives detection system for checked baggage to respond to covert test findings that TSOs failed to detect certain simulated explosive devices. The updated training included images of simulated explosives hidden in checked bags that replicated scenarios similar to the failed covert tests.

In other cases, T&D developed TSO training in response to new or updated standard operating procedures for using technologies. T&D officials said that for this type of TSO training, they wait until TSA’s Requirements, Capabilities, and Analysis office updates or establishes new standard operating procedures for using new technologies and then develops training based on the revisions. For example, T&D developed TSO training to cover the differences between a prior and updated version of the standard operating procedure for screening passengers and belongings at security checkpoints. T&D included curriculum to cover the major changes in the standard operating procedure and incorporated additional training to address a covert test in which TSOs failed to detect a simulated explosive device at a screening checkpoint. T&D also developed training for TSOs who check passenger IDs and travel documents. The training focused on updates to the standard operating procedure and included procedures specific to the 2005 REAL ID Act, which TSA will fully implement in 2020. Additionally, T&D incorporated this new training to address covert tests that had found issues with identifying false or fraudulent travel documents.

In addition to updating or developing new training involving instructor-led courses, TSA responded to identified risks by developing or updating job aids or briefings for TSOs. For example, TSA developed the “It’s Not the Container” briefing in 2017 to address risks highlighted by an attempted attack in Australia and included tactics used to conceal explosives in benign items. The briefing provided best practices for using screening technologies to identify concealed explosives, which aligns with current standard operating procedures. T&D also developed the “Electronics vs. Electrical Devices Job Aid”—covering how TSOs should handle the devices at checkpoints—which instructors circulated during classroom training and provided to TSOs at the screening checkpoints.

TSA Uses Established Models for Updating and Evaluating TSO Training and Has Followed Leading Practices

TSA uses established models and processes for updating and evaluating TSO training, and these processes follow leading practices for training and evaluation development. TSA updates its trainings using a training development process that can be segmented into five broad, interrelated elements, and is typically referred to as the ADDIE model. The elements include (1) analysis, (2) design, (3) development, (4) implementation, and (5) evaluation. In our prior work, we have found that these five elements of the ADDIE model help to produce a strategic approach to federal agencies’ training and development efforts. See figure 1 for how T&D aligns its training development process with the ADDIE model.

T&D’s guidance and our prior work on federal agency training development identify various leading practice attributes for developing training. Such attributes include that the training development process: (1) is formal and based on industry recognized standards; (2) provides the ability to update training based on changing conditions and, if necessary, quickly; (3) includes mechanisms to ensure programs provide training that addresses identified needs; (4) ensures measures of effectiveness are included in training program; (5) prevents duplication of effort and allows for consistent message; (6) allows for stakeholder feedback; (7) provides for continuous evaluation of effort; and (8) includes mechanisms to ensure training programs are evaluated.

We found that T&D’s training development process incorporates all of the identified leading practice attributes, as shown in table 1.

Two examples of TSA’s implementation of selected leading practice attributes are that T&D (1) has methods for updating training quickly, if needed, and (2) has mechanisms to ensure TSO training is evaluated. Specifically: Methods to quickly update training. In alignment with the leading practice that agencies should have a process to enable quick updates to training to respond to changing conditions, T&D has alternative processes to develop and deliver training to TSOs faster than the approximately 6 months its standard process takes to develop or revise training. For example, in 2018, T&D formalized a set of alternative processes to rapidly develop and deliver training to TSOs. One such alternative is for T&D to use its Rapid Response process, which allows for a response time to the field of 72 hours. Additional options include the Rapid Update/Revision or Rapid Development (Priority Training) processes to allow for a new training to be issued in approximately 30 days. T&D officials said that the rapid development processes are used when an issue, such as an emerging threat, requires a response in days or weeks. T&D’s guidance outlines situations when these processes are appropriate for use and provides checklists to help T&D personnel follow key steps.

Mechanisms to help ensure evaluations of training effectiveness. T&D has mechanisms for ensuring it evaluates the effectiveness of its TSO training programs. In particular, T&D uses the Kirkpatrick model to evaluate its training and, according to its policy, all of its courses are to be evaluated at Level 1 of the model, which measures training participants’ reaction to, and satisfaction with, the training. T&D is also to plan course evaluations for each training during the curriculum development process, determine the formal review cycle, and include it in the curriculum development paperwork. According to its policy, T&D must complete a curriculum review at least once every 5 years, but may do so at shorter intervals. During the curriculum review, T&D examines the training to confirm the content is valid with respect to the applicable listing of tasks and competencies, current law, policy, procedures, and equipment. As a part of this process, T&D assesses participant evaluations to determine whether changes to TSO training are needed.

As of October 2019, T&D’s efforts to evaluate new or updated TSO training made from May 2015 through June 2019 are in line with its policy. For example, T&D officials said they updated participant evaluations for TSO training they changed during this time period to address risks identified by covert testing and reports on emerging threats. These officials told us that they had not yet formally analyzed the results of the evaluations. This progress is in line with T&D policy, which requires a review of each training every 5 years. We verified this by obtaining evaluations T&D collected for the six selected sample courses we reviewed. T&D provided us level 1 survey responses it had collected that measure training participants’ reaction to, and satisfaction with, the training programs for four of the courses. T&D implemented the four courses from calendar years 2015 to 2019. Based on those dates and T&D policy, T&D should complete curriculum reviews for the courses between 2020 and 2024.

TSA Monitors Training Compliance, but Its Process Does Not Look for Trends across Fiscal Years and Is Not Fully Documented

TSA relies on a database that both field and headquarters staff use to monitor TSO training compliance. According to TSA policy, TSA documents and maintains the training status of all TSOs across approximately 440 commercial airports through its Online Learning Center database. Within the database, TSA records training completion in three ways: 1. TSOs self-certify they completed the training activity, such as reading 2. A training staff member at a commercial airport will record training completion on behalf of a TSO for instructor-led courses and on-the- job training; 3. The database automatically records completion for training actions, such as online training.

After recording training completion, the database calculates the percentage of TSOs at a given airport who are on pace for completing their required annual training. According to TSA guidance, the agency has set its annual TSO target compliance rate at 90 percent per commercial airport.

While TSA has guidance outlining roles and responsibilities for training oversight at a high level, TSA headquarters and field officials told us their processes for monitoring training compliance—including analyzing training compliance data, reporting their results, and taking action to address the results—were not documented. Below are descriptions of these roles and responsibilities at the field and headquarters levels, based on what officials from each level told us.

TSA personnel in the field have various responsibilities for overseeing training compliance:

FSDs. FSDs, who oversee operations at one or more airports, have the primary responsibility for ensuring that TSOs within the airports they oversee have fulfilled their training requirements. FSDs are assessed on training compliance among TSOs at their respective airports during their annual performance reviews. All seven FSDs we interviewed said they use the Online Learning Center database to verify that TSOs are on track for meeting their training requirements. Further, these FSDs said they meet regularly with their on-site training staff to discuss how training is going and whether TSOs are at risk of not meeting their training requirements.

Executive Directors. Executive Directors oversee the FSDs who work within their respective portfolios and discuss training compliance with the FSDs during their annual performance review. To monitor FSDs’ efforts, Executive Directors also review data from TSA’s Online Learning Center database on TSO training compliance for airports within their area of responsibility. According to an Executive Director we spoke with, if an Executive Director notices that TSO training compliance rates for an airport whose FSD they oversee are lower than the 90 percent compliance target, he or she may reach out to the FSD to obtain information on the causes and discuss an action plan to improve training compliance.

TSA personnel at headquarters also have various responsibilities for overseeing training compliance:

T&D. T&D officials said that on a monthly basis they analyze TSO training compliance data from TSA’s Online Learning Center database to identify how TSOs nationwide are meeting requirements and whether there may be trends that indicate a need for changes to training during the fiscal year. For example, officials told us that in fiscal year 2019 they noticed that airports were generally behind in meeting annual training requirements and determined this was due to the effects of the federal government shutdown. In response, they stated they adjusted the duration of some training courses to shorten the amount of time it would take TSOs to complete the training within the remainder of the fiscal year.

Security Operations. Security Operations tracks individual airport progress toward meeting TSA’s annual 90 percent compliance target. Security Operations officials said they receive and review monthly training compliance reports from T&D. They are responsible for analyzing the data to monitor whether airports are on pace toward meeting the annual TSO training compliance target. For example, TSA has set the required training completion pace goal at 8.3 percent per month for each commercial airport—-so that by maintaining the pace, by the end of the fiscal year, TSOs at each airport will have completed their required annual training. Officials told us that if they identify instances where an airport’s overall TSO training compliance rate for a given month is below this goal during the course of a fiscal year, they will reach out to the FSD responsible. They will provide the FSD a point of contact at a comparable airport with higher compliance rates to share best practices for addressing the issue.

While TSA headquarters officials from Security Operations and T&D are responsible for analyzing and addressing TSO training compliance, they focus on monthly airport progress toward the 90 percent TSO training target, rather than annual changes in compliance rates. In particular, they do not look back at prior year airport compliance data to assess whether airports did not meet the compliance target across fiscal years, and whether they require corrective action at the headquarters level.

However, we reviewed annual TSO training compliance data across fiscal years for each of the 435 commercial airports that reported data from fiscal years 2016 through 2018. We found that while all airports met TSA’s 90 percent training compliance target in fiscal years 2016 and 2017, the compliance rates for five airports dropped well below 90 percent in 2018. These five airports’ TSO compliance rates dropped 15 to 26 percentage points from their reported compliance rate in 2017.

T&D and Security Operations headquarters officials said they were not aware that five airports had not met TSA’s TSO training compliance target in fiscal year 2018, nor the causes for it. Headquarters officials said that they did not identify this development because their focus is on monthly nationwide trends, rather than instances of noncompliance at individual airports across fiscal years, which field officials would be responsible for addressing. However, unlike headquarters officials, field officials do not have the visibility to identify if or when such noncompliance may be occurring across other commercial airports; and whether it may indicate a broader issue. For example, the five airports whose TSO compliance rates dropped significantly between fiscal years 2017 and 2018 varied by size and location. As a result, FSDs and Executive Directors would generally not have been aware that other airports experienced noncompliance or been in a position to determine whether the noncompliance was due to related reasons. Based on TSA’s process, TSA headquarters officials from T&D and Security Operations are best positioned to identify training compliance trends and their causes when they occur, as they have visibility into training compliance data across the agency in a way that field officials do not.

Headquarters officials from T&D and Security Operations told us the field- level processes for overseeing training compliance are not documented because TSA has intentionally given field officials the flexibility to manage TSO workload and training to meet the individual needs of their airports. They said they did not document their processes at the headquarters level because they already understood what to do and were not required to document the analysis results. However, the headquarters officials said there may be a benefit to documenting the headquarters process to ensure consistency in how they carry out the process in the event of attrition.

Standards for Internal Control in the Federal Government calls for agencies to develop and maintain documentation of their internal control system. This documentation allows management to retain organizational knowledge and communicate that knowledge to external parties. This documentation of controls is also evidence that controls are identified, can be communicated to those responsible for their performance, and can be monitored and evaluated by the entity. Moreover, internal control standards state that internal control monitoring should generally be designed to ensure that ongoing monitoring occurs in the course of normal operations to ensure that known weaknesses are resolved.

By documenting its headquarters process for monitoring TSO training compliance—including its process for analyzing monthly training compliance data, the results of its analyses, and actions taken in response—TSA could better ensure its headquarters staff are aware of their responsibilities for overseeing TSO training compliance and consistently carry these responsibilities out as staff change over time. Additionally, by monitoring for instances of TSO noncompliance at individual airports across fiscal years in its analysis of training compliance data, TSA headquarters would be better positioned to determine whether they constitute a trend warranting corrective action at the headquarters level.

Conclusions

TSOs’ ability to perform their duties effectively in screening passengers and their belongings is crucial to the security of the nation’s aviation system. While TSA has made updates to its TSO training programs to address risks identified in covert testing, additional actions could improve its processes for monitoring TSO training compliance so that the agency can identify and address any potential training issues. In particular, by documenting its process for monitoring TSO training compliance— including those for analyzing monthly training compliance data, reporting the results of its monitoring efforts, and taking action to address potential issues—TSA could help ensure that all of the various officials responsible for monitoring training compliance, including new staff over time, understand the process and can consistently implement it. Further, by monitoring for instances of airport TSO non-compliance across fiscal years in its analysis of training compliance data, TSA would be better positioned to ensure that it is aware of potential trends so it may determine whether corrective action at the headquarters level is warranted.

Recommendations for Executive Action

We are making the following two recommendations to TSA: The TSA Administrator should direct T&D and Security Operations to document their processes for monitoring TSO training compliance— including those for analyzing training compliance data, reporting the results from their analysis, and actions taken to address the results. (Recommendation 1)

The TSA Administrator should direct T&D and Security Operations to monitor for instances of TSO non-compliance by individual commercial airports across fiscal years that could potentially warrant corrective action at the headquarters level. (Recommendation 2)

Agency Comments

We provided a draft of our report to DHS for review and comment. In its comments, reproduced in appendix I, DHS concurred with both of our recommendations. DHS also provided technical comments, which we incorporated as appropriate.

With respect to our first recommendation that TSA document its process for monitoring TSO training compliance, DHS stated that, among other things, Security Operations will collaborate with T&D to develop and maintain an internal control mechanism that will document responsibilities at the field and headquarters level for monitoring TSO training completion compliance, and actions taken to address the results.

With respect to our second recommendation that TSA monitor for instances of TSO noncompliance by individual commercial airports across fiscal years, DHS stated that T&D and Security Operations will begin monitoring trends in non-compliance at individual airports and for specific courses. Further, T&D has developed an internal website to share its findings with Security Operations through monthly compliance reports.

We are sending this report to the appropriate congressional committees and to the acting Secretary of Homeland Security. In addition, this report is available at no charge on the GAO website at http://gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-8777 or russellw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II.

Appendix I: Comments from the Department of Homeland Security

Appendix II: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Jason Berman (Assistant Director), Julia Vieweg (Analyst-in-Charge), Benjamin Crossley, Elizabeth Dretsch, Michael Dworman, Eric Hauswirth, Susan Hsu, Tom Lombardi, and Heidi Nielson made key contributions to this report.