Aviation Security: Screener Training and Performance Measurement Strengthened, but More Work Remains
Released: 2005-05-02T00:00:00

Letter

Background

The performance of passenger and checked baggage screeners in detecting threat objects at the nation’s airports has been a long-standing concern. In 1978, screeners failed to detect 13 percent of the potentially dangerous objects that Federal Aviation Administration (FAA) agents carried through airport screening checkpoints during tests. In 1987, screeners did not detect 20 percent of the objects in similar tests. In tests conducted during the late 1990s, as the testing objects became more realistic, screeners’ abilities to detect dangerous objects declined further. In April 2004, we, along with the DHS Office of the Inspector General (OIG), testified that the performance of screeners continued to be a concern. More recent tests conducted by TSA’s Office of Internal Affairs and Program Review (OIAPR) also identified weaknesses in the ability of screeners to detect threat objects, and separate DHS OIG tests identified comparable screener performance weaknesses. In its July 2004 report, The National Commission on Terrorist Attacks Upon the United States, known widely as the 9/11 Commission, also identified the need to improve screener performance and to better understand the reasons for performance problems.

After the terrorist attacks of September 11, 2001, the President signed the Aviation and Transportation Security Act (ATSA) into law on November 19, 2001, with the primary goal of strengthening the security of the nation’s aviation system. ATSA created TSA as an agency with responsibility for securing all modes of transportation, including aviation. As part of this responsibility, TSA oversees security operations at the nation’s more than 450 commercial airports, including passenger and checked baggage screening operations. Prior to the passage of ATSA, air carriers were responsible for screening passengers and checked baggage, and most used private security firms to perform this function. FAA was responsible for ensuring compliance with screening regulations.

Today, TSA security activities at airports are overseen by FSDs. Each FSD is responsible for overseeing security activities, including passenger and checked baggage screening, at one or more commercial airports. TSA classifies the over 450 commercial airports in the United States into one of five security risk categories (X, I, II, III, and IV) based on various factors, such as the total number of takeoffs and landings annually, the extent to which passengers are screened at the airport, and other special security considerations. In general, category X airports have the largest number of passenger boardings and category IV airports have the smallest. TSA periodically reviews airports in each category and, if appropriate, updates airport categorizations to reflect current operations. Figure 1 shows the number of commercial airports by airport security category as of December 2003.

In addition to establishing TSA and giving it responsibility for passenger and checked baggage screening operations, ATSA set forth specific enhancements to screening operations for TSA to implement, with deadlines for completing many of them. These requirements included assuming responsibility for screeners and screening operations at more than 450 commercial airports by November 19, 2002; establishing a basic screener training program composed of a minimum of 40 hours of classroom instruction and 60 hours of on-the-job training; conducting an annual proficiency review of all screeners; conducting operational testing of screeners; requiring remedial training for any screener who fails an operational test; and screening all checked baggage for explosives using explosives detection systems by December 31, 2002.

Passenger screening is a process by which authorized TSA personnel inspect individuals and property to deter and prevent the carriage of any unauthorized explosive, incendiary, weapon, or other dangerous item aboard an aircraft or into a sterile area. Passenger screeners must inspect individuals for prohibited items at designated screening locations. The four passenger screening functions are: X-ray screening of property, walk-through metal detector screening of individuals, hand-wand or pat-down screening of individuals, and physical search of property and trace detection for explosives.

Checked baggage screening is a process by which authorized security screening personnel inspect checked baggage to deter, detect, and prevent the carriage of any unauthorized explosive, incendiary, or weapon onboard an aircraft. Checked baggage screening is accomplished through the use of explosive detection systems (EDS) or explosive trace detection (ETD) systems, and through the use of alternative means, such as manual searches, K-9 teams, and positive passenger bag match, when EDS and ETD systems are unavailable on a temporary basis. Figure 2 provides an illustration of passenger and checked baggage screening operations.

There are several positions within TSA for employees that perform and directly supervise passenger and checked baggage screening functions. Figure 3 provides a description of these positions.

To prepare screeners to perform screening functions, to keep their skills current, and to address performance deficiencies, TSA provides three categories of required screener training. Table 1 provides a description of the required training.

In September 2003, we reported on our preliminary observations of TSA’s efforts to ensure that screeners were effectively trained and supervised and to measure screener performance. We found that TSA had established and deployed a basic screener training program and required remedial training but had not fully developed or deployed a recurrent training program for screeners or supervisors. We also reported that TSA had collected limited data to measure screener performance. Specifically, TSA had conducted limited covert testing, the Threat Image Projection System was not fully operational, and TSA had not implemented the annual screener proficiency testing required by ATSA. In subsequent products, we reported progress TSA had made in these areas and challenges TSA continued to face in making training available to screeners and in measuring and enhancing screener performance. A summary of our specific findings is included in appendix I.

TSA Has Enhanced and Expanded Training, but Some Screeners Have Encountered Difficulty Accessing and Completing Recurrent Training

TSA has taken a number of actions to enhance the training of screeners and Screening Supervisors but has encountered difficulties in providing access to recurrent training. TSA has enhanced basic training by, among other things, adding a dual-function (passenger and checked baggage) screening course for new employees. Furthermore, in response to the need for frequent and ongoing training, TSA has implemented an Online Learning Center with self-guided training courses available to employees over TSA’s intranet and the Internet and developed and deployed a number of hands-on training tools. Moreover, TSA now requires screeners to participate in 3 hours of recurrent training per week, averaged over each quarter year. TSA has also implemented leadership and technical training programs for Screening Supervisors. However, some FSDs, in response to open-ended survey questions, identified a desire for more training in specific areas, including leadership, communication, and supervision. Further, despite the progress TSA has made in enhancing and expanding screener and supervisory training, TSA has faced challenges in providing access to recurrent training. FSDs reported that insufficient staffing and a lack of high-speed Internet/intranet connectivity at some training facilities have made it difficult to fully utilize these programs and to meet training requirements. TSA has acknowledged that challenges exist in recurrent screener training delivery and is taking steps to address these challenges, including factoring training requirements into workforce planning efforts and distributing training through written materials and CD-ROMs until full Internet/intranet connectivity is achieved. However, TSA does not have a plan for prioritizing and scheduling the deployment of high-speed connectivity to all airport training facilities once funding is available. The absence of such a plan limits TSA’s ability to make prudent decisions about how to move forward with deploying connectivity to all airports to provide screeners access to online training.

TSA Has Enhanced Basic Screener Training

TSA has enhanced its basic screener training program by updating the training to reflect changes to standard operating procedures, deploying a new dual-function (passenger and checked baggage screening) basic training curriculum, and allowing the option of training delivery by local staff. As required by ATSA, TSA established a basic training program for screeners composed of a minimum of 40 hours of classroom instruction and 60 hours of on-the-job training. TSA also updated the initial basic screener training courses at the end of 2003 to incorporate changes to standard operating procedures and directives, which contain detailed information on how to perform TSA-approved screening methods. However, a recent study by the DHS OIG found that while incorporating the standard operating procedures into the curricula was a positive step, a number of screener job tasks were incompletely addressed in or were absent from the basic training courses.

In addition to updates to the training curriculum, in April 2004, TSA developed and implemented a new basic screener training program, dual- function screener training that covers the technical aspects of both passenger and checked baggage screening. Initially, new hire basic training was performed by a contractor and provided a screener with training in either passenger or checked baggage screening functions. A screener could then receive basic training in the other function later, at the discretion of the FSD, but could not be trained in both functions immediately upon hire. The new dual-training program is modular in design. Thus, FSDs can chose whether newly hired screeners will receive instruction in one or both of the screening functions during the initial training. In addition, the individual modules can also be used to provide recurrent training, such as refreshing checked baggage screening skills for a screener who has worked predominately as a passenger screener. TSA officials stated that this new approach provides the optimum training solution based on the specific needs of each airport and reflects the fact that at some airports the FSD does not require all screeners to be fully trained in both passenger and checked baggage screening functions.

Some FSDs, particularly those at smaller airports, have made use of the flexibility offered by the modular design of the new course to train screeners immediately upon hire in both passenger and checked baggage screening functions. Such training up front allows FSDs to use screeners for either the passenger or the checked baggage screening function, immediately upon completion of basic training. Figure 4 shows that 58 percent (3,324) of newly hired screeners trained between April 1, 2004, and September 1, 2004, had completed the dual-function training.

In April 2004, TSA also provided FSDs with the flexibility to deliver basic screener training using local instructors. TSA’s Workforce Performance and Training Office developed basic screener training internally, and initially, contractors delivered all of the basic training. Since then, TSA has provided FSDs with the discretion to provide the training using local TSA employees or to use contractors. The flexibility to use local employees allows FSDs and members of the screener workforce to leverage their first-hand screening knowledge and experience and address situations unique to individual airports. As of December 10, 2004, TSA had trained 1,021 local FSD staff (representing 218 airports) in how to instruct the dual-function screener training course. TSA officials stated that they expect the use of TSA-approved instructors to increase over time.

TSA Has Deployed Recurrent Screener Training and Provided Additional Training Tools, but Some FSDs Identified the Need for More Courses

“Numerous interviews revealed concerns with training curriculum, communication, and coordination issues that directly affect security screening. Unsatisfied with the quantity and breadth of topics, many Training Coordinators have developed supplementary lectures on both security and non-security related topics. These additional lectures…have been very highly received by screeners.”

In October 2003, TSA introduced the Online Learning Center to provide screeners with remote access to self-guided training courses. As of September 14, 2004, TSA had provided access to over 550 training courses via the Online Learning Center and made the system available via the Internet and its intranet. TSA also developed and deployed a number of hands-on training modules and associated training tools for screeners at airports nationwide. These training modules cover topics including hand- wanding and pat-down techniques, physical bag searches, X-ray images, prohibited items, and customer service. Additionally, TSA instituted another module for the Online Learning Center called Threat in the Spotlight, that, based on intelligence TSA receives, provides screeners with the latest in threat information regarding terrorist attempts to get threat objects past screening checkpoints. Appendix III provides a summary of the recurrent training tools TSA has deployed to airports and the modules currently under development.

In December 2003, TSA issued a directive requiring screeners to receive 3 hours of recurrent training per week averaged over a quarter year. One hour is required to be devoted to X-ray image interpretation and the other 2 hours to screening techniques, review of standard operating procedures, or other mandatory administrative training, such as ethics and privacy act training.

In January 2004, TSA provided FSDs with additional tools to facilitate and enhance screener training. Specifically, TSA provided airports with at least one modular bomb set (MBS II) kit—containing components of an improvised explosive device—and one weapons training kit, in part because screeners had consistently told TSA’s OIAPR inspectors that they would like more training with objects similar to ones used in covert testing.

Although TSA has made progress with the implementation of recurrent training, some FSDs identified the need for several additional courses, including courses that address more realistic threats. TSA acknowledged that additional screener training is needed, and officials stated that the agency is in the process of developing new and improved screener training, including additional recurrent training modules (see app. III).

TSA Provides Leadership and Technical Training for Supervisors, but Some FSDs Would Like More Training for Screening Supervisors and Lead Screeners

TSA has arranged for leadership training for screening supervisors through the Department of Agriculture Graduate School and has developed leadership and technical training courses for screening supervisors. However, some FSDs reported the need for more training for Screening Supervisors and Lead Screeners. The quality of Screening Supervisors has been a long-standing concern. In testifying before the 9/11 Commission in May 2003, a former FAA Assistant Administrator for Civil Aviation Security stated that following a series of covert tests at screening checkpoints to determine which were strongest, which were weakest, and why, invariably the checkpoint seemed to be as strong or as weak as the supervisor who was running it. Similarly, TSA’s OIAPR identified a lack of supervisory training as a cause for screener covert testing failures. Further, in a July 2003 internal study of screener performance, TSA identified poor supervision at the screening checkpoints as a cause for screener performance problems. In particular, TSA acknowledged that many Lead Screeners, Screening Supervisors, and Screening Managers did not demonstrate supervisory and management skills (i.e., mentoring, coaching, and positive reinforcement) and provided little or no timely feedback to guide and improve screener performance. In addition, the internal study found that because of poor supervision at the checkpoint, supervisors or peers were not correcting incorrect procedures, optimal performance received little reinforcement, and not enough breaks were provided to screeners. A September 2004 report by the DHS OIG supported these findings, noting that Screening Supervisors and Screening Managers needed to be more attentive in identifying and correcting improper or inadequate screener performance.

TSA recognizes the importance of Screener Supervisors and has established training programs to enhance their performance and effectiveness. In September 2003, we reported that TSA had begun working with the Department of Agriculture Graduate School to tailor the school’s off-the-shelf supervisory course to meet the specific needs of Screening Supervisors, and had started training the existing supervisors at that time through this course until the customized course was fielded. According to TSA’s training records, as of September 2004, about 3,800 Screening Supervisors had completed the course—approximately 92 percent of current Screening Supervisors. In response to our survey, one FSD noted that the supervisory training was long overdue because most of the supervisors had no prior federal service or, in some cases, no leadership experience. This FSD also noted that “leadership and supervisory skills should be continuously honed; thus, the development of our supervisors should be an extended and sequential program with numerous opportunities to develop skills—not just a one-time class.”

In addition to the Department of Agriculture Graduate School course, TSA’s Online Learning Center includes over 60 supervisory courses designed to develop leadership and coaching skills. In April 2004, TSA included in the Online Learning Center a Web-based technical training course—required for all Lead Screeners and Screening Supervisors. This course covers technical issues, such as resolving alarms at screening checkpoints. TSA introduced this course to the field in March 2004, and although the course is a requirement, TSA officials stated that they have not set goals for when all Lead Screeners and Screening Supervisors should have completed the course. In June 2004, TSA training officials stated that a second supervisor technical course was planned for development and introduction later in 2004. However, in December 2004, the training officials stated that planned funding for supervisory training may be used to support other TSA initiatives. The officials acknowledged that this would reduce TSA’s ability to provide the desired type and level of supervisory training to its Lead Screener, Screening Supervisor, and Screening Manager staff. TSA plans to revise its plans to provide Lead Screener, Screening Supervisor, and Screening Manager training based on funding availability.

Although TSA has developed leadership and technical courses for Screening Supervisors, many FSDs, in response to our general survey, identified additional types of training needed to enhance screener supervision. Table 2 provides a summary of the additional training needs that FSDs reported.

TSA training officials stated that the Online Learning Center provides several courses that cover these topics. Such courses include Situation Leadership II; Communicating with Difficult People: Handling Difficult Co-Workers; Team Participation: Resolving Conflict in Teams; Employee Performance: Resolving Conflict; High Impact Hiring; Team Conflict: Overcoming Conflict with Communication; Correcting Performance Problems: Disciplining Employees; Team Conflict: Working in Diversified Teams; Correcting Performance Problems: Identifying Performance Problems; Resolving Interpersonal Skills; Grammar, Skills, Punctuation, Mechanics and Word Usage; and Crisis in Organizations: Managing Crisis Situations.

TSA training officials acknowledged that for various reasons FSDs might not be aware that the supervisory and leadership training is available. For example, FSDs at airports without high-speed Internet/intranet access to the Online Learning Center might not have access to all of these courses. It is also possible that certain FSDs have not fully browsed the contents of the Online Learning Center and therefore are not aware that the training is available. Furthermore, officials stated that online learning is relatively new to government and senior field managers, and some of the FSDs may expect traditional instructor-led classes rather than online software.

Some FSDs Reported Impediments to Screeners Receiving Recurrent Training

Some FSDs responded to our general survey that they faced challenges with screeners receiving recurrent training, including insufficient staffing to allow all screeners to complete training within normal duty hours and a lack of high-speed Internet/intranet connectivity at some training facilities. According to our guide for assessing training, to foster an environment conducive to effective training and development, agencies must take actions to provide sufficient time, space, and equipment to employees to complete required training. TSA has set a requirement for 3 hours of recurrent training per week averaged over a quarter year, for both full-time and part-time screeners. However, FSDs for about 18 percent (48 of 263) of the airports in our airport-specific survey reported that screeners received less than 9 to 12 hours of recurrent training per month. Additionally, FSDs for 48 percent (125 of 263) of the airports in the survey reported that there was not sufficient time for screeners to receive recurrent training within regular work hours.

At 66 percent of those airports where the FSD reported that there was not sufficient time for screeners to receive recurrent training within regular work hours, the FSDs cited screener staffing shortages as the primary reason. We reported in February 2004 that FSDs at 11 of the 15 category X airports we visited reported that they were below their authorized staffing levels because of attrition and difficulties in hiring new staff. In addition, three of these FSDs noted that they had never been successful in hiring up to the authorized staffing levels. We also reported in February 2004 that FSDs stated that because of staffing shortages, they were unable to let screeners participate in training because it affected the FSD’s ability to provide adequate coverage at the checkpoints. In response to our survey, FSDs across all categories of airports reported that screeners must work overtime in order to participate in training. A September 2004 DHS OIG report recommended that TSA examine the workforce implications of the 3-hour training requirement and take steps to correct identified imbalances in future workforce planning to ensure that all screeners are able to meet the recurrent training standard. The 3-hours-per-week training standard represents a staff time commitment of 7.5 percent of full- time and between 9 and 15 percent of part-time screeners’ nonovertime working hours. TSA headquarters officials have stated that because the 3- hours-per-week requirement is averaged over a quarter, it provides flexibility to account for the operational constraints that exist at airports. However, TSA headquarters officials acknowledged that many airports are facing challenges in meeting the 3-hour recurrent training requirement. TSA data for the fourth quarter of fiscal year 2004 reported that 75 percent of airports were averaging less than 3 hours of recurrent training per week per screener. The current screener staffing model, which is used to determine the screener staffing allocations for each airport, does not take the 3-hours-per-week recurrent training requirement into account. However, TSA headquarters officials said that they are factoring this training requirement into their workforce planning efforts, including the staffing model currently under development.

Another barrier to providing recurrent training is the lack of high-speed Internet/intranet access at some of TSA’s training locations. TSA officials acknowledged that many of the features of the Online Learning Center, including some portions of the training modules and some Online Learning Center course offerings, are difficult or impossible to use in the absence of high-speed Internet/intranet connectivity. As one FSD put it, “the delayed deployment of the high-speed Internet package limits the connectivity to TSA HQ for various online programs that are mandated for passenger screening operations including screener training.” One FSD for a category IV airport noted the lack of a high-speed connection for the one computer at an airport he oversees made the Online Learning Center “nearly useless.”

TSA began deploying high-speed access to its training sites and checkpoints in May 2003 and has identified high-speed connectivity as necessary in order to deliver continuous training to screeners. TSA’s July 2003 Performance Improvement Study recommended accelerating high- speed Internet/intranet access in order to provide quick and systematic distribution of information and, thus, reduce uncertainty caused by the day-to-day changes in local and national procedures and policy. In October 2003, TSA reported plans to have an estimated 350 airports online with high-speed connectivity within 6 months. However, in June 2004, TSA reported that it did not have the resources to reach this goal.

TSA records show that as of October 2004, TSA had provided high-speed access for training purposes to just 109 airports, where 1,726 training computers were fully connected. These 109 airports had an authorized staffing level of over 24,900 screeners, meaning that nearly 20,100 screeners (45 percent of TSA’s authorized screening workforce) still did not have high-speed Internet/intranet access to the Online Learning Center at their training facility. In October 2004, TSA officials stated that TSA’s Office of Information Technology had selected an additional 16 airport training facilities with a total of 205 training computers to receive high- speed connectivity by the end of December 2004. As of January 19, 2005, TSA was unable to confirm that these facilities had received high-speed connectivity. Additionally, they could not provide a time frame for when they expected to provide high-speed connectivity to all airport training facilities because of funding uncertainties. Furthermore, TSA does not have a plan for prioritizing and scheduling the deployment of high-speed connectivity to all airport training facilities once funding is available. Without a plan, TSA’s strategy and timeline for implementing connectivity to airport training facilities is unclear. The absence of such a plan limits TSA’s ability to make prudent decisions about how to move forward with deploying connectivity once funding is available. Figure 5 shows the percentage of airports reported to have high-speed connectivity for their training computers by category of airport as of October 2004.

To mitigate airport connectivity issues in the interim, on April 1, 2004, TSA made the Online Learning Center courses accessible through public Internet connections, which enable screeners to log on to the Online Learning Center from home, a public library, or other locations. However, TSA officials stated that the vast majority of screeners who have used the Online Learning Center have logged in from airports with connectivity at their training facilities. TSA also distributes new required training products using multiple delivery channels, including written materials and CD-ROMs for those locations where access to the Online Learning Center is limited. Specifically, TSA officials stated that they provided airports without high-speed connectivity with CD-ROMs for the 50 most commonly used optional commercial courseware titles covering topics such as information technology skills, customer service, and teamwork. Additionally, officials stated that as technical courses are added to the Online Learning Center, they are also distributed via CD-ROM and that until full connectivity is achieved, TSA will continue to distribute new training products using multiple delivery channels.

TSA Lacks Adequate Controls to Provide Reasonable Assurance That Screeners Receive Required Training

Because of a lack of internal controls, TSA cannot provide reasonable assurance that screeners are completing required training. First, TSA policy does not clearly define responsibility for ensuring that screeners have completed all required training. Additionally, TSA has no formally defined policies or procedures for documenting completion of remedial training, or a system designed to facilitate review of this documentation for purposes of monitoring. Further, TSA headquarters does not have formal policies and procedures for monitoring completion of basic training and lacks procedures for monitoring recurrent training. Finally, at airports without high-speed connectivity, training records must be entered manually, making it challenging for some airports to keep accurate and up- to-date training records.

TSA Policy Does Not Clearly Define Responsibility for Ensuring That Screeners Are in Compliance with Training Requirements

TSA’s current guidance for FSDs regarding the training of the screener workforce does not clearly identify responsibility for tracking and ensuring compliance with training requirements. In a good control environment, areas of authority and responsibility are clearly defined and appropriate lines of reporting are established. In addition, internal control standards also require that responsibilities be communicated within an organization. The Online Learning Center provides TSA with a standardized, centralized tool capable of maintaining all training records in one system. It replaces an ad hoc system previously used during initial rollout of federalized screeners in which contractors maintained training records. A February 2004 management directive states that FSDs are responsible for ensuring the completeness, accuracy, and timeliness of training records maintained in the Online Learning Center for their employees. For basic and recurrent training, information is to be entered into the Online Learning Center within 30 days of completion of the training activity. However, the directive does not clearly identify who is responsible for ensuring that employees comply with training requirements. Likewise, a December 2003 directive requiring that screeners complete 3 hours of training per week averaged over a quarter states that FSDs are responsible for ensuring that training records for each screener are maintained in the Online Learning Center. Although both directives include language that requires FSDs to ensure training records are maintained in the Online Learning Center, neither specifies whether FSDs or headquarters officials are responsible for ensuring compliance with the basic, recurrent, and remedial training requirements. Even so, TSA headquarters officials told us that FSDs are ultimately responsible for ensuring screeners receive required training. However, officials provided no documentation clearly defining this responsibility. Without a clear designation of responsibility for monitoring training completion, this function may not receive adequate attention, leaving TSA unable to provide reasonable assurance that its screening workforce receives required training. In April 2005, TSA officials responsible for training stated that they were updating the February 2004 management directive on training records to include a specific requirement for FSDs to ensure that screeners complete required training. They expect to release the revised directive in May 2005.

TSA Lacks Formal Policies and Procedures for Monitoring Completion of Required Training

TSA has not established and documented policies and procedures for monitoring completion of basic and recurrent training. Internal control standards advise that internal controls should be designed so that monitoring is ongoing and ingrained in agency operations. However, TSA headquarters officials stated that they have no formal policy for monitoring screeners’ completion of basic training. They also stated that they have neither informal nor formal procedures for monitoring the completion of screeners’ recurrent training requirements, and acknowledged that TSA policy does not address what is to occur if a screener does not meet the recurrent training requirement. Officials further stated that individual FSDs have the discretion to determine what action, if any, to take when screeners do not meet this requirement.

In July 2004, TSA training officials stated that headquarters staff recently began running a report in the Online Learning Center to review training records to ensure that newly hired screeners had completed required basic training. In addition, they stated that in June 2004, they began generating summary-level quarterly reports from the Online Learning Center to quantify and analyze hours expended for recurrent screener training. Specifically, TSA training officials stated that reports showing airport-level compliance with the 3-hour recurrent requirement were generated for the third and fourth quarters of fiscal year 2004 and delivered to the Office of Aviation Operations for further analysis and sharing with the field. However, Aviation Operations officials stated that they did not use these reports to monitor the status of screener compliance with the 3-hour recurrent training requirement and do not provide them to the field unless requested by an FSD. TSA training officials said that while headquarters intends to review recurrent training activity on an ongoing basis at a national and airport level, they view FSDs and FSD training staff as responsible for ensuring that individuals receive all required training. Further, they acknowledged that weaknesses existed in the reporting capability of the Online Learning Center and stated that they plan to upgrade the Online Learning Center with improved reporting tools by the end of April 2005. Without clearly defined policies and procedures for monitoring the completion of training, TSA lacks a structure to support continuous assurance that screeners are meeting training requirements.

TSA Lacks Clearly Defined Policies and Procedures for Documenting Completion of Remedial Training

TSA has not established clear policies and procedures for documenting completion of required remedial training. The Standards for Internal Control state that agencies should document all transactions and other significant events and should be able to make this documentation readily available for examination. A TSA training bulletin dated October 15, 2002, specifies that when remedial training is required, FSDs must ensure the training is provided and a remedial training reporting form is completed and maintained with the screener’s local records. However, when we asked to review these records, we found confusion as to how and where they were to be maintained. TSA officials stated that they are waiting for a decision regarding how to maintain these records because of their sensitive nature. In the meantime, where and by whom the records should be maintained remains unclear.

In September 2004, officials from TSA’s OIAPR—responsible for conducting covert testing—stated that they maintain oversight to ensure screeners requiring remedial training receive required training by providing a list of screeners that failed covert testing and therefore need remedial training to TSA’s Office of Aviation Operations. Aviation Operations is then to confirm via memo that each of the screeners has received the necessary remedial training and report back to OIAPR. Accordingly, we asked TSA for all Aviation Operations memos confirming completion of remedial training, but we were only able to obtain 1 of the 12 memos.

In addition, during our review, we asked to review the remedial training reporting forms at five airports to determine whether screeners received required training, but we encountered confusion about requirements for maintaining training records and inconsistency in record keeping on the part of local TSA officials. Because of the unclear policies and procedures for recording completion of remedial training, TSA does not have adequate assurance that screeners are receiving legislatively mandated remedial training.

Lack of High-Speed Connectivity Limits TSA’s Ability to Document and Track the Completion of Screener Training

Although training computers with high-speed Internet/intranet connectivity automatically record completion of training in the Online Learning Center, airports without high-speed access at their training facility must have these records entered manually. The February 2004 management directive that describes responsibility for entering training records into the Online Learning Center also established that all TSA employees are required to have an official TSA training record in the Online Learning Center that includes information on all official training that is funded wholly or in part with government funds. Without high- speed access, TSA officials stated that it can be a challenge for airports to keep the Online Learning Center up to date with the most recent training records. TSA headquarters officials further stated that when they want to track compliance with mandatory training such as ethics or civil rights training, they provide the Training Coordinators with a spreadsheet on which to enter the data rather than relying on the Online Learning Center. As one FSD told us, without high-speed connectivity at several of the airports he oversees, “this is very time consuming and labor intensive and strains my limited resources.” The difficulty that airports encounter in maintaining accurate records when high-speed access is absent could compromise TSA’s ability to provide reasonable assurance that screeners are receiving mandated basic and remedial training.

Progress Has Been Made in Implementing Tools to Measure and Enhance Screener Performance, but Key Performance Targets Have Not Been Finalized

TSA has improved its efforts to measure and enhance screener performance. However, these efforts have primarily focused on passenger screening rather than checked baggage screening, and TSA has not yet finalized performance targets for several key performance measures. For example, TSA has increased the amount of covert testing it performs at airports. These tests have identified that, overall, weaknesses and vulnerabilities continue to exist in the passenger and checked baggage screening systems. TSA also enabled FSDs to conduct local covert testing, fully deployed the Threat Image Projection (TIP) system to passenger screening checkpoints at commercial airports nationwide, and completed the 2003/2004 annual screener recertification program for all eligible screeners. However, not all of these performance measurement and enhancement tools are available for checked baggage screening. Specifically, TIP is not currently operational at checked baggage screening checkpoints, and the recertification program does not include an image recognition component for checked baggage screeners. However, TSA is taking steps to address the overall imbalance in passenger and checked baggage screening performance data, including working toward implementing TIP for checked baggage screening and developing an image recognition module for checked baggage screener recertification. To enhance screener and screening system performance, TSA has also conducted a passenger screener performance improvement study and subsequently developed an improvement plan consisting of multiple action items, many of which TSA has completed. However, TSA has not conducted a similar study for checked baggage screeners. In addition, TSA has established over 20 performance measures for the passenger and checked baggage screening systems as well as two performance indexes (one for passenger and one for checked baggage screening). However, TSA has not established performance targets for each of the component indicators within the indexes, such as covert testing. According to The Office of Management and Budget, performance goals are target levels of performance expressed as a measurable objective, against which actual achievement can be compared. Performance goals should incorporate measures (indicators used to gauge performance); targets (characteristics that tell how well a program must accomplish the measure), and time frames. Without these targets, TSA’s performance management system, and these performance indexes, specifically, may not provide the agency with the complete information necessary to assess achievements and make decisions about where to direct performance improvement efforts. Although TSA has not yet established performance targets for each of the component indicators, TSA plans to finalize performance targets for the indicators by the end of fiscal year 2005.

TSA Has Increased Its Covert Testing and Allowed Local Covert Testing at Passenger Screening Checkpoints

TSA headquarters has increased the amount of covert testing it performs and enabled FSDs to conduct additional local covert testing at passenger screening checkpoints. TSA’s OIAPR conducts unannounced covert tests of screeners to assess their ability to detect threat objects and to adhere to TSA-approved procedures. These tests, in which undercover OIAPR inspectors attempt to pass threat objects through passenger screening checkpoints and in checked baggage, are designed to measure vulnerabilities in passenger and checked baggage screening systems and to identify systematic problems affecting performance of screeners in the areas of training, policy, and technology. TSA considers its covert testing as a “snapshot” of a screener’s ability to detect threat objects at a particular point in time and as one of several indicators of system wide screener performance.

OIAPR conducts tests at passenger screening checkpoints and checked baggage screening checkpoints. According to OIAPR, these tests are designed to approximate techniques terrorists might use. These covert test results are one source of data on screener performance in detecting threat objects as well as an important mechanism for identifying areas in passenger and checked baggage screening needing improvement. In testimony before the 9/11 Commission, the Department of Transportation Inspector General stated that emphasis must be placed on implementing an aggressive covert testing program to evaluate operational effectiveness of security systems and equipment.

Between September 10, 2002, and September 30, 2004, OIAPR conducted a total of 3,238 covert tests at 279 different airports. In September 2003, we reported that OIAPR had conducted limited covert testing but planned to double the amount of tests it conducted during fiscal year 2004, based on an anticipated increase in its staff from about 100 full-time equivalents to about 200 full-time equivalents. TSA officials stated that based on budget constraints, OIAPR’s fiscal year 2004 staffing authorization was limited to 183 full-time-equivalents, of which about 60 are located in the field. Despite a smaller than expected staff increase, by the end of the second quarter of fiscal year 2004, OIAPR had already surpassed the number of tests it performed during fiscal year 2003, as shown in table 3.

In October 2003, OIAPR committed to testing between 90 and 150 airports by April 2004 as part of TSA’s short-term screening performance improvement plan. OAIPR officials stated that this was a onetime goal to increase testing. This initiative accounts for the spike in testing for the second quarter of fiscal year 2004.

OIAPR has created a testing schedule designed to test all airports at least once during a 3-year time frame. Specifically, the schedule calls for OIAPR to test all category X airports once a year, category I and II airports once every 2 years, and category III and IV airports at least once every 3 years.

In September 2003 and April 2004, we reported that TSA covert testing results had identified weaknesses in screeners’ ability to detect threat objects. More recently, in April 2005, we, along with the DHS OIG, identified that screener performance continued to be a concern. Specifically, our analysis of TSA’s covert testing results for tests conducted between September 2002 and September 2004 identified that overall, weaknesses still existed in the ability of screeners to detect threat objects on passengers, in their carry-on bags, and in checked baggage. Covert testing results in this analysis cannot be generalized either to the airports where the tests were conducted or to airports nationwide. These weaknesses and vulnerabilities were identified at airports of all sizes, at airports with federal screeners, and airports with private-sector screeners. For the two-year period reviewed, overall failure rates for covert tests (passenger and checked baggage) conducted at airports using private- sector screeners were somewhat lower than failure rates for the same tests conducted at airports using federal screeners for the airports tested during this period. Since these test results cannot be generalized as discussed above, each airport’s test results should not be considered a comprehensive measurement of the airport’s performance or any individual screener’s performance in detecting threat objects, or in determining whether airports with private sector screeners performed better than airports with federal screeners.

On the basis of testing data through September 30, 2004, we determined that OIAPR had performed covert testing at 61 percent of the nation’s commercial airports. TSA has until September 30, 2005, to test the additional 39 percent of airports and meet its goal of testing all airports within 3 years. Although officials stated that they have had to divert resources from airport testing to conduct testing of other modes and that testing for other modes of transportation may affect their ability to conduct airport testing, they still expect to meet the goal.

In February 2004, TSA provided protocols to help FSDs conduct their own covert testing of local airport passenger screening activities—a practice that TSA had previously prohibited. Results of local testing using these protocols are to be entered into the Online Learning Center. This information, in conjunction with OAIPR covert test results and TIP threat detection results, is intended to assist TSA in identifying specific training and performance improvement efforts. In February 2005, TSA released a general procedures document for local covert testing at checked baggage screening locations.

TSA officials said that they had not yet begun to use data from local covert testing to identify training and performance needs because of difficulties in ensuring that local covert testing is implemented consistently nationwide. These officials said that after a few months of collecting and assessing the data, they will have a better idea of how the data can be used.

TSA Has Reinstated the Threat Image Projection System and Plans to Use Its Data to Improve the Screening System

TSA has nearly completed the reactivation of the TIP system at airports nationwide and plans to use data it is collecting to improve the effectiveness of the passenger screening system. TIP is designed to test passenger screeners’ detection capabilities by projecting threat images, including guns, knives, and explosives, onto bags as they are screened during actual operations. Screeners are responsible for identifying the threat image and calling for the bag to be searched. Once prompted, TIP identifies to the screener whether the threat is real and then records the screener’s performance in a database that could be analyzed for performance trends. TSA is evaluating the possibility of developing an adaptive functionality to TIP. Specifically, as individual screeners become proficient in identifying certain threat images, such as guns or knives, they will receive fewer of those images and more images that they are less proficient at detecting, such as improvised explosive devices.

TIP was activated by FAA in 1999 with about 200 threat images, but it was shut down immediately following the September 11 terrorist attacks because of concerns that it would result in screening delays and panic, as screeners might think that they were actually viewing threat objects. In October 2003, TSA began reactivating and expanding TIP. In April 2004, we reported that TSA was reactivating TIP with an expanded library of 2,400 images at all but one of the more than 1,800 checkpoint lanes nationwide. To further enhance screener training and performance, TSA also plans to develop at least an additional 50 images each month.

Despite these improvements, TIP is not yet available for checked baggage screening. In April 2004, we reported that TSA officials stated that they were working to resolve technical challenges associated with using TIP for checked baggage screening on EDS machines and have started EDS TIP image development. The DHS OIG reported in September 2004 that TSA plans to implement TIP on all EDS machines at checked baggage stations nationwide in fiscal year 2005. However, in December 2004, TSA officials stated that because of severe budget reductions, TSA will be unable to begin implementing a TIP program for checked baggage in fiscal year 2005. They did not specify when such a program might begin.

TSA plans to use TIP data to improve the passenger screening system in two ways. First, TIP data can be used to measure screener threat detection effectiveness by different threats. Second, TSA plans to use TIP results to help identify specific recurrent training needs within and across airports and to tailor screeners’ recurrent training to focus on threat category areas that indicate a need for improvement. TSA considers February 2004 as the first full month of TIP reporting with the new library of 2,400 images. TSA began collecting these data in early March 2004 and is using the data to determine more precisely how they can be used to measure screener performance in detecting threat objects and to determine what the data identify about screener performance.

TSA does not currently plan to use TIP data as an indicator of individual screener performance because TSA does not believe that TIP by itself adequately reflects a screener’s performance. Nevertheless, in April 2004, TSA gave FSDs the capability to query and analyze TIP data in a number of ways, including by screener, checkpoint, and airport. FSDs for over 60 percent of the airports included in our airport-specific survey stated that they use or plan to use TIP data as a source of information in their evaluations of individual screener performance. Additionally, FSDs for 50 percent of the airports covered in our survey reported using data generated by TIP to identify specific training needs for individual screeners.

In September 2004, the DHS OIG reported that TSA is assessing the cost and feasibility of modifying TIP so that it recognizes and responds to specific threat objects with which individual screeners are most and least competent in detecting, over time. This feature would increase the utility of TIP as a training tool. The DHS OIG also reported that TSA is considering linking TIP over a network, which would facilitate TSA’s collection, analysis, and information-sharing efforts around TIP user results. The report recommended that TSA continue to pursue each of these initiatives, and TSA agreed. However, in December 2004, TSA officials stated that the availability of funding will determine whether or not they pursue these efforts further.

TSA Has Completed Its First Round of Screener Recertification Testing, but Testing for Checked Baggage Screeners Is Not as Comprehensive as for Passenger Screeners

TSA has completed its first round of the screener recertification program, and the second round is now under way. However, TSA does not currently include an image recognition component in the test for checked baggage screener recertification. ATSA requires that each screener receive an annual proficiency review to ensure he or she continues to meet all qualifications and standards required to perform the screening function. In September 2003, we reported that TSA had not yet implemented this requirement. To meet this requirement, TSA established a recertification program, and it began recertification testing in October 2003 and completed the testing in March 2004. The first recertification program was composed of two assessment components, one of screeners’ performance and the other of screeners’ knowledge and skills. During the performance assessment component of the recertification program, screeners are rated on both organizational and individual goals, such as maintaining the nation’s air security, vigilantly carrying out duties with utmost attention to tasks that will prevent security threats, and demonstrating the highest levels of courtesy to travelers to maximize their levels of satisfaction with screening services. The knowledge and skills assessment component consists of three modules: (1) knowledge of standard operating procedures, (2) image recognition, and (3) practical demonstration of skills. Table 4 provides a summary of these three modules.

To be recertified, screeners must have a rating of “met” or “exceeded” standards on their annual performance assessments and have passed each of the applicable knowledge and skills modules. Screeners that failed any of the three modules were to receive study time or remedial training as well as a second opportunity to take and pass the modules. Screeners who failed on their second attempt were to be removed from screening duties and subject to termination. Screeners could also be terminated for receiving a rating of below “met” standards.

TSA completed its analysis of the recertification testing and performance evaluations in May 2004. TSA’s analysis shows that less than 1 percent of screeners subject to recertification failed to complete this requirement. Figure 6 shows the recertification results.

Across all airports screeners performed well on the recertification testing. Over 97 percent of screeners passed the standard operating procedures test on their first attempt. Screeners faced the most difficulty on the practical demonstration of skills component. However, following remediation, 98.6 percent of the screeners who initially failed this component passed on their second attempt. Table 5 shows the results of the recertification testing by module.

As shown in table 6, screeners hired as checked baggage screeners were not required to complete the image recognition module in the first round of the recertification testing.

In addition, during the first year of recertification testing, which took place from October 2003 through May 2004, dual-function screeners who were actively working as both passenger and checked baggage screeners were required to take only the recertification test for passenger screeners. They were therefore not required to take the recertification testing modules required for checked baggage, even though they worked in that capacity.

TSA began implementing the second annual recertification testing in October 2004 and plans to complete it no later than June 2005. This recertification program includes components for dual-function screeners. However, TSA still has not included an image recognition module for checked baggage screeners—which would include dual-function screeners performing checked baggage screening. TSA officials stated that a decision was made to not include an image recognition module for checked baggage screeners during this cycle because not all checked baggage screeners would have completed training on the onscreen resolution protocol by the time recertification testing was conducted at their airports. In December 2004, TSA officials stated that they plan on developing an image recognition module for checked baggage and dual- function screeners, and that this test should be available for next year’s recertification program. The development and implementation of the image recognition test will be contingent, they stated, upon the availability of funds.

TSA Has Identified and Implemented Efforts to Enhance Screener Performance, but These Efforts Primarily Focused on Passenger Screeners

TSA has implemented a number of improvements designed to enhance screener performance, based on concerns it identified in a July 2003 Passenger Screener Performance Improvement Study and recommendations from OIAPR. To date, however, these efforts have primarily focused on the performance of passenger screeners, and TSA has not yet undertaken a comparable performance study for checked baggage screeners. The Passenger Screener Performance Improvement Study relied in part on the findings of OIAPR’s covert testing. At the time the study was issued, OIAPR had conducted fewer than 50 tests of checked baggage screeners. The July 2003 study focused on and included numerous recommendations for improving the performance of passenger screeners, but recommended waiting to analyze the performance of checked baggage screeners until some time after implementation of the recommendations, some of which TSA indicated, also applied to checked baggage screeners. TSA officials told us that this analysis has been postponed until they have reviewed the impact of implementing the recommendations on passenger screening performance.

In October 2003, to address passenger screener performance deficiencies identified in the Screener Performance Improvement Study, TSA developed a Short-Term Screening Performance Improvement Plan. This plan included specific action items in nine broad categories—such as enhance training, increase covert testing, finish installing TIP, and expedite high-speed connectivity to checkpoints and training computers— that TSA planned to pursue to provide tangible improvements in passenger screener performance and security (see app. IV for additional information on the action items). In June 2004, TSA reported that it had completed 57 of the 62 specific actions. As of December 2004, two of these actions still had not been implemented—full deployment of high-speed connectivity and a time and attendance package—both of which continue to be deferred pending the identification of appropriate resources.

In addition to the Performance Improvement Study and corresponding action plans, TSA’s OIAPR makes recommendations in its reports on covert testing results. These recommendations address deficiencies identified during testing and are intended to improve screening effectiveness. As of December 2004, OIAPR had issued 18 reports to TSA management on the results of its checkpoint and checked baggage covert testing. These reports include 14 distinct recommendations, some of which were included in TSA’s screener improvement action plan. All but two of these reports included recommendations on corrective actions needed to enhance the effectiveness of passenger and checked baggage screening.

TSA Has Established Screening Performance Measures and Indexes but Has Not Established Key Performance Targets

TSA has established performance measures, indexes, and targets for the passenger and checked baggage screening systems, but has not established targets for the various components of the screening indexes. The Government Performance and Results Act of 1993 provides, among other things, that federal agencies establish program performance measures, including the assessment of relevant outputs and outcomes of each program activity. Performance measures are meant to cover key aspects of performance and help decision makers to assess program accomplishments and improve program performance. A performance target is a desired level of performance expressed as a tangible, measurable objective, against which actual achievement will be compared. By analyzing the gap between target and actual levels of performance, management can target those processes that are most in need of improvement, set improvement goals, and identify appropriate process improvements or other actions.

An April 2004 consultant study commissioned by TSA found that FSDs and FSD staffs generally believed the lack of key performance indicators available to monitor passenger and checked baggage screening performance represented a significant organizational weakness. Since then, TSA has established over 20 performance measures for the passenger and checked baggage screening systems. For example, TSA measures the percentage of screeners meeting a threshold score on the annual recertification testing on their first attempt, the percentage of screeners scoring above the national standard level on TIP performance, and the number of passengers screened, by airport category.

TSA also has developed two performance indexes to measure the effectiveness of the passenger and checked baggage screening systems. These indexes measure overall performance through a composite of indicators and are derived by combining specific performance measures relating to passenger and checked baggage screening, respectively. Specifically, these indexes measure the effectiveness of the screening systems through machine probability of detection and covert testing results; efficiency through a calculation of dollars spent per passenger or bag screened; and customer satisfaction through a national poll, customer surveys, and customer complaints at both airports and TSA’s national call center. According to TSA officials, the agency has finalized targets for the two overall indexes, but these targets have not yet been communicated throughout the agency. Further, TSA plans to provide the FSDs with only the performance index score, not the value of each of the components, because the probabilities of detection are classified as secret and TSA is concerned that by releasing components, those probabilities could be deduced. Table 7 summarizes the components of the performance indexes developed by TSA.

TSA has not yet established performance targets for the various components of the screening indexes, including performance targets for covert testing (person probability of detection). TSA’s strategic plan states that the agency will use the performance data it collects to make tactical decisions based on performance. The screening performance indexes developed by TSA can be a useful analysis tool, but without targets for each component of the index, TSA will have difficulty performing meaningful analyses of the parts that add up to the index. For example, without performance targets for covert testing, TSA will not have identified a desired level of performance related to screener detection of threat objects. Performance targets for covert testing would enable TSA to focus its improvement efforts on areas determined to be most critical, as 100 percent detection capability may not be attainable. In January 2005, TSA officials stated that the agency plans to track the performance of individual index components and establish performance targets against which to measure these components. They further stated that they are currently collecting and analyzing baseline data to establish these targets and plan to finalize them by the end of fiscal year 2005.

Conclusions

It has been over 2 years since TSA assumed responsibility for passenger and checked baggage screening operations at the nation’s commercial airports. TSA has made significant accomplishments over this period in meeting congressional mandates related to establishing these screening operations. With the congressional mandates now largely met, TSA has turned its attention to assessing and enhancing the effectiveness of its passenger and checked baggage screening systems. An important tool in enhancing screener performance is ongoing training. As threats and technology change, the training and development of screeners to ensure they have the competencies—knowledge, skills, abilities, and behaviors— needed to successfully perform their screening functions become vital to strengthening aviation security. Without addressing the challenges to delivering ongoing training, including installing high-speed connectivity at airport training facilities, TSA may have difficulty maintaining a screening workforce that possesses the critical skills needed to perform at a desired level. In addition, without adequate internal controls designed to help ensure screeners receive required training that are also communicated throughout the agency, TSA cannot effectively provide reasonable assurances that screeners receive all required training. Given the importance of the Online Learning Center in both delivering training and serving as the means by which the completion of screener training is documented, TSA would benefit from having a clearly defined plan for prioritizing the deployment of high-speed Internet/intranet connectivity to all airport training facilities. Such a plan would help enable TSA to move forward quickly and effectively in deploying high-speed connectivity once funding is available.

Additionally, history demonstrates that U.S. commercial aircraft have long been a target for terrorist attacks through the use of explosives carried in checked baggage, and covert testing conducted by TSA and DHS OIG have identified that weaknesses and vulnerabilities continue to exist in the passenger and checked baggage screening systems, including the ability of screeners to detect threat objects. While covert test results provide an indicator of screening performance, they cannot solely be used as a comprehensive measure of any airport’s screening performance or any individual screener’s performance, or in determining the overall performance of federal versus private-sector screening. Rather, these data should be considered in the larger context of additional performance data, such as TIP and recertification test results, when assessing screener performance. While TSA has undertaken efforts to measures and strengthen performance, these efforts have primarily focused on passenger screening and not on checked baggage screening. TSA’s plans for implementing TIP for checked baggage screening, and establishing an image recognition component for the checked baggage screeners recertification testing—plans made during the course of our review— represent significant steps forward in its efforts to strengthen checked baggage screening functions. Additionally, although TSA has developed passenger and checked baggage screening effectiveness measures, the agency has not yet established performance targets for the individual components of these measures. Until such targets are established, it will be difficult for TSA to draw more meaningful conclusions about its performance and how to most effectively direct its improvement efforts. For example, performance targets for covert testing would enable TSA to focus its improvement efforts on areas determined to be most critical, as 100 percent detection capability may not be attainable. We are encouraged by TSA’s recent plan to establish targets for the individual components of the performance indexes. This effort, along with the additional performance data TSA plans to collect on checked baggage screening operations, should assist TSA in measuring and enhancing screening performance and provide TSA with more complete information with which to prioritize and focus its screening improvement efforts.

Recommendations for Executive Action

To help ensure that all screeners have timely and complete access to screener training available in the Online Learning Center and to help provide TSA management with reasonable assurance that all screeners are receiving required passenger and checked baggage screener training, we recommend that the Secretary of the Department of Homeland Security direct the Assistant Secretary, Transportation Security Administration, to take the following two actions: develop a plan that prioritizes and schedules the deployment of high-speed Internet/intranet connectivity to all TSA’s airport training facilities to help facilitate the delivery of screener training and the documentation of training completion, and develop internal controls, such as specific directives, clearly defining responsibilities for monitoring and documenting the completion of required training, and clearly communicate these responsibilities throughout the agency.

Agency Comments and Our Evaluation

We provided a draft of this report to DHS for review and comment. On February 4, 2005, we received written comments on the draft report, which are reproduced in full in appendix V. DHS generally concurred with the findings and recommendations in the report, and agreed that efforts to implement our recommendations are critical to successful passenger and checked baggage screening training and performance. With regard to our recommendation that TSA develop a plan that prioritizes and schedules the deployment of high-speed Internet/intranet connectivity to all TSA’s airport training facilities, DHS stated that TSA has developed such a plan. However, although we requested a copy of the plan several times during our review and after receiving written comments from DHS, TSA did not provide us with a copy of the plan. Therefore, we cannot assess the extent to which the plan DHS referenced in its written comments fulfills our recommendation. In addition, regarding our recommendation that TSA develop internal controls clearly defining responsibilities for monitoring and documenting the completion of required training, and clearly communicate those responsibilities throughout TSA, DHS stated that it is taking steps to define responsibility for monitoring the completion of required training and to insert this accountability into the performance plans of all TSA supervisors. TSA’s successful completion of these ongoing and planned activities should address the concerns we raised in this report. DHS has also provided technical comments on our draft report, which we incorporated where appropriate.

As agreed with your office, we will send copies of this report to relevant congressional committees and subcommittees and to the Secretary of the Department of Homeland Security. We will also make copies available to others upon request. In addition, the report will be made available at no charge on GAO’s Web site at http://www.gao.gov.

If you have any questions about this report or wish to discuss it further, please contact me at (202) 512-8777. Key contributors to this report are listed in appendix VI.

Appendix I: Summary of Previous Findings Related to Screener Training and Performance

Summary of previous findings related to screener training and performance

The Transportation Security Administration (TSA) had deployed a basic screener training program and required remedial training but had not fully developed or deployed a recurrent training program for screeners or supervisors.

TSA had collected little information to measure screener performance in detecting threat objects.

TSA’s Office of Internal Affairs and Program Review’s (OIAPR) covert testing was the primary source of information collected on screeners’ ability to detect threat objects. However, TSA did not consider the covert testing a measure of screener performance.

TSA was not using the Threat Image Projection system (TIP) but planned to fully activate the system with significantly more threat images than previously used in October 2003.

TSA had not yet implemented an annual proficiency review to ensure that screeners met all qualifications and standards required to perform their assigned screening functions.

Although little data existed on the effectiveness of passenger screening, TSA was implementing several efforts to collect performance data.

Aviation Security: Efforts to Measure Effectiveness and Address Challenges planned to double the number of tests it conducted during fiscal year 2004.

TSA only recently began activating TIP on a wide-scale basis and expected it to be fully operational at every checkpoint at all airports by April 2004.

TSA only recently began implementing the annual recertification program and did not expect to complete testing at all airports until March 2004.

TSA was developing performance indexes for individual screeners and the screening system as a whole but had not fully established these indexes. TSA expected to have them in place by the end of fiscal year 2004.

Aviation Security: Efforts to Measure Effectiveness and Strengthen Security Programs working with the U.S. Department of Agriculture’s Graduate School to tailor its off-the-shelf supervisory course to meet the specific training needs of screening supervisors.

While TSA had taken steps to enhance its screener training programs, staffing imbalances, and lack of high-speed connectivity at airport training facilities had made it difficult for screeners at some airports to fully utilize these programs.

Although TSA was making progress in measuring the performance of passenger screeners, it had collected limited performance data related to its checked baggage screening operations. However, TSA had begun collecting additional performance data related to its checked baggage screening operations and planned to increase these efforts in the future.

As part of its efforts to develop performance indexes, TSA was developing baseline data for fiscal year 2004 and planned to report the indexes to DHS in fiscal year 2005.

With the exception of covert testing and recent TIP data, data were not yet available to assess how well screeners were performing and what steps if any TSA needed to take to improve performance. Also, TSA was not using TIP as a formal indicator of screening performance, but instead was using it to identify individual screener training needs.

Appendix II: Objectives, Scope, and Methodology

To examine efforts by the Transportation Security Administration to enhance their passenger and checked baggage screening programs, we addressed the following questions: (1) What actions has TSA taken to enhance training for screeners and supervisors? (2) How does TSA monitor compliance with screener training requirements? (3) What is the status of TSA’s efforts to assess and enhance screener performance in detecting threat objects?

To determine how TSA has enhanced training for screeners and supervisors and how TSA has monitored compliance with screener training requirements, we obtained and analyzed relevant legislation, as well as TSA’s training plans, guidance, and curriculum. We reviewed data from TSA’s Online Learning Center and assessed the reliability of the Online Learning Center database. We compared TSA’s procedures for ensuring that screeners receive required training according to Standards for Internal Controls in the Federal Government. We interviewed TSA officials from the Office of Workforce Performance and Training and the Office of Aviation Operations in Arlington, Virginia. At the airports we visited, we interviewed Federal Security Directors and their staffs, such as Training Coordinators. We also met with officials from four aviation associations—the American Association of Airport Executives, Airports Council International, the Air Transport Association, and the Regional Airline Association. We did not assess the methods used to develop TSA’s screener training program, nor did we analyze the contents of TSA’s curriculum. Although we could not independently verify the reliability of all of this information, we compared the information with other supporting documents, when available, to determine data consistency and reasonableness. We found the data to be sufficiently reliable for our purposes.

To determine what efforts TSA has taken to assess and to enhance screener performance in detecting threat objects, we reviewed related reports from the Department of Transportation and the Department of Homeland Security (DHS) Inspector General, Congressional Research Service, and TSA, as well as prior GAO reports. We obtained and reviewed TSA’s covert test data and results of the annual recertification testing. (Results of the covert testing are classified and will be the subject of a separate classified GAO report.) We discussed methods for inputting, compiling, and maintaining the data with TSA officials. We also assessed the methodology of TSA’s covert tests and questioned OIAPR officials about the procedures used to ensure the reliability of the covert test data. When we found discrepancies between the data OIAPR maintained in spreadsheets and the data included in the hard copy reports we obtained from TSA, we worked with OIAPR to resolve the discrepancies. Further, we visited TSA headquarters to review TSA’s annual recertification testing modules and discuss TSA’s process for validating the recertification exams. As a result, we determined that the data provided by TSA were sufficiently reliable for the purposes of our review. We also reviewed TSA’s performance measures, targets, and indexes. Finally, we interviewed TSA headquarters officials from several offices in Arlington, Virginia, including Aviation Operations, Workforce Performance and Training, Strategic Management and Analysis, and Internal Affairs and Program Review.

In addition, in accomplishing our objectives, we also conducted site visits at select airports nationwide to interview Federal Security Directors and their staffs and conducted two Web-based surveys of Federal Security Directors. Specifically, we conducted site visits at 29 airports (13 category X airports, 9 category I airports, 3 category II airports, 3 category III airports, and 1 category IV airport) to observe airport security screening procedures and discuss issues related to the screening process with TSA, airport, and airline officials. We chose these airports to obtain a cross- section of all airports by size and geographic distribution. In addition, we selected each of the five contract screening pilot airports. The results from our airport visits provide examples of screening operations and issues but cannot be generalized beyond the airports visited because we did not use statistical sampling techniques in selecting the airports. The category X airports we visited were Baltimore Washington International Airport, Boston Logan International Airport, Chicago O’Hare International Airport, Dallas/Fort Worth International Airport, Denver International Airport, Washington Dulles International Airport, John F. Kennedy International Airport, Los Angeles International Airport, Newark Liberty International Airport, Orlando International Airport, Ronald Reagan Washington National Airport, San Francisco International Airport, Seattle-Tacoma International Airport. The category I airports we visited were Burbank- Glendale-Pasadena Airport, John Wayne Airport, Chicago Midway International Airport, Dallas Love Field, Kansas City International Airport, Little Rock National Airport, Metropolitan Oakland International Airport, Portland International Airport, and Tampa International Airport. The category II airports we visited were Jackson International Airport, Dane County Regional Airport, and Greater Rochester International Airport. The category III airports we visited were Idaho Falls Regional Airport, Jackson Hole Airport, and Orlando Sanford International Airport. The category IV airport we visited was Tupelo Regional Airport.

Further, we administered two Web-based surveys to all 155 Federal Security Directors who oversee security at each of the airports falling under TSA’s jurisdiction. One survey, the general survey, contained questions covering local and national efforts to train screeners and supervisors and the status of TSA’s efforts to evaluate screener performance, including the annual recertification program and TIP. The second survey attempted to gather more specific airport security information on an airport(s) under the Federal Security Director’s supervision. For the airport-specific survey, each Federal Security Director received one or two surveys to complete, depending on the number of airports they were responsible for. Where a Federal Security Director was responsible for more than two airports, we selected the first airport based on the Federal Security Director’s location and the second airport to obtain a cross-section of all airports by size and geographic distribution. In all, we requested information on 265 airports. However, two airports were dropped from our initial selection because the airlines serving these airports suspended operations and TSA employees were redeployed to other airports. As a result our sample size was reduced to 263 airports, which included all 21 category X, and 60, 49, 73, and 60 category I through IV airports respectively. In that we did not use probability sampling methods to select the sample of airports that were included in our airport-specific survey, we cannot generalize our findings beyond the selected airports.

A GAO survey specialist designed the surveys in combination with other GAO staff knowledgeable about airport security issues. We conducted pretest interviews with six Federal Security Directors to ensure that the questions were clear, concise, and comprehensive. In addition, TSA managers and an independent GAO survey specialist reviewed the survey.

We conducted these Web-based surveys from late March to mid-May 2004. We received completed general surveys from all 155 Federal Security Directors and completed airport-specific surveys for all 263 separate airports for which we sought information, for 100 percent response rates. The surveys’ results are not subject to sampling errors because all Federal Security Directors were asked to participate in the surveys and we did not use probability-sampling techniques to select specific airports. However, the practical difficulties of conducting any survey may introduce other errors, commonly referred to as nonsampling errors. For example, difficulties in how a particular question is interpreted, in the sources of information that are available to respondents, or in how the data are entered into a database or were analyzed can introduce unwanted variability into the survey results. We took steps in the development of the surveys, the data collection, and the data editing and analysis to minimize these nonsampling errors. Also, in that these were Web-based surveys whereby respondents entered their responses directly into our database, there was little possibility of data entry or transcription error. In addition, all computer programs used to analyze the data were peer reviewed and verified to ensure that the syntax was written and executed correctly.

We performed our work from May 2003 through April 2005 in accordance with generally accepted government auditing standards. Certain information we obtained and analyzed regarding screener training and performance are classified or are considered by TSA to be sensitive security information. Accordingly, the results of our review of this information are not included in this report.

Appendix III: TSA Screener Training Tools Designed to Help Improve Screener Performance

Provide an informative and effective learning tool to enhance screeners’ skills in the areas of hand-wanding and pat-down searches of passengers.

This tool allows screeners to touch actual improvised explosive device (IED) components and build their own devices. This experiential learning will enable screeners to more readily detect real IEDs during screening. These weapons are also used to assist in training by using them for live testing conducted by FSD staff.

This tool allows screeners to touch actual firearms and begin to understand how they can be broken down into various parts. By understanding this and experiencing it, screeners are better able to see the components of a firearm during actual screening. These weapons are also used to assist in training by using them for live testing conducted by FSD staff.

Deployed January 26, 2004 Maintain and enhance the screeners’ X-ray image operational skills. Deployed February 5, 2004 Provide a tool that includes about 14,000 image combinations to practice threat identification.

These teams go into airports where data shows performance needs attention. The team offers a variety of services to assist in improving the performance, such as on-the-spot training and consulting services. Team visits can be initiated by FSDs, Internal Affairs reports, Quality Assurance trips, or MTAT Supervisors proactively visiting the airport and FSD.

Site visits completed from October 2003 through December 3, 2004: North Central (37 visits)

South Central (51 visits)

Northeast (25 visits)

Southeast (60 visits)

Western (53 visits)

Improve screener supervisors’ knowledge of federal government and TSA personnel rules and how to effectively coach and communicate with employees.

Approximately 3,800 supervisors have been trained.

Certification of screeners to perform supervisory maintenance tasks above and beyond operator training.

Provide students with basic skills needed to verify the identity of flying armed law enforcement officers.

This weekly product brings to light actual cases of weapons being found by law enforcement, with an explanation of how those weapons could be used to attack aviation.

Provide interactive, performance based recurrent Web-based training modules for checked baggage explosive detection systems (EDS).

Improve screener performance by providing an interactive tool complementary to Hand Held Metal Detector and Pat Down Video that allows the screener to practice proper techniques and receive immediate feedback.

Reinforces TSA’s customer service principles and places the screener in various situations requiring effective customer service responses.

Provide interactive, performance-based recurrent training modules for checkpoint and checked baggage operations.

Physical Bag Search Video Maintain and enhance screeners’ explosive trace detection (ETD) and physical bag search skills for carry-on and checked baggage.

Provide interactive recurrent Web-based training modules for ETD and physical bag search.

Provide an interactive, performance-based training tool to enhance screener’s ability to identify prohibited items.

Provide an informative and effective learning tool to maintain and enhance the skills of screeners in the areas of persons with prosthetics.

Provide a tool to practice threat identification with about 10,000,000 image combinations.

Sharing the X-Ray Tutor Version 2 library, this tool will allow screeners to practice finding threat items using the full capabilities of the TIP-ready X-ray machines.

Provide an interactive, performance-based tool to convey how the supervisor is to handle screening situations, handed off by the screening, following standard operator procedures.

Provide a Web-based training that will engage the student with 3- dimensional representations of the muscular frame, showing proper lifting techniques and the results of improper techniques.

Appendix IV: Summary of TSA’s Short-Term Action Items for Strengthening Passenger Screener Performance

Appendix V: Comments from the Department of Homeland Security

Appendix VI: GAO Contacts and Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to those named above, David Alexander, Leo Barbour, Lisa Brown, Elizabeth Curda, Kevin Dooley, Kathryn Godfrey, David Hooper, Christopher Jones, Stuart Kaufman, Kim Gianopoulos, Thomas Lombardi, Cady S. Panetta, Minette Richardson, Sidney Schwartz, Su Jin Yon, and Susan Zimmerman were key contributors to this report.