Aviation Security: Screener Training and Performance Measurement Strengthened, but More Work Remains
Published: 2005-05-02T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The screening of airport passengers and their checked baggage is a critical component in securing our nation's commercial aviation system. Since May 2003, GAO has issued six products related to screener training and performance. This report updates the information presented in the prior products and incorporates results from GAO's survey of 155 Federal Security Directors--the ranking Transportation Security Administration (TSA) authority responsible for the leadership and coordination of TSA security activities at the nation's commercial airports. Specifically, this report addresses (1) actions TSA has taken to enhance training for passenger and checked baggage screeners and screening supervisors, (2) how TSA ensures that screeners complete required training, and (3) actions TSA has taken to measure and enhance screener performance in detecting threat objects.
section_title: What GAO Found
paragraphs: TSA has initiated a number of actions designed to enhance screener training, such as updating the basic screener training course. TSA also established a recurrent training requirement and introduced the Online Learning Center, which makes self-guided training courses available over TSA's intranet and the Internet. Even with these efforts, Federal Security Directors reported that insufficient screener staffing and a lack of high-speed Internet/intranet connectivity at some training facilities have made it difficult to fully utilize training programs and to meet the recurrent training requirement of 3 hours per week, averaged over a quarter year, within regular duty hours. TSA acknowledged that challenges exist in recurrent training delivery and is taking steps to address these challenges, including factoring training into workforce planning efforts and distributing training through written materials and CD-ROMs. However, TSA has not established a plan prioritizing the deployment of high-speed Internet/intranet connectivity to all airport training facilities to facilitate screener access to training materials. TSA lacks adequate internal controls to provide reasonable assurance that screeners receive legislatively mandated basic and remedial training, and to monitor its recurrent training program. Specifically, TSA policy does not clearly specify the responsibility for ensuring that screeners have completed all required training. In addition, TSA officials have no formal policies or methods for monitoring the completion of required training and were unable to provide documentation identifying the completion of remedial training. TSA has implemented and strengthened efforts to measure and enhance screener performance. For example, TSA has increased the number of covert tests it conducts at airports, which test screeners' ability to detect threat objects on passengers, in their carry-on baggage, and in checked baggage. These tests identified that overall, weaknesses and vulnerabilities continue to exist in passenger and checked baggage screening systems at airports of all sizes, at airports with federal screeners, and at airports with private-sector screeners. While these test results are an indicator of performance, they cannot solely be used as a comprehensive measure of any airport's screening performance or any individual screener's performance. We also found that TSA's efforts to measure and enhance screener performance have primarily focused on passenger screening, not checked baggage screening. For example, TSA only uses threat image software on passenger screening X-ray machines, and the recertification testing program does not include an image recognition module for checked baggage screeners. TSA is taking steps to address the overall imbalance in passenger and checked baggage screening performance data. TSA also established performance indexes for the passenger and checked baggage screening systems, to identify an overall desired level of performance. However, TSA has not established performance targets for each of the component indicators that make up the performance indexes, including performance targets for covert testing. TSA plans to finalize these targets by the end of fiscal year 2005.

section_title: Letter
paragraphs: The screening of airport passengers and their checked baggage is a critical  component in securing our nation’s commercial aviation system. In an  effort to strengthen the security of commercial aviation, the President  signed the Aviation and Transportation Security Act on  November 19, 2001. The act created the Transportation Security  Administration (TSA) and mandated actions designed to strengthen  aviation security, including requiring that TSA assume responsibility for  conducting passenger and checked baggage screening at over 450  commercial airports in the United States by November 19, 2002. It has  been over 2 years since TSA assumed this responsibility, and the agency  has spent billions of dollars and implemented a wide range of initiatives to  enhance its passenger and checked baggage screening operations. Despite  the attention to passenger and checked baggage screening operations,  however, concerns about the effectiveness of the screening system  remain. For example, covert testing conducted by TSA’s Office of Internal  Affairs and Program Review and the Department of Homeland Security’s  (DHS) Office of Inspector General identified weaknesses in the ability of  screeners to detect threat objects. (The results of our analysis of TSA’s  covert testing data and test program are included in a separate classified  GAO report.)
To determine the progress TSA has made in strengthening its passenger  and checked baggage screening operations, the Subcommittee on  Aviation, House Committee on Transportation and Infrastructure,  requested that we examine TSA efforts to train screeners and to measure  and enhance screener performance. Since we began our work in May 2003,  we have issued six products that address issues related to screener  training and performance, including four to this Subcommittee (see app.  I). This report updates some of the information presented in our prior  products. In addition, it incorporates results from our surveys of 155  Federal Security Directors (FSD). The surveys were designed to obtain  information related to, among other issues, TSA’s efforts to train screeners  and supervisors and assess screener performance in detecting threat  objects. Specifically, this report addresses the following questions:  (1) What actions has TSA taken to enhance training for passenger and  checked baggage screeners and screening supervisors? (2) How does TSA  ensure that screeners complete required training? (3) What actions has  TSA taken to measure and enhance screener performance in detecting  threat objects?
In conducting our work, we reviewed TSA documentation related to  screener training requirements and performance testing. We also analyzed  data from our survey responses from all 155 FSDs about screener training,  supervision and performance. We also visited 29 airports of various sizes  and geographic locations to obtain a cross-section of all airports, including  the 5 airports with private-sector screeners. To gain a better  understanding of training and performance issues, during these visits, we  interviewed FSDs, members of their management teams, passenger and  checked baggage screeners, and airport officials. However, information  obtained during these visits cannot be generalized to all airports across the  nation. Additionally, we interviewed officials at TSA headquarters and  TSA’s transportation security laboratory about their experiences with  training and screener performance. We compared TSA practices and  procedures for monitoring completion of training with the Comptroller  General’s Standards for Internal Control in the Federal Government. We  assessed the reliability of the data we acquired from TSA regarding  screener testing and training completion, and found the data to be  sufficiently reliable for our purposes. A more detailed description of our  scope and methodology is contained in appendix II.
We conducted our work from May 2003 through April 2005 in accordance  with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: The performance of passenger and checked baggage screeners in  detecting threat objects at the nation’s airports has been a long-standing  concern. In 1978, screeners failed to detect 13 percent of the potentially  dangerous objects that Federal Aviation Administration (FAA) agents  carried through airport screening checkpoints during tests. In 1987,  screeners did not detect 20 percent of the objects in similar tests. In tests  conducted during the late 1990s, as the testing objects became more  realistic, screeners’ abilities to detect dangerous objects declined further.  In April 2004, we, along with the DHS Office of the Inspector General  (OIG), testified that the performance of screeners continued to be a  concern. More recent tests conducted by TSA’s Office of Internal Affairs  and Program Review (OIAPR) also identified weaknesses in the ability of  screeners to detect threat objects, and separate DHS OIG tests identified  comparable screener performance weaknesses. In its July 2004 report, The  National Commission on Terrorist Attacks Upon the United States, known  widely as the 9/11 Commission, also identified the need to improve  screener performance and to better understand the reasons for  performance problems.
After the terrorist attacks of September 11, 2001, the President signed the  Aviation and Transportation Security Act (ATSA) into law on November  19, 2001, with the primary goal of strengthening the security of the nation’s  aviation system. ATSA created TSA as an agency with responsibility for  securing all modes of transportation, including aviation. As part of this  responsibility, TSA oversees security operations at the nation’s more than  450 commercial airports, including passenger and checked baggage  screening operations. Prior to the passage of ATSA, air carriers were  responsible for screening passengers and checked baggage, and most used  private security firms to perform this function. FAA was responsible for  ensuring compliance with screening regulations.
Today, TSA security activities at airports are overseen by FSDs. Each FSD  is responsible for overseeing security activities, including passenger and  checked baggage screening, at one or more commercial airports. TSA  classifies the over 450 commercial airports in the United States into one of  five security risk categories (X, I, II, III, and IV) based on various factors,  such as the total number of takeoffs and landings annually, the extent to  which passengers are screened at the airport, and other special security  considerations. In general, category X airports have the largest number of  passenger boardings and category IV airports have the smallest. TSA  periodically reviews airports in each category and, if appropriate, updates  airport categorizations to reflect current operations. Figure 1 shows the  number of commercial airports by airport security category as of  December 2003.
In addition to establishing TSA and giving it responsibility for passenger  and checked baggage screening operations, ATSA set forth specific  enhancements to screening operations for TSA to implement, with  deadlines for completing many of them. These requirements included    assuming responsibility for screeners and screening operations at more  than 450 commercial airports by November 19, 2002;   establishing a basic screener training program composed of a minimum of  40 hours of classroom instruction and 60 hours of on-the-job training;   conducting an annual proficiency review of all screeners;    conducting operational testing of screeners;   requiring remedial training for any screener who fails an operational test;  and   screening all checked baggage for explosives using explosives detection  systems by December 31, 2002.
Passenger screening is a process by which authorized TSA personnel  inspect individuals and property to deter and prevent the carriage of any  unauthorized explosive, incendiary, weapon, or other dangerous item  aboard an aircraft or into a sterile area. Passenger screeners must inspect  individuals for prohibited items at designated screening locations. The  four passenger screening functions are:    X-ray screening of property,    walk-through metal detector screening of individuals,    hand-wand or pat-down screening of individuals, and    physical search of property and trace detection for explosives.
Checked baggage screening is a process by which authorized security  screening personnel inspect checked baggage to deter, detect, and prevent  the carriage of any unauthorized explosive, incendiary, or weapon  onboard an aircraft. Checked baggage screening is accomplished through  the use of explosive detection systems (EDS) or explosive trace detection  (ETD) systems, and through the use of alternative means, such as manual  searches, K-9 teams, and positive passenger bag match, when EDS and  ETD systems are unavailable on a temporary basis. Figure 2 provides an  illustration of passenger and checked baggage screening operations.
There are several positions within TSA for employees that perform and  directly supervise passenger and checked baggage screening functions.  Figure 3 provides a description of these positions.
To prepare screeners to perform screening functions, to keep their skills  current, and to address performance deficiencies, TSA provides three  categories of required screener training. Table 1 provides a description of  the required training.
In September 2003, we reported on our preliminary observations of TSA’s  efforts to ensure that screeners were effectively trained and supervised  and to measure screener performance. We found that TSA had  established and deployed a basic screener training program and required  remedial training but had not fully developed or deployed a recurrent  training program for screeners or supervisors. We also reported that TSA  had collected limited data to measure screener performance. Specifically,  TSA had conducted limited covert testing, the Threat Image Projection  System was not fully operational, and TSA had not implemented the  annual screener proficiency testing required by ATSA. In subsequent  products, we reported progress TSA had made in these areas and  challenges TSA continued to face in making training available to screeners  and in measuring and enhancing screener performance. A summary of our  specific findings is included in appendix I.
section_title: TSA Has Enhanced and Expanded Training, but Some Screeners Have Encountered Difficulty Accessing and Completing Recurrent Training
paragraphs: TSA has taken a number of actions to enhance the training of screeners  and Screening Supervisors but has encountered difficulties in providing  access to recurrent training. TSA has enhanced basic training by, among  other things, adding a dual-function (passenger and checked baggage)  screening course for new employees. Furthermore, in response to the  need for frequent and ongoing training, TSA has implemented an Online  Learning Center with self-guided training courses available to employees  over TSA’s intranet and the Internet and developed and deployed a  number of hands-on training tools. Moreover, TSA now requires  screeners to participate in 3 hours of recurrent training per week,  averaged over each quarter year. TSA has also implemented leadership  and technical training programs for Screening Supervisors. However,  some FSDs, in response to open-ended survey questions, identified a  desire for more training in specific areas, including leadership,  communication, and supervision. Further, despite the progress TSA has  made in enhancing and expanding screener and supervisory training, TSA  has faced challenges in providing access to recurrent training. FSDs  reported that insufficient staffing and a lack of high-speed  Internet/intranet connectivity at some training facilities have made it  difficult to fully utilize these programs and to meet training requirements.  TSA has acknowledged that challenges exist in recurrent screener training  delivery and is taking steps to address these challenges, including  factoring training requirements into workforce planning efforts and  distributing training through written materials and CD-ROMs until full  Internet/intranet connectivity is achieved. However, TSA does not have a  plan for prioritizing and scheduling the deployment of high-speed  connectivity to all airport training facilities once funding is available. The  absence of such a plan limits TSA’s ability to make prudent decisions  about how to move forward with deploying connectivity to all airports to  provide screeners access to online training.
subsections: section_title: TSA Has Enhanced Basic Screener Training
paragraphs: TSA has enhanced its basic screener training program by updating the  training to reflect changes to standard operating procedures, deploying a  new dual-function (passenger and checked baggage screening) basic  training curriculum, and allowing the option of training delivery by local  staff. As required by ATSA, TSA established a basic training program for  screeners composed of a minimum of 40 hours of classroom instruction  and 60 hours of on-the-job training. TSA also updated the initial basic  screener training courses at the end of 2003 to incorporate changes to  standard operating procedures and directives, which contain detailed  information on how to perform TSA-approved screening methods.  However, a recent study by the DHS OIG found that while incorporating  the standard operating procedures into the curricula was a positive step, a  number of screener job tasks were incompletely addressed in or were  absent from the basic training courses.
In addition to updates to the training curriculum, in April 2004, TSA  developed and implemented a new basic screener training program, dual- function screener training that covers the technical aspects of both  passenger and checked baggage screening. Initially, new hire basic  training was performed by a contractor and provided a screener with  training in either passenger or checked baggage screening functions. A  screener could then receive basic training in the other function later, at the  discretion of the FSD, but could not be trained in both functions  immediately upon hire. The new dual-training program is modular in  design. Thus, FSDs can chose whether newly hired screeners will receive  instruction in one or both of the screening functions during the initial  training. In addition, the individual modules can also be used to provide  recurrent training, such as refreshing checked baggage screening skills for  a screener who has worked predominately as a passenger screener. TSA  officials stated that this new approach provides the optimum training  solution based on the specific needs of each airport and reflects the fact  that at some airports the FSD does not require all screeners to be fully  trained in both passenger and checked baggage screening functions.
Some FSDs, particularly those at smaller airports, have made use of the  flexibility offered by the modular design of the new course to train  screeners immediately upon hire in both passenger and checked baggage  screening functions. Such training up front allows FSDs to use screeners  for either the passenger or the checked baggage screening function,  immediately upon completion of basic training. Figure 4 shows that  58 percent (3,324) of newly hired screeners trained between April 1, 2004,  and September 1, 2004, had completed the dual-function training.
In April 2004, TSA also provided FSDs with the flexibility to deliver basic  screener training using local instructors. TSA’s Workforce Performance  and Training Office developed basic screener training internally, and  initially, contractors delivered all of the basic training. Since then, TSA has  provided FSDs with the discretion to provide the training using local TSA  employees or to use contractors. The flexibility to use local employees  allows FSDs and members of the screener workforce to leverage their  first-hand screening knowledge and experience and address situations  unique to individual airports. As of December 10, 2004, TSA had trained  1,021 local FSD staff (representing 218 airports) in how to instruct the  dual-function screener training course. TSA officials stated that they  expect the use of TSA-approved instructors to increase over time.
section_title: TSA Has Deployed Recurrent Screener Training and Provided Additional Training Tools, but Some FSDs Identified the Need for More Courses
paragraphs: “Numerous interviews revealed concerns with training curriculum, communication, and  coordination issues that directly affect security screening. Unsatisfied with the quantity  and breadth of topics, many Training Coordinators have developed supplementary lectures  on both security and non-security related topics. These additional lectures…have been very  highly received by screeners.”
In October 2003, TSA introduced the Online Learning Center to provide  screeners with remote access to self-guided training courses. As of  September 14, 2004, TSA had provided access to over 550 training courses  via the Online Learning Center and made the system available via the  Internet and its intranet. TSA also developed and deployed a number of  hands-on training modules and associated training tools for screeners at  airports nationwide. These training modules cover topics including hand- wanding and pat-down techniques, physical bag searches, X-ray images,  prohibited items, and customer service. Additionally, TSA instituted  another module for the Online Learning Center called Threat in the  Spotlight, that, based on intelligence TSA receives, provides screeners  with the latest in threat information regarding terrorist attempts to get  threat objects past screening checkpoints. Appendix III provides a  summary of the recurrent training tools TSA has deployed to airports and  the modules currently under development.
In December 2003, TSA issued a directive requiring screeners to receive  3 hours of recurrent training per week averaged over a quarter year. One  hour is required to be devoted to X-ray image interpretation and the other  2 hours to screening techniques, review of standard operating procedures,  or other mandatory administrative training, such as ethics and privacy act  training.
In January 2004, TSA provided FSDs with additional tools to facilitate and  enhance screener training. Specifically, TSA provided airports with at least  one modular bomb set (MBS II) kit—containing components of an  improvised explosive device—and one weapons training kit, in part  because screeners had consistently told TSA’s OIAPR inspectors that they  would like more training with objects similar to ones used in covert  testing.
Although TSA has made progress with the implementation of recurrent  training, some FSDs identified the need for several additional courses,  including courses that address more realistic threats. TSA acknowledged  that additional screener training is needed, and officials stated that the  agency is in the process of developing new and improved screener  training, including additional recurrent training modules (see app. III).
section_title: TSA Provides Leadership and Technical Training for Supervisors, but Some FSDs Would Like More Training for Screening Supervisors and Lead Screeners
paragraphs: TSA has arranged for leadership training for screening supervisors through  the Department of Agriculture Graduate School and has developed  leadership and technical training courses for screening supervisors.  However, some FSDs reported the need for more training for Screening  Supervisors and Lead Screeners. The quality of Screening Supervisors has  been a long-standing concern. In testifying before the 9/11 Commission in  May 2003, a former FAA Assistant Administrator for Civil Aviation Security  stated that following a series of covert tests at screening checkpoints to  determine which were strongest, which were weakest, and why, invariably  the checkpoint seemed to be as strong or as weak as the supervisor who  was running it. Similarly, TSA’s OIAPR identified a lack of supervisory  training as a cause for screener covert testing failures. Further, in a July  2003 internal study of screener performance, TSA identified poor  supervision at the screening checkpoints as a cause for screener  performance problems. In particular, TSA acknowledged that many Lead  Screeners, Screening Supervisors, and Screening Managers did not  demonstrate supervisory and management skills (i.e., mentoring, coaching,  and positive reinforcement) and provided little or no timely feedback to  guide and improve screener performance. In addition, the internal study  found that because of poor supervision at the checkpoint, supervisors or  peers were not correcting incorrect procedures, optimal performance  received little reinforcement, and not enough breaks were provided to  screeners. A September 2004 report by the DHS OIG supported these  findings, noting that Screening Supervisors and Screening Managers  needed to be more attentive in identifying and correcting improper or  inadequate screener performance.
TSA recognizes the importance of Screener Supervisors and has  established training programs to enhance their performance and  effectiveness. In September 2003, we reported that TSA had begun  working with the Department of Agriculture Graduate School to tailor the  school’s off-the-shelf supervisory course to meet the specific needs of  Screening Supervisors, and had started training the existing supervisors at  that time through this course until the customized course was fielded.  According to TSA’s training records, as of September 2004, about 3,800  Screening Supervisors had completed the course—approximately  92 percent of current Screening Supervisors. In response to our survey,  one FSD noted that the supervisory training was long overdue because  most of the supervisors had no prior federal service or, in some cases, no  leadership experience. This FSD also noted that “leadership and  supervisory skills should be continuously honed; thus, the development of  our supervisors should be an extended and sequential program with  numerous opportunities to develop skills—not just a one-time class.”
In addition to the Department of Agriculture Graduate School course,  TSA’s Online Learning Center includes over 60 supervisory courses  designed to develop leadership and coaching skills. In April 2004, TSA  included in the Online Learning Center a Web-based technical training  course—required for all Lead Screeners and Screening Supervisors. This  course covers technical issues, such as resolving alarms at screening  checkpoints. TSA introduced this course to the field in March 2004, and  although the course is a requirement, TSA officials stated that they have  not set goals for when all Lead Screeners and Screening Supervisors  should have completed the course. In June 2004, TSA training officials  stated that a second supervisor technical course was planned for  development and introduction later in 2004. However, in December 2004,  the training officials stated that planned funding for supervisory training  may be used to support other TSA initiatives. The officials acknowledged  that this would reduce TSA’s ability to provide the desired type and level  of supervisory training to its Lead Screener, Screening Supervisor, and  Screening Manager staff. TSA plans to revise its plans to provide Lead  Screener, Screening Supervisor, and Screening Manager training based on  funding availability.
Although TSA has developed leadership and technical courses for  Screening Supervisors, many FSDs, in response to our general survey,  identified additional types of training needed to enhance screener  supervision. Table 2 provides a summary of the additional training needs  that FSDs reported.
TSA training officials stated that the Online Learning Center provides  several courses that cover these topics. Such courses include    Situation Leadership II;    Communicating with Difficult People: Handling Difficult Co-Workers;    Team Participation: Resolving Conflict in Teams;    Employee Performance: Resolving Conflict;    High Impact Hiring;    Team Conflict: Overcoming Conflict with Communication;    Correcting Performance Problems: Disciplining Employees;    Team Conflict: Working in Diversified Teams;    Correcting Performance Problems: Identifying Performance Problems;    Resolving Interpersonal Skills;    Grammar, Skills, Punctuation, Mechanics and Word Usage; and    Crisis in Organizations: Managing Crisis Situations.
TSA training officials acknowledged that for various reasons FSDs might  not be aware that the supervisory and leadership training is available. For  example, FSDs at airports without high-speed Internet/intranet access to  the Online Learning Center might not have access to all of these courses. It  is also possible that certain FSDs have not fully browsed the contents of  the Online Learning Center and therefore are not aware that the training is  available. Furthermore, officials stated that online learning is relatively  new to government and senior field managers, and some of the FSDs may  expect traditional instructor-led classes rather than online software.
section_title: Some FSDs Reported Impediments to Screeners Receiving Recurrent Training
paragraphs: Some FSDs responded to our general survey that they faced challenges  with screeners receiving recurrent training, including insufficient staffing  to allow all screeners to complete training within normal duty hours and a  lack of high-speed Internet/intranet connectivity at some training facilities.  According to our guide for assessing training, to foster an environment  conducive to effective training and development, agencies must take  actions to provide sufficient time, space, and equipment to employees to  complete required training. TSA has set a requirement for 3 hours of  recurrent training per week averaged over a quarter year, for both full-time  and part-time screeners. However, FSDs for about 18 percent (48 of 263)  of the airports in our airport-specific survey reported that screeners  received less than 9 to 12 hours of recurrent training per month.  Additionally, FSDs for 48 percent (125 of 263) of the airports in the survey  reported that there was not sufficient time for screeners to receive  recurrent training within regular work hours.
At 66 percent of those airports where the FSD reported that there was not  sufficient time for screeners to receive recurrent training within regular  work hours, the FSDs cited screener staffing shortages as the primary  reason. We reported in February 2004 that FSDs at 11 of the 15 category X  airports we visited reported that they were below their authorized staffing  levels because of attrition and difficulties in hiring new staff. In addition,  three of these FSDs noted that they had never been successful in hiring up  to the authorized staffing levels. We also reported in February 2004 that  FSDs stated that because of staffing shortages, they were unable to let  screeners participate in training because it affected the FSD’s ability to  provide adequate coverage at the checkpoints. In response to our survey,  FSDs across all categories of airports reported that screeners must work  overtime in order to participate in training. A September 2004 DHS OIG  report recommended that TSA examine the workforce implications of the  3-hour training requirement and take steps to correct identified  imbalances in future workforce planning to ensure that all screeners are  able to meet the recurrent training standard. The 3-hours-per-week  training standard represents a staff time commitment of 7.5 percent of full- time and between 9 and 15 percent of part-time screeners’ nonovertime  working hours. TSA headquarters officials have stated that because the 3- hours-per-week requirement is averaged over a quarter, it provides  flexibility to account for the operational constraints that exist at airports.  However, TSA headquarters officials acknowledged that many airports are  facing challenges in meeting the 3-hour recurrent training requirement.  TSA data for the fourth quarter of fiscal year 2004 reported that 75 percent  of airports were averaging less than 3 hours of recurrent training per week  per screener. The current screener staffing model, which is used to  determine the screener staffing allocations for each airport, does not take  the 3-hours-per-week recurrent training requirement into account.  However, TSA headquarters officials said that they are factoring this  training requirement into their workforce planning efforts, including the  staffing model currently under development.
Another barrier to providing recurrent training is the lack of high-speed  Internet/intranet access at some of TSA’s training locations. TSA officials  acknowledged that many of the features of the Online Learning Center,  including some portions of the training modules and some Online Learning  Center course offerings, are difficult or impossible to use in the absence of  high-speed Internet/intranet connectivity. As one FSD put it, “the delayed  deployment of the high-speed Internet package limits the connectivity to  TSA HQ for various online programs that are mandated for passenger  screening operations including screener training.” One FSD for a category  IV airport noted the lack of a high-speed connection for the one computer  at an airport he oversees made the Online Learning Center “nearly  useless.”
TSA began deploying high-speed access to its training sites and  checkpoints in May 2003 and has identified high-speed connectivity as  necessary in order to deliver continuous training to screeners. TSA’s July  2003 Performance Improvement Study recommended accelerating high- speed Internet/intranet access in order to provide quick and systematic  distribution of information and, thus, reduce uncertainty caused by the  day-to-day changes in local and national procedures and policy. In  October 2003, TSA reported plans to have an estimated 350 airports online  with high-speed connectivity within 6 months. However, in June 2004, TSA  reported that it did not have the resources to reach this goal.
TSA records show that as of October 2004, TSA had provided high-speed  access for training purposes to just 109 airports, where 1,726 training  computers were fully connected. These 109 airports had an authorized  staffing level of over 24,900 screeners, meaning that nearly 20,100  screeners (45 percent of TSA’s authorized screening workforce) still did  not have high-speed Internet/intranet access to the Online Learning Center  at their training facility. In October 2004, TSA officials stated that TSA’s  Office of Information Technology had selected an additional 16 airport  training facilities with a total of 205 training computers to receive high- speed connectivity by the end of December 2004. As of January 19, 2005,  TSA was unable to confirm that these facilities had received high-speed  connectivity. Additionally, they could not provide a time frame for when  they expected to provide high-speed connectivity to all airport training  facilities because of funding uncertainties. Furthermore, TSA does not  have a plan for prioritizing and scheduling the deployment of high-speed  connectivity to all airport training facilities once funding is available.  Without a plan, TSA’s strategy and timeline for implementing connectivity  to airport training facilities is unclear. The absence of such a plan limits  TSA’s ability to make prudent decisions about how to move forward with  deploying connectivity once funding is available. Figure 5 shows the  percentage of airports reported to have high-speed connectivity for their  training computers by category of airport as of October 2004.
To mitigate airport connectivity issues in the interim, on April 1, 2004, TSA  made the Online Learning Center courses accessible through public  Internet connections, which enable screeners to log on to the Online  Learning Center from home, a public library, or other locations. However,  TSA officials stated that the vast majority of screeners who have used the  Online Learning Center have logged in from airports with connectivity at  their training facilities. TSA also distributes new required training  products using multiple delivery channels, including written materials and  CD-ROMs for those locations where access to the Online Learning Center  is limited. Specifically, TSA officials stated that they provided airports  without high-speed connectivity with CD-ROMs for the 50 most commonly  used optional commercial courseware titles covering topics such as  information technology skills, customer service, and teamwork.  Additionally, officials stated that as technical courses are added to the  Online Learning Center, they are also distributed via CD-ROM and that  until full connectivity is achieved, TSA will continue to distribute new  training products using multiple delivery channels.
section_title: TSA Lacks Adequate Controls to Provide Reasonable Assurance That Screeners Receive Required Training
paragraphs: Because of a lack of internal controls, TSA cannot provide reasonable  assurance that screeners are completing required training. First, TSA  policy does not clearly define responsibility for ensuring that screeners  have completed all required training. Additionally, TSA has no formally  defined policies or procedures for documenting completion of remedial  training, or a system designed to facilitate review of this documentation  for purposes of monitoring. Further, TSA headquarters does not have  formal policies and procedures for monitoring completion of basic training  and lacks procedures for monitoring recurrent training. Finally, at airports  without high-speed connectivity, training records must be entered  manually, making it challenging for some airports to keep accurate and up- to-date training records.
subsections: section_title: TSA Policy Does Not Clearly Define Responsibility for Ensuring That Screeners Are in Compliance with Training Requirements
paragraphs: TSA’s current guidance for FSDs regarding the training of the screener  workforce does not clearly identify responsibility for tracking and  ensuring compliance with training requirements. In a good control  environment, areas of authority and responsibility are clearly defined and  appropriate lines of reporting are established. In addition, internal  control standards also require that responsibilities be communicated  within an organization. The Online Learning Center provides TSA with a  standardized, centralized tool capable of maintaining all training records  in one system. It replaces an ad hoc system previously used during initial  rollout of federalized screeners in which contractors maintained training  records. A February 2004 management directive states that FSDs are  responsible for ensuring the completeness, accuracy, and timeliness of  training records maintained in the Online Learning Center for their  employees. For basic and recurrent training, information is to be entered  into the Online Learning Center within 30 days of completion of the  training activity. However, the directive does not clearly identify who is  responsible for ensuring that employees comply with training  requirements. Likewise, a December 2003 directive requiring that  screeners complete 3 hours of training per week averaged over a quarter  states that FSDs are responsible for ensuring that training records for each  screener are maintained in the Online Learning Center. Although both  directives include language that requires FSDs to ensure training records  are maintained in the Online Learning Center, neither specifies whether  FSDs or headquarters officials are responsible for ensuring compliance  with the basic, recurrent, and remedial training requirements. Even so,  TSA headquarters officials told us that FSDs are ultimately responsible for  ensuring screeners receive required training. However, officials provided  no documentation clearly defining this responsibility. Without a clear  designation of responsibility for monitoring training completion, this  function may not receive adequate attention, leaving TSA unable to  provide reasonable assurance that its screening workforce receives  required training. In April 2005, TSA officials responsible for training  stated that they were updating the February 2004 management directive on  training records to include a specific requirement for FSDs to ensure that  screeners complete required training. They expect to release the revised  directive in May 2005.
section_title: TSA Lacks Formal Policies and Procedures for Monitoring Completion of Required Training
paragraphs: TSA has not established and documented policies and procedures for  monitoring completion of basic and recurrent training. Internal control  standards advise that internal controls should be designed so that  monitoring is ongoing and ingrained in agency operations. However, TSA  headquarters officials stated that they have no formal policy for  monitoring screeners’ completion of basic training. They also stated that  they have neither informal nor formal procedures for monitoring the  completion of screeners’ recurrent training requirements, and  acknowledged that TSA policy does not address what is to occur if a  screener does not meet the recurrent training requirement. Officials  further stated that individual FSDs have the discretion to determine what  action, if any, to take when screeners do not meet this requirement.
In July 2004, TSA training officials stated that headquarters staff recently  began running a report in the Online Learning Center to review training  records to ensure that newly hired screeners had completed required basic  training. In addition, they stated that in June 2004, they began generating  summary-level quarterly reports from the Online Learning Center to  quantify and analyze hours expended for recurrent screener training.  Specifically, TSA training officials stated that reports showing airport-level  compliance with the 3-hour recurrent requirement were generated for the  third and fourth quarters of fiscal year 2004 and delivered to the Office of  Aviation Operations for further analysis and sharing with the field.  However, Aviation Operations officials stated that they did not use these  reports to monitor the status of screener compliance with the 3-hour  recurrent training requirement and do not provide them to the field unless  requested by an FSD. TSA training officials said that while headquarters  intends to review recurrent training activity on an ongoing basis at a  national and airport level, they view FSDs and FSD training staff as  responsible for ensuring that individuals receive all required training.  Further, they acknowledged that weaknesses existed in the reporting  capability of the Online Learning Center and stated that they plan to  upgrade the Online Learning Center with improved reporting tools by the  end of April 2005. Without clearly defined policies and procedures for  monitoring the completion of training, TSA lacks a structure to support  continuous assurance that screeners are meeting training requirements.
section_title: TSA Lacks Clearly Defined Policies and Procedures for Documenting Completion of Remedial Training
paragraphs: TSA has not established clear policies and procedures for documenting  completion of required remedial training. The Standards for Internal  Control state that agencies should document all transactions and other  significant events and should be able to make this documentation readily  available for examination. A TSA training bulletin dated October 15, 2002,  specifies that when remedial training is required, FSDs must ensure the  training is provided and a remedial training reporting form is completed  and maintained with the screener’s local records. However, when we  asked to review these records, we found confusion as to how and where  they were to be maintained. TSA officials stated that they are waiting for a  decision regarding how to maintain these records because of their  sensitive nature. In the meantime, where and by whom the records should  be maintained remains unclear.
In September 2004, officials from TSA’s OIAPR—responsible for  conducting covert testing—stated that they maintain oversight to ensure  screeners requiring remedial training receive required training by  providing a list of screeners that failed covert testing and therefore need  remedial training to TSA’s Office of Aviation Operations. Aviation  Operations is then to confirm via memo that each of the screeners has  received the necessary remedial training and report back to OIAPR.  Accordingly, we asked TSA for all Aviation Operations memos confirming  completion of remedial training, but we were only able to obtain 1 of the  12 memos.
In addition, during our review, we asked to review the remedial training  reporting forms at five airports to determine whether screeners received  required training, but we encountered confusion about requirements for  maintaining training records and inconsistency in record keeping on the  part of local TSA officials. Because of the unclear policies and procedures  for recording completion of remedial training, TSA does not have adequate  assurance that screeners are receiving legislatively mandated remedial  training.
section_title: Lack of High-Speed Connectivity Limits TSA’s Ability to Document and Track the Completion of Screener Training
paragraphs: Although training computers with high-speed Internet/intranet  connectivity automatically record completion of training in the Online  Learning Center, airports without high-speed access at their training  facility must have these records entered manually. The February 2004  management directive that describes responsibility for entering training  records into the Online Learning Center also established that all TSA  employees are required to have an official TSA training record in the  Online Learning Center that includes information on all official training  that is funded wholly or in part with government funds. Without high- speed access, TSA officials stated that it can be a challenge for airports to  keep the Online Learning Center up to date with the most recent training  records. TSA headquarters officials further stated that when they want to  track compliance with mandatory training such as ethics or civil rights  training, they provide the Training Coordinators with a spreadsheet on  which to enter the data rather than relying on the Online Learning Center.  As one FSD told us, without high-speed connectivity at several of the  airports he oversees, “this is very time consuming and labor intensive and  strains my limited resources.” The difficulty that airports encounter in  maintaining accurate records when high-speed access is absent could  compromise TSA’s ability to provide reasonable assurance that screeners  are receiving mandated basic and remedial training.
section_title: Progress Has Been Made in Implementing Tools to Measure and Enhance Screener Performance, but Key Performance Targets Have Not Been Finalized
paragraphs: TSA has improved its efforts to measure and enhance screener  performance. However, these efforts have primarily focused on passenger  screening rather than checked baggage screening, and TSA has not yet  finalized performance targets for several key performance measures. For  example, TSA has increased the amount of covert testing it performs at  airports. These tests have identified that, overall, weaknesses and  vulnerabilities continue to exist in the passenger and checked baggage  screening systems. TSA also enabled FSDs to conduct local covert testing,  fully deployed the Threat Image Projection (TIP) system to passenger  screening checkpoints at commercial airports nationwide, and completed  the 2003/2004 annual screener recertification program for all eligible  screeners. However, not all of these performance measurement and  enhancement tools are available for checked baggage screening.  Specifically, TIP is not currently operational at checked baggage screening  checkpoints, and the recertification program does not include an image  recognition component for checked baggage screeners. However, TSA is  taking steps to address the overall imbalance in passenger and checked  baggage screening performance data, including working toward  implementing TIP for checked baggage screening and developing an image  recognition module for checked baggage screener recertification. To  enhance screener and screening system performance, TSA has also  conducted a passenger screener performance improvement study and  subsequently developed an improvement plan consisting of multiple action  items, many of which TSA has completed. However, TSA has not  conducted a similar study for checked baggage screeners. In addition, TSA  has established over 20 performance measures for the passenger and  checked baggage screening systems as well as two performance indexes  (one for passenger and one for checked baggage screening). However,  TSA has not established performance targets for each of the component  indicators within the indexes, such as covert testing. According to The  Office of Management and Budget, performance goals are target levels of  performance expressed as a measurable objective, against which actual  achievement can be compared. Performance goals should incorporate  measures (indicators used to gauge performance); targets (characteristics  that tell how well a program must accomplish the measure), and time  frames. Without these targets, TSA’s performance management system,  and these performance indexes, specifically, may not provide the agency  with the complete information necessary to assess achievements and  make decisions about where to direct performance improvement efforts.  Although TSA has not yet established performance targets for each of the  component indicators, TSA plans to finalize performance targets for the  indicators by the end of fiscal year 2005.
subsections: section_title: TSA Has Increased Its Covert Testing and Allowed Local Covert Testing at Passenger Screening Checkpoints
paragraphs: TSA headquarters has increased the amount of covert testing it performs  and enabled FSDs to conduct additional local covert testing at passenger  screening checkpoints. TSA’s OIAPR conducts unannounced covert tests  of screeners to assess their ability to detect threat objects and to adhere to  TSA-approved procedures. These tests, in which undercover OIAPR  inspectors attempt to pass threat objects through passenger screening  checkpoints and in checked baggage, are designed to measure  vulnerabilities in passenger and checked baggage screening systems and  to identify systematic problems affecting performance of screeners in the  areas of training, policy, and technology. TSA considers its covert testing  as a “snapshot” of a screener’s ability to detect threat objects at a  particular point in time and as one of several indicators of system wide  screener performance.
OIAPR conducts tests at passenger screening checkpoints and checked  baggage screening checkpoints. According to OIAPR, these tests are  designed to approximate techniques terrorists might use. These covert test  results are one source of data on screener performance in detecting threat  objects as well as an important mechanism for identifying areas in  passenger and checked baggage screening needing improvement. In  testimony before the 9/11 Commission, the Department of Transportation  Inspector General stated that emphasis must be placed on implementing  an aggressive covert testing program to evaluate operational effectiveness  of security systems and equipment.
Between September 10, 2002, and September 30, 2004, OIAPR conducted a  total of 3,238 covert tests at 279 different airports. In September 2003, we  reported that OIAPR had conducted limited covert testing but planned to  double the amount of tests it conducted during fiscal year 2004, based on  an anticipated increase in its staff from about 100 full-time equivalents to  about 200 full-time equivalents. TSA officials stated that based on budget  constraints, OIAPR’s fiscal year 2004 staffing authorization was limited to  183 full-time-equivalents, of which about 60 are located in the field.  Despite a smaller than expected staff increase, by the end of the second  quarter of fiscal year 2004, OIAPR had already surpassed the number of  tests it performed during fiscal year 2003, as shown in table 3.
In October 2003, OIAPR committed to testing between 90 and 150 airports  by April 2004 as part of TSA’s short-term screening performance  improvement plan. OAIPR officials stated that this was a onetime goal to  increase testing. This initiative accounts for the spike in testing for the  second quarter of fiscal year 2004.
OIAPR has created a testing schedule designed to test all airports at least  once during a 3-year time frame. Specifically, the schedule calls for OIAPR  to test all category X airports once a year, category I and II airports once  every 2 years, and category III and IV airports at least once every 3 years.
In September 2003 and April 2004, we reported that TSA covert testing  results had identified weaknesses in screeners’ ability to detect threat  objects. More recently, in April 2005, we, along with the DHS OIG,  identified that screener performance continued to be a concern.  Specifically, our analysis of TSA’s covert testing results for tests  conducted between September 2002 and September 2004 identified that  overall, weaknesses still existed in the ability of screeners to detect threat  objects on passengers, in their carry-on bags, and in checked baggage.  Covert testing results in this analysis cannot be generalized either to the  airports where the tests were conducted or to airports nationwide. These  weaknesses and vulnerabilities were identified at airports of all sizes, at  airports with federal screeners, and airports with private-sector screeners.  For the two-year period reviewed, overall failure rates for covert tests  (passenger and checked baggage) conducted at airports using private- sector screeners were somewhat lower than failure rates for the same  tests conducted at airports using federal screeners for the airports tested  during this period. Since these test results cannot be generalized as  discussed above, each airport’s test results should not be considered a  comprehensive measurement of the airport’s performance or any  individual screener’s performance in detecting threat objects, or in  determining whether airports with private sector screeners performed  better than airports with federal screeners.
On the basis of testing data through September 30, 2004, we determined  that OIAPR had performed covert testing at 61 percent of the nation’s  commercial airports. TSA has until September 30, 2005, to test the  additional 39 percent of airports and meet its goal of testing all airports  within 3 years. Although officials stated that they have had to divert  resources from airport testing to conduct testing of other modes and that  testing for other modes of transportation may affect their ability to  conduct airport testing, they still expect to meet the goal.
In February 2004, TSA provided protocols to help FSDs conduct their own  covert testing of local airport passenger screening activities—a practice  that TSA had previously prohibited. Results of local testing using these  protocols are to be entered into the Online Learning Center. This  information, in conjunction with OAIPR covert test results and TIP threat  detection results, is intended to assist TSA in identifying specific training  and performance improvement efforts. In February 2005, TSA released a  general procedures document for local covert testing at checked baggage  screening locations.
TSA officials said that they had not yet begun to use data from local covert  testing to identify training and performance needs because of difficulties  in ensuring that local covert testing is implemented consistently  nationwide. These officials said that after a few months of collecting and  assessing the data, they will have a better idea of how the data can be  used.
section_title: TSA Has Reinstated the Threat Image Projection System and Plans to Use Its Data to Improve the Screening System
paragraphs: TSA has nearly completed the reactivation of the TIP system at airports  nationwide and plans to use data it is collecting to improve the  effectiveness of the passenger screening system. TIP is designed to test  passenger screeners’ detection capabilities by projecting threat images,  including guns, knives, and explosives, onto bags as they are screened  during actual operations. Screeners are responsible for identifying the  threat image and calling for the bag to be searched. Once prompted, TIP  identifies to the screener whether the threat is real and then records the  screener’s performance in a database that could be analyzed for  performance trends. TSA is evaluating the possibility of developing an  adaptive functionality to TIP. Specifically, as individual screeners become  proficient in identifying certain threat images, such as guns or knives, they  will receive fewer of those images and more images that they are less  proficient at detecting, such as improvised explosive devices.
TIP was activated by FAA in 1999 with about 200 threat images, but it was  shut down immediately following the September 11 terrorist attacks  because of concerns that it would result in screening delays and panic, as  screeners might think that they were actually viewing threat objects. In  October 2003, TSA began reactivating and expanding TIP. In April 2004,  we reported that TSA was reactivating TIP with an expanded library of  2,400 images at all but one of the more than 1,800 checkpoint lanes  nationwide. To further enhance screener training and performance, TSA  also plans to develop at least an additional 50 images each month.
Despite these improvements, TIP is not yet available for checked baggage  screening. In April 2004, we reported that TSA officials stated that they  were working to resolve technical challenges associated with using TIP for  checked baggage screening on EDS machines and have started EDS TIP  image development. The DHS OIG reported in September 2004 that TSA  plans to implement TIP on all EDS machines at checked baggage stations  nationwide in fiscal year 2005. However, in December 2004, TSA officials  stated that because of severe budget reductions, TSA will be unable to  begin implementing a TIP program for checked baggage in fiscal year 2005.  They did not specify when such a program might begin.
TSA plans to use TIP data to improve the passenger screening system in  two ways. First, TIP data can be used to measure screener threat detection  effectiveness by different threats. Second, TSA plans to use TIP results to  help identify specific recurrent training needs within and across airports  and to tailor screeners’ recurrent training to focus on threat category areas  that indicate a need for improvement. TSA considers February 2004 as the  first full month of TIP reporting with the new library of 2,400 images. TSA  began collecting these data in early March 2004 and is using the data to  determine more precisely how they can be used to measure screener  performance in detecting threat objects and to determine what the data  identify about screener performance.
TSA does not currently plan to use TIP data as an indicator of individual  screener performance because TSA does not believe that TIP by itself  adequately reflects a screener’s performance. Nevertheless, in April 2004,  TSA gave FSDs the capability to query and analyze TIP data in a number of  ways, including by screener, checkpoint, and airport. FSDs for over  60 percent of the airports included in our airport-specific survey stated  that they use or plan to use TIP data as a source of information in their  evaluations of individual screener performance. Additionally, FSDs for  50 percent of the airports covered in our survey reported using data  generated by TIP to identify specific training needs for individual  screeners.
In September 2004, the DHS OIG reported that TSA is assessing the cost  and feasibility of modifying TIP so that it recognizes and responds to  specific threat objects with which individual screeners are most and least  competent in detecting, over time. This feature would increase the utility  of TIP as a training tool. The DHS OIG also reported that TSA is  considering linking TIP over a network, which would facilitate TSA’s  collection, analysis, and information-sharing efforts around TIP user  results. The report recommended that TSA continue to pursue each of  these initiatives, and TSA agreed. However, in December 2004, TSA  officials stated that the availability of funding will determine whether or  not they pursue these efforts further.
section_title: TSA Has Completed Its First Round of Screener Recertification Testing, but Testing for Checked Baggage Screeners Is Not as Comprehensive as for Passenger Screeners
paragraphs: TSA has completed its first round of the screener recertification program,  and the second round is now under way. However, TSA does not currently  include an image recognition component in the test for checked baggage  screener recertification. ATSA requires that each screener receive an  annual proficiency review to ensure he or she continues to meet all  qualifications and standards required to perform the screening function. In  September 2003, we reported that TSA had not yet implemented this  requirement. To meet this requirement, TSA established a recertification  program, and it began recertification testing in October 2003 and  completed the testing in March 2004. The first recertification program  was composed of two assessment components, one of screeners’  performance and the other of screeners’ knowledge and skills. During the  performance assessment component of the recertification program,  screeners are rated on both organizational and individual goals, such as  maintaining the nation’s air security, vigilantly carrying out duties with  utmost attention to tasks that will prevent security threats, and  demonstrating the highest levels of courtesy to travelers to maximize their  levels of satisfaction with screening services. The knowledge and skills  assessment component consists of three modules: (1) knowledge of  standard operating procedures, (2) image recognition, and (3) practical  demonstration of skills. Table 4 provides a summary of these three  modules.
To be recertified, screeners must have a rating of “met” or “exceeded”  standards on their annual performance assessments and have passed each  of the applicable knowledge and skills modules. Screeners that failed any  of the three modules were to receive study time or remedial training as  well as a second opportunity to take and pass the modules. Screeners who  failed on their second attempt were to be removed from screening duties  and subject to termination. Screeners could also be terminated for  receiving a rating of below “met” standards.
TSA completed its analysis of the recertification testing and performance  evaluations in May 2004. TSA’s analysis shows that less than 1 percent of  screeners subject to recertification failed to complete this requirement.  Figure 6 shows the recertification results.
Across all airports screeners performed well on the recertification testing.  Over 97 percent of screeners passed the standard operating procedures  test on their first attempt. Screeners faced the most difficulty on the  practical demonstration of skills component. However, following  remediation, 98.6 percent of the screeners who initially failed this  component passed on their second attempt. Table 5 shows the results of  the recertification testing by module.
As shown in table 6, screeners hired as checked baggage screeners were  not required to complete the image recognition module in the first round  of the recertification testing.
In addition, during the first year of recertification testing, which took place  from October 2003 through May 2004, dual-function screeners who were  actively working as both passenger and checked baggage screeners were  required to take only the recertification test for passenger screeners. They  were therefore not required to take the recertification testing modules  required for checked baggage, even though they worked in that capacity.
TSA began implementing the second annual recertification testing in  October 2004 and plans to complete it no later than June 2005. This  recertification program includes components for dual-function screeners.  However, TSA still has not included an image recognition module for  checked baggage screeners—which would include dual-function screeners  performing checked baggage screening. TSA officials stated that a decision  was made to not include an image recognition module for checked  baggage screeners during this cycle because not all checked baggage  screeners would have completed training on the onscreen resolution  protocol by the time recertification testing was conducted at their  airports. In December 2004, TSA officials stated that they plan on  developing an image recognition module for checked baggage and dual- function screeners, and that this test should be available for next year’s  recertification program. The development and implementation of the  image recognition test will be contingent, they stated, upon the availability  of funds.
section_title: TSA Has Identified and Implemented Efforts to Enhance Screener Performance, but These Efforts Primarily Focused on Passenger Screeners
paragraphs: TSA has implemented a number of improvements designed to enhance  screener performance, based on concerns it identified in a July 2003  Passenger Screener Performance Improvement Study and  recommendations from OIAPR. To date, however, these efforts have  primarily focused on the performance of passenger screeners, and TSA  has not yet undertaken a comparable performance study for checked  baggage screeners. The Passenger Screener Performance Improvement  Study relied in part on the findings of OIAPR’s covert testing. At the time  the study was issued, OIAPR had conducted fewer than 50 tests of  checked baggage screeners. The July 2003 study focused on and included  numerous recommendations for improving the performance of passenger  screeners, but recommended waiting to analyze the performance of  checked baggage screeners until some time after implementation of the  recommendations, some of which TSA indicated, also applied to checked  baggage screeners. TSA officials told us that this analysis has been  postponed until they have reviewed the impact of implementing the  recommendations on passenger screening performance.
In October 2003, to address passenger screener performance deficiencies  identified in the Screener Performance Improvement Study, TSA  developed a Short-Term Screening Performance Improvement Plan. This  plan included specific action items in nine broad categories—such as  enhance training, increase covert testing, finish installing TIP, and  expedite high-speed connectivity to checkpoints and training computers— that TSA planned to pursue to provide tangible improvements in passenger  screener performance and security (see app. IV for additional information  on the action items). In June 2004, TSA reported that it had completed 57  of the 62 specific actions. As of December 2004, two of these actions still  had not been implemented—full deployment of high-speed connectivity  and a time and attendance package—both of which continue to be  deferred pending the identification of appropriate resources.
In addition to the Performance Improvement Study and corresponding  action plans, TSA’s OIAPR makes recommendations in its reports on  covert testing results. These recommendations address deficiencies  identified during testing and are intended to improve screening  effectiveness. As of December 2004, OIAPR had issued 18 reports to TSA  management on the results of its checkpoint and checked baggage covert  testing. These reports include 14 distinct recommendations, some of  which were included in TSA’s screener improvement action plan. All but  two of these reports included recommendations on corrective actions  needed to enhance the effectiveness of passenger and checked baggage  screening.
section_title: TSA Has Established Screening Performance Measures and Indexes but Has Not Established Key Performance Targets
paragraphs: TSA has established performance measures, indexes, and targets for the  passenger and checked baggage screening systems, but has not  established targets for the various components of the screening indexes.  The Government Performance and Results Act of 1993 provides, among  other things, that federal agencies establish program performance  measures, including the assessment of relevant outputs and outcomes of  each program activity. Performance measures are meant to cover key  aspects of performance and help decision makers to assess program  accomplishments and improve program performance. A performance  target is a desired level of performance expressed as a tangible,  measurable objective, against which actual achievement will be compared.  By analyzing the gap between target and actual levels of performance,  management can target those processes that are most in need of  improvement, set improvement goals, and identify appropriate process  improvements or other actions.
An April 2004 consultant study commissioned by TSA found that FSDs and  FSD staffs generally believed the lack of key performance indicators  available to monitor passenger and checked baggage screening  performance represented a significant organizational weakness. Since  then, TSA has established over 20 performance measures for the  passenger and checked baggage screening systems. For example, TSA  measures the percentage of screeners meeting a threshold score on the  annual recertification testing on their first attempt, the percentage of  screeners scoring above the national standard level on TIP performance,  and the number of passengers screened, by airport category.
TSA also has developed two performance indexes to measure the  effectiveness of the passenger and checked baggage screening systems.  These indexes measure overall performance through a composite of  indicators and are derived by combining specific performance measures  relating to passenger and checked baggage screening, respectively.  Specifically, these indexes measure the effectiveness of the screening  systems through machine probability of detection and covert testing  results; efficiency through a calculation of dollars spent per passenger or  bag screened; and customer satisfaction through a national poll, customer  surveys, and customer complaints at both airports and TSA’s national call  center. According to TSA officials, the agency has finalized targets for the  two overall indexes, but these targets have not yet been communicated  throughout the agency. Further, TSA plans to provide the FSDs with only  the performance index score, not the value of each of the components,  because the probabilities of detection are classified as secret and TSA is  concerned that by releasing components, those probabilities could be  deduced. Table 7 summarizes the components of the performance  indexes developed by TSA.
TSA has not yet established performance targets for the various  components of the screening indexes, including performance targets for  covert testing (person probability of detection). TSA’s strategic plan states  that the agency will use the performance data it collects to make tactical  decisions based on performance. The screening performance indexes  developed by TSA can be a useful analysis tool, but without targets for  each component of the index, TSA will have difficulty performing  meaningful analyses of the parts that add up to the index. For example,  without performance targets for covert testing, TSA will not have  identified a desired level of performance related to screener detection of  threat objects. Performance targets for covert testing would enable TSA to  focus its improvement efforts on areas determined to be most critical, as  100 percent detection capability may not be attainable. In January 2005,  TSA officials stated that the agency plans to track the performance of  individual index components and establish performance targets against  which to measure these components. They further stated that they are  currently collecting and analyzing baseline data to establish these targets  and plan to finalize them by the end of fiscal year 2005.
section_title: Conclusions
paragraphs: It has been over 2 years since TSA assumed responsibility for passenger  and checked baggage screening operations at the nation’s commercial  airports. TSA has made significant accomplishments over this period in  meeting congressional mandates related to establishing these screening  operations. With the congressional mandates now largely met, TSA has  turned its attention to assessing and enhancing the effectiveness of its  passenger and checked baggage screening systems. An important tool in  enhancing screener performance is ongoing training. As threats and  technology change, the training and development of screeners to ensure  they have the competencies—knowledge, skills, abilities, and behaviors— needed to successfully perform their screening functions become vital to  strengthening aviation security. Without addressing the challenges to  delivering ongoing training, including installing high-speed connectivity at  airport training facilities, TSA may have difficulty maintaining a screening  workforce that possesses the critical skills needed to perform at a desired  level. In addition, without adequate internal controls designed to help  ensure screeners receive required training that are also communicated  throughout the agency, TSA cannot effectively provide reasonable  assurances that screeners receive all required training. Given the  importance of the Online Learning Center in both delivering training and  serving as the means by which the completion of screener training is  documented, TSA would benefit from having a clearly defined plan for  prioritizing the deployment of high-speed Internet/intranet connectivity to  all airport training facilities. Such a plan would help enable TSA to move  forward quickly and effectively in deploying high-speed connectivity once  funding is available.
Additionally, history demonstrates that U.S. commercial aircraft have long  been a target for terrorist attacks through the use of explosives carried in  checked baggage, and covert testing conducted by TSA and DHS OIG have  identified that weaknesses and vulnerabilities continue to exist in the  passenger and checked baggage screening systems, including the ability of  screeners to detect threat objects. While covert test results provide an  indicator of screening performance, they cannot solely be used as a  comprehensive measure of any airport’s screening performance or any  individual screener’s performance, or in determining the overall  performance of federal versus private-sector screening. Rather, these data  should be considered in the larger context of additional performance data,  such as TIP and recertification test results, when assessing screener  performance. While TSA has undertaken efforts to measures and  strengthen performance, these efforts have primarily focused on  passenger screening and not on checked baggage screening. TSA’s plans  for implementing TIP for checked baggage screening, and establishing an  image recognition component for the checked baggage screeners  recertification testing—plans made during the course of our review— represent significant steps forward in its efforts to strengthen checked  baggage screening functions. Additionally, although TSA has developed  passenger and checked baggage screening effectiveness measures, the  agency has not yet established performance targets for the individual  components of these measures. Until such targets are established, it will  be difficult for TSA to draw more meaningful conclusions about its  performance and how to most effectively direct its improvement efforts.  For example, performance targets for covert testing would enable TSA to  focus its improvement efforts on areas determined to be most critical, as  100 percent detection capability may not be attainable. We are encouraged  by TSA’s recent plan to establish targets for the individual components of  the performance indexes. This effort, along with the additional  performance data TSA plans to collect on checked baggage screening  operations, should assist TSA in measuring and enhancing screening  performance and provide TSA with more complete information with which  to prioritize and focus its screening improvement efforts.
section_title: Recommendations for Executive Action
paragraphs: To help ensure that all screeners have timely and complete access to  screener training available in the Online Learning Center and to help  provide TSA management with reasonable assurance that all screeners are  receiving required passenger and checked baggage screener training, we  recommend that the Secretary of the Department of Homeland Security  direct the Assistant Secretary, Transportation Security Administration, to  take the following two actions:    develop a plan that prioritizes and schedules the deployment of high-speed  Internet/intranet connectivity to all TSA’s airport training facilities to help  facilitate the delivery of screener training and the documentation of  training completion, and    develop internal controls, such as specific directives, clearly defining  responsibilities for monitoring and documenting the completion of  required training, and clearly communicate these responsibilities  throughout the agency.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DHS for review and comment. On  February 4, 2005, we received written comments on the draft report, which  are reproduced in full in appendix V. DHS generally concurred with the  findings and recommendations in the report, and agreed that efforts to  implement our recommendations are critical to successful passenger and  checked baggage screening training and performance. With regard to our  recommendation that TSA develop a plan that prioritizes and schedules  the deployment of high-speed Internet/intranet connectivity to all TSA’s  airport training facilities, DHS stated that TSA has developed such a plan.  However, although we requested a copy of the plan several times during  our review and after receiving written comments from DHS, TSA did not  provide us with a copy of the plan. Therefore, we cannot assess the extent  to which the plan DHS referenced in its written comments fulfills our  recommendation. In addition, regarding our recommendation that TSA  develop internal controls clearly defining responsibilities for monitoring  and documenting the completion of required training, and clearly  communicate those responsibilities throughout TSA, DHS stated that it is  taking steps to define responsibility for monitoring the completion of  required training and to insert this accountability into the performance  plans of all TSA supervisors. TSA’s successful completion of these ongoing  and planned activities should address the concerns we raised in this  report. DHS has also provided technical comments on our draft report,  which we incorporated where appropriate.
As agreed with your office, we will send copies of this report to relevant  congressional committees and subcommittees and to the Secretary of the  Department of Homeland Security. We will also make copies available to  others upon request. In addition, the report will be made available at no  charge on GAO’s Web site at http://www.gao.gov.
If you have any questions about this report or wish to discuss it further,  please contact me at (202) 512-8777. Key contributors to this report are  listed in appendix VI.
section_title: Appendix I: Summary of Previous Findings Related to Screener Training and Performance
subsections: section_title: Summary of previous findings related to screener training and performance
paragraphs: The Transportation Security Administration (TSA) had deployed a basic screener  training program and required remedial training but had not fully developed or  deployed a recurrent training program for screeners or supervisors.
TSA had collected little information to measure screener performance in detecting  threat objects.
TSA’s Office of Internal Affairs and Program Review’s (OIAPR) covert testing  was the primary source of information collected on screeners’ ability to detect  threat objects. However, TSA did not consider the covert testing a measure of  screener performance.
TSA was not using the Threat Image Projection system (TIP) but planned to  fully activate the system with significantly more threat images than previously  used in October 2003.
TSA had not yet implemented an annual proficiency review to ensure that  screeners met all qualifications and standards required to perform their  assigned screening functions.
Although little data existed on the effectiveness of passenger screening, TSA was  implementing several efforts to collect performance data.
Aviation Security: Efforts to  Measure Effectiveness and  Address Challenges  planned to double the number of tests it conducted during fiscal year 2004.
TSA only recently began activating TIP on a wide-scale basis and expected it  to be fully operational at every checkpoint at all airports by April 2004.
TSA only recently began implementing the annual recertification program and  did not expect to complete testing at all airports until March 2004.
TSA was developing performance indexes for individual screeners and the  screening system as a whole but had not fully established these indexes. TSA  expected to have them in place by the end of fiscal year 2004.
Aviation Security: Efforts to  Measure Effectiveness and  Strengthen Security  Programs    working with the U.S. Department of Agriculture’s Graduate School to tailor its  off-the-shelf supervisory course to meet the specific training needs of screening  supervisors.
While TSA had taken steps to enhance its screener training programs, staffing  imbalances, and lack of high-speed connectivity at airport training facilities had  made it difficult for screeners at some airports to fully utilize these programs.
Although TSA was making progress in measuring the performance of passenger  screeners, it had collected limited performance data related to its checked  baggage screening operations. However, TSA had begun collecting additional  performance data related to its checked baggage screening operations and  planned to increase these efforts in the future.
As part of its efforts to develop performance indexes, TSA was developing  baseline data for fiscal year 2004 and planned to report the indexes to DHS in  fiscal year 2005.
With the exception of covert testing and recent TIP data, data were not yet  available to assess how well screeners were performing and what steps if any  TSA needed to take to improve performance. Also, TSA was not using TIP as a  formal indicator of screening performance, but instead was using it to identify  individual screener training needs.
section_title: Appendix II: Objectives, Scope, and Methodology
paragraphs: To examine efforts by the Transportation Security Administration to  enhance their passenger and checked baggage screening programs, we  addressed the following questions: (1) What actions has TSA taken to  enhance training for screeners and supervisors? (2) How does TSA  monitor compliance with screener training requirements? (3) What is the  status of TSA’s efforts to assess and enhance screener performance in  detecting threat objects?
To determine how TSA has enhanced training for screeners and  supervisors and how TSA has monitored compliance with screener  training requirements, we obtained and analyzed relevant legislation, as  well as TSA’s training plans, guidance, and curriculum. We reviewed data  from TSA’s Online Learning Center and assessed the reliability of the  Online Learning Center database. We compared TSA’s procedures for  ensuring that screeners receive required training according to Standards  for Internal Controls in the Federal Government. We interviewed TSA  officials from the Office of Workforce Performance and Training and the  Office of Aviation Operations in Arlington, Virginia. At the airports we  visited, we interviewed Federal Security Directors and their staffs, such as  Training Coordinators. We also met with officials from four aviation  associations—the American Association of Airport Executives, Airports  Council International, the Air Transport Association, and the Regional  Airline Association. We did not assess the methods used to develop TSA’s  screener training program, nor did we analyze the contents of TSA’s  curriculum. Although we could not independently verify the reliability of  all of this information, we compared the information with other supporting  documents, when available, to determine data consistency and  reasonableness. We found the data to be sufficiently reliable for our  purposes.
To determine what efforts TSA has taken to assess and to enhance  screener performance in detecting threat objects, we reviewed related  reports from the Department of Transportation and the Department of  Homeland Security (DHS) Inspector General, Congressional Research  Service, and TSA, as well as prior GAO reports. We obtained and reviewed  TSA’s covert test data and results of the annual recertification testing.  (Results of the covert testing are classified and will be the subject of a  separate classified GAO report.) We discussed methods for inputting,  compiling, and maintaining the data with TSA officials. We also assessed  the methodology of TSA’s covert tests and questioned OIAPR officials  about the procedures used to ensure the reliability of the covert test data.  When we found discrepancies between the data OIAPR maintained in  spreadsheets and the data included in the hard copy reports we obtained  from TSA, we worked with OIAPR to resolve the discrepancies. Further,  we visited TSA headquarters to review TSA’s annual recertification testing  modules and discuss TSA’s process for validating the recertification  exams. As a result, we determined that the data provided by TSA were  sufficiently reliable for the purposes of our review. We also reviewed  TSA’s performance measures, targets, and indexes. Finally, we interviewed  TSA headquarters officials from several offices in Arlington, Virginia,  including Aviation Operations, Workforce Performance and Training,  Strategic Management and Analysis, and Internal Affairs and Program  Review.
In addition, in accomplishing our objectives, we also conducted site visits  at select airports nationwide to interview Federal Security Directors and  their staffs and conducted two Web-based surveys of Federal Security  Directors. Specifically, we conducted site visits at 29 airports (13 category  X airports, 9 category I airports, 3 category II airports, 3 category III  airports, and 1 category IV airport) to observe airport security screening  procedures and discuss issues related to the screening process with TSA,  airport, and airline officials. We chose these airports to obtain a cross- section of all airports by size and geographic distribution. In addition, we  selected each of the five contract screening pilot airports. The results from  our airport visits provide examples of screening operations and issues but  cannot be generalized beyond the airports visited because we did not use  statistical sampling techniques in selecting the airports. The category X  airports we visited were Baltimore Washington International Airport,  Boston Logan International Airport, Chicago O’Hare International Airport,  Dallas/Fort Worth International Airport, Denver International Airport,  Washington Dulles International Airport, John F. Kennedy International  Airport, Los Angeles International Airport, Newark Liberty International  Airport, Orlando International Airport, Ronald Reagan Washington  National Airport, San Francisco International Airport, Seattle-Tacoma  International Airport. The category I airports we visited were Burbank- Glendale-Pasadena Airport, John Wayne Airport, Chicago Midway  International Airport, Dallas Love Field, Kansas City International Airport,  Little Rock National Airport, Metropolitan Oakland International Airport,  Portland International Airport, and Tampa International Airport. The  category II airports we visited were Jackson International Airport, Dane  County Regional Airport, and Greater Rochester International Airport. The  category III airports we visited were Idaho Falls Regional Airport, Jackson  Hole Airport, and Orlando Sanford International Airport. The category IV  airport we visited was Tupelo Regional Airport.
Further, we administered two Web-based surveys to all 155 Federal  Security Directors who oversee security at each of the airports falling  under TSA’s jurisdiction. One survey, the general survey, contained  questions covering local and national efforts to train screeners and  supervisors and the status of TSA’s efforts to evaluate screener  performance, including the annual recertification program and TIP. The  second survey attempted to gather more specific airport security  information on an airport(s) under the Federal Security Director’s  supervision. For the airport-specific survey, each Federal Security  Director received one or two surveys to complete, depending on the  number of airports they were responsible for. Where a Federal Security  Director was responsible for more than two airports, we selected the first  airport based on the Federal Security Director’s location and the second  airport to obtain a cross-section of all airports by size and geographic  distribution. In all, we requested information on 265 airports. However,  two airports were dropped from our initial selection because the airlines  serving these airports suspended operations and TSA employees were  redeployed to other airports. As a result our sample size was reduced to  263 airports, which included all 21 category X, and 60, 49, 73, and  60 category I through IV airports respectively. In that we did not use  probability sampling methods to select the sample of airports that were  included in our airport-specific survey, we cannot generalize our findings  beyond the selected airports.
A GAO survey specialist designed the surveys in combination with other  GAO staff knowledgeable about airport security issues. We conducted  pretest interviews with six Federal Security Directors to ensure that the  questions were clear, concise, and comprehensive. In addition, TSA  managers and an independent GAO survey specialist reviewed the survey.
We conducted these Web-based surveys from late March to mid-May 2004.  We received completed general surveys from all 155 Federal Security  Directors and completed airport-specific surveys for all 263 separate  airports for which we sought information, for 100 percent response rates.  The surveys’ results are not subject to sampling errors because all Federal  Security Directors were asked to participate in the surveys and we did not  use probability-sampling techniques to select specific airports. However,  the practical difficulties of conducting any survey may introduce other  errors, commonly referred to as nonsampling errors. For example,  difficulties in how a particular question is interpreted, in the sources of  information that are available to respondents, or in how the data are  entered into a database or were analyzed can introduce unwanted  variability into the survey results. We took steps in the development of the  surveys, the data collection, and the data editing and analysis to minimize  these nonsampling errors. Also, in that these were Web-based surveys  whereby respondents entered their responses directly into our database,  there was little possibility of data entry or transcription error. In addition,  all computer programs used to analyze the data were peer reviewed and  verified to ensure that the syntax was written and executed correctly.
We performed our work from May 2003 through April 2005 in accordance  with generally accepted government auditing standards. Certain  information we obtained and analyzed regarding screener training and  performance are classified or are considered by TSA to be sensitive  security information. Accordingly, the results of our review of this  information are not included in this report.
section_title: Appendix III: TSA Screener Training Tools Designed to Help Improve Screener Performance
subsections: section_title: Provide an informative and effective learning tool to enhance screeners’ skills in the areas of hand-wanding and pat-down searches of passengers.
paragraphs: This tool allows screeners to touch actual improvised explosive  device (IED) components and build their own devices. This  experiential learning will enable screeners to more readily detect real  IEDs during screening. These weapons are also used to assist in  training by using them for live testing conducted by FSD staff.
This tool allows screeners to touch actual firearms and begin to  understand how they can be broken down into various parts. By  understanding this and experiencing it, screeners are better able to  see the components of a firearm during actual screening. These  weapons are also used to assist in training by using them for live  testing conducted by FSD staff.
Deployed January 26, 2004  Maintain and enhance the screeners’ X-ray image operational skills.  Deployed February 5, 2004  Provide a tool that includes about 14,000 image combinations to  practice threat identification.
These teams go into airports where data shows performance needs  attention. The team offers a variety of services to assist in improving  the performance, such as on-the-spot training and consulting  services. Team visits can be initiated by FSDs, Internal Affairs  reports, Quality Assurance trips, or MTAT Supervisors proactively  visiting the airport and FSD.
Site visits completed from  October 2003 through  December 3, 2004:    North Central (37 visits)
South Central (51 visits)
Northeast (25 visits)
Southeast (60 visits)
Western (53 visits)
Improve screener supervisors’ knowledge of federal government and  TSA personnel rules and how to effectively coach and communicate  with employees.
Approximately 3,800 supervisors  have been trained.
Certification of screeners to perform supervisory maintenance tasks  above and beyond operator training.
Provide students with basic skills needed to verify the identity of flying  armed law enforcement officers.
This weekly product brings to light actual cases of weapons being  found by law enforcement, with an explanation of how those weapons  could be used to attack aviation.
Provide interactive, performance based recurrent Web-based training  modules for checked baggage explosive detection systems (EDS).
Improve screener performance by providing an interactive tool  complementary to Hand Held Metal Detector and Pat Down Video  that allows the screener to practice proper techniques and receive  immediate feedback.
Reinforces TSA’s customer service principles and places the  screener in various situations requiring effective customer service  responses.
Provide interactive, performance-based recurrent training modules for  checkpoint and checked baggage operations.
Physical Bag Search Video  Maintain and enhance screeners’ explosive trace detection (ETD)  and physical bag search skills for carry-on and checked baggage.
Provide interactive recurrent Web-based training modules for ETD  and physical bag search.
Provide an interactive, performance-based training tool to enhance  screener’s ability to identify prohibited items.
Provide an informative and effective learning tool to maintain and  enhance the skills of screeners in the areas of persons with  prosthetics.
Provide a tool to practice threat identification with about 10,000,000  image combinations.
Sharing the X-Ray Tutor Version 2 library, this tool will allow  screeners to practice finding threat items using the full capabilities of  the TIP-ready X-ray machines.
Provide an interactive, performance-based tool to convey how the  supervisor is to handle screening situations, handed off by the  screening, following standard operator procedures.
Provide a Web-based training that will engage the student with 3- dimensional representations of the muscular frame, showing proper  lifting techniques and the results of improper techniques.
section_title: Appendix IV: Summary of TSA’s Short-Term Action Items for Strengthening Passenger Screener Performance
section_title: Appendix V: Comments from the Department of Homeland Security
section_title: Appendix VI: GAO Contacts and Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to those named above, David Alexander, Leo Barbour,  Lisa Brown, Elizabeth Curda, Kevin Dooley, Kathryn Godfrey,  David Hooper, Christopher Jones, Stuart Kaufman, Kim Gianopoulos,  Thomas Lombardi, Cady S. Panetta, Minette Richardson, Sidney Schwartz,  Su Jin Yon, and Susan Zimmerman were key contributors to this report.