Aviation Security: Risk, Experience, and Customer Concerns Drive Changes to Airline Passenger Screening Procedures, but Evaluation and Documentation of Proposed Changes Could Be Improved
Released: 2007-05-07T00:00:00

Letter

Background

Passenger Checkpoint Screening System

Passenger screening is a process by which personnel authorized by TSA inspect individuals and property to deter and prevent the carriage of any unauthorized explosive, incendiary, weapon, or other dangerous item onboard an aircraft or into a sterile area. Passenger screening personnel must inspect individuals for prohibited items at designated screening locations. As shown in figure 1, the four passenger screening functions are X-ray screening of property, walk-through metal detector screening of individuals, hand-wand or pat-down screening of individuals, and physical search of property and trace detection for explosives.

Typically, passengers are only subjected to X-ray screening of their carry- on items and screening by the walk-through metal detector. Passengers whose carry-on baggage alarms the X-ray machine, who alarm the walk- through metal detector, or who are designated as selectees—that is, passengers selected by the Computer-Assisted Passenger Prescreening System (CAPPS) or other TSA-approved processes to receive additional screening—are screened by hand-wand or pat-down and have their carry- on items screened for explosives traces or physically searched.

The passenger checkpoint screening system is composed of three elements: the people responsible for conducting the screening of airline passengers and their carry-on items—TSOs, the technology used during the screening process, and the procedures TSOs are to follow to conduct screening. Collectively, these elements help to determine the effectiveness and efficiency of passenger checkpoint screening.

Transportation Security Officers

TSOs screen all passengers and their carry-on baggage prior to allowing passengers access to their departure gates. There are several positions within TSA that perform and directly supervise passenger screening functions. Figure 2 provides a description of these positions.

In May 2005, we reported on TSA’s efforts to train TSOs and to measure and enhance TSO performance. We found that TSA had initiated a number of actions designed to enhance passenger TSO, checked baggage TSO, and supervisory TSO training. However, at some airports TSOs encountered difficulty accessing and completing recurrent (refresher) training because of technological and staffing constraints. We also found that TSA lacked adequate internal controls to provide reasonable assurance that TSOs were receiving legislatively mandated basic and remedial training, and to monitor the status of its recurrent training program. Further, we reported that TSA had implemented and strengthened efforts to collect TSO performance data as part of its overall effort to enhance TSO performance. We recommended that TSA develop a plan for completing the deployment of high-speed Internet/intranet connectivity to all TSA airport training facilities, and establish appropriate responsibilities and other internal controls for monitoring and documenting TSO compliance with training requirements. DHS generally concurred with our recommendations and stated that TSA has taken steps to implement them.

Screening Technology

There are typically four types of technology used to screen airline passengers and their carry-on baggage at the checkpoint: walk-through metal detectors, X-ray machines, hand-held metal detectors, and explosive trace detection (ETD) equipment.

The President’s fiscal year 2007 budget request noted that emerging checkpoint technology will enhance the detection of prohibited items, especially firearms and explosives, on passengers. As of December 2006, TSA plans to conduct operational tests of three types of passenger screening technologies within the next year. TSA has conducted other tests in the past; for example, during fiscal year 2005, TSA operationally tested document scanners, which use explosive trace detection technology to detect explosives residue on passengers’ boarding passes or identification cards. TSA decided not to expand the use of the document scanner, in part because of the extent to which explosives traces had to be sampled manually. TSA also plans to begin operational tests of technology that would screen bottles for liquid explosives. We are currently evaluating the Department of Homeland Security’s and TSA’s progress in planning for, managing, and deploying research and development programs in support of airport checkpoint screening operations. We expect to report our results in August 2007.

Standard Operating Procedures

TSA has developed checkpoint screening standard operating procedures, which are the focus of this report, that establish the process and standards by which TSOs are to screen passengers and their carry-on items at screening checkpoints. Between April 2005 and December 2005, based on available documentation, TSA deliberated 189 proposed changes to passenger checkpoint screening SOPs, 92 of which were intended to modify the way in which passengers and their carry-on items are screened. TSA issued six versions of the passenger checkpoint screening SOPs during this period.

TSA modified passenger checkpoint screening SOPs to enhance the traveling public’s perception of the screening process, improve the efficiency of the screening process, and enhance the detection of prohibited items and suspicious persons. As shown in table 1, 48 of the 92 proposed modifications to passenger checkpoint screening SOPs were implemented, and the types of modifications made or proposed generally fell into one of three categories—customer satisfaction, screening efficiency, and security.

TSA Considered Risk, Experience, and Customer Concerns when Modifying Passenger Screening Procedures, but Could Improve Its Evaluation and Documentation of Proposed Procedures

TSA used various processes between April 2005 and December 2005 to modify passenger checkpoint screening SOPs, and a variety of factors guided TSA’s decisions to modify SOPs. TSA’s processes for modifying SOPs generally involved TSA staff recommending proposed modifications, reviewing and commenting on proposed modifications, and TSA senior leadership making final decisions as to whether proposed modifications should be implemented. During our 9-month review period, TSA officials considered 92 proposed modifications to the way in which passengers and their carry-on items were screened, and 48 were implemented. TSA officials proposed SOP modifications based on risk factors (threat and vulnerability information), day-to-day experiences of airport staff, and concerns and complaints raised by passengers. TSA then made efforts to balance security, efficiency, and customer service when deciding which proposed SOP modifications to implement. Consistent with our prior work that has shown the importance of data collection and analyses to support agency decision making, TSA conducted data collection and analysis for certain proposed SOP modifications that were tested before they were implemented at all airports. Nevertheless, we found that TSA could improve its data collection and analysis to assist the agency in determining whether the proposed procedures would enhance detection or free up TSO resources, when intended. In addition, TSA did not maintain complete documentation of proposed SOP modifications; therefore, we could not fully assess the basis for proposed SOP modifications or the reasons why certain proposed modifications were not implemented. TSA officials acknowledged that it is beneficial to maintain documentation on the reasoning behind decisions to implement or reject SOP modifications deemed significant.

TSA’s Processes for Modifying SOPs Were Driven by Input from TSA Field and Headquarters Staff

Proposed SOP modifications were submitted and reviewed under two processes during our 9-month review period, and for each process, TSA senior leadership made the final decision as to whether the proposed modifications would be implemented. One of the processes TSA used to modify passenger checkpoint screening SOPs involved TSA field staff or headquarters officials, and, to a lesser extent, TSA senior leadership, suggesting ways in which passenger checkpoint screening SOPs could be modified. These suggestions were submitted through various mechanisms, including electronic mail and an SOP panel review conducted by TSA airport personnel. (These methods are described in more detail in app. II.) Eighty-two of the 92 proposed modifications were considered under this process.

If TSA officials determined, based on their professional judgment, that the recommended SOP modifications—whether from headquarters or the field—merited further consideration, or if a specific modification was proposed by TSA senior leadership, the following chain of events occurred: First, the procedures branch of the Office of Security Operations drafted SOP language for each of the proposed modifications. Second, the draft language for each proposed modification was disseminated to representatives of various TSA divisions for review, and the language was revised as needed.

Third, TSA officials tested proposed modifications in the airport operating environment if they found it necessary to: assess the security impact of the proposed modification, evaluate the impact of the modification on the amount of time taken for passengers to clear the checkpoint, measure the impact of the proposed modification on passengers and industry partners, or determine training needs created by the proposed modification.

Fourth, the revised SOP language for proposed modifications was sent to the heads of several TSA divisions for comment.

Fifth, considering the comments of the TSA division heads, the head of the Office of Security Operations or other TSA senior leadership made the final decision as to whether proposed modifications would be implemented.

Another process for modifying passenger checkpoint screening SOPs during our 9-month review period was carried out by TSA’s Explosives Detection Improvement Task Force. The task force was established in October 2005 by the TSA Assistant Secretary to respond to the threat of improvised explosive devices (IED) being carried through the checkpoint. The goal of the task force was to apply a risk-based approach to screening passengers and their baggage in order to enhance TSA’s ability to detect IEDs. The task force developed 13 of the 92 proposed SOP modifications that were considered by TSA between April 2005 and December 2005. The task force solicited and incorporated feedback from representatives of various TSA divisions on these proposed modifications and presented them to TSA senior leadership for review and approval. TSA senior leadership decided that 8 of the 13 proposed modifications should be operationally tested—that is, temporarily implemented in the airport environment for the purposes of data collection and evaluation—to better inform decisions regarding whether the proposed modifications should be implemented. Following the testing of these proposed modifications in the airport environment, TSA senior leadership decided to implement 7 of the 8 operationally tested changes. (The task force’s approach to testing these procedures is discussed in more detail below.) Following our 9-month period of review, the changes that TSA made to its passenger checkpoint screening SOPs in response to the alleged August 2006 liquid explosives terror plot were decided upon by DHS and TSA senior leadership, with some input from TSA field staff, aviation industry representatives, and officials from other federal agencies.

Risk Factors, Day-to-Day Experiences, and Customer Concerns Were the Basis for Proposed SOP Modifications

Based on available documentation, risk factors (i.e., threats to commercial aviation and vulnerability to those threats), day-to-day experiences of airport staff, and complaints and concerns raised by passengers were the basis for TSA staff and officials proposing modifications to passenger checkpoint screening SOPs.

Fourteen of the 92 procedure modifications recommended by TSA staff and officials were based on reported or perceived threats to commercial aviation, and existing vulnerabilities to those threats. For example, the Explosives Detection Improvement Task Force proposed SOP modifications based on threat reports developed by TSA’s Intelligence and Analysis division. Specifically, in an August 2005 civil aviation threat assessment, the division reported that terrorists are likely to seek novel ways to evade U.S. airport security screening. Subsequently, the task force proposed that the pat-down procedure performed on passengers selected for additional screening be revised to include not only the torso area, which is what the previous pat-down procedure entailed, but additional areas of the body such as the legs. The August 2005 threat assessment also stated that terrorists may attempt to carry separate components of an IED through the checkpoint, then assemble the components while onboard the aircraft. To address this threat, the task force proposed a new procedure to enhance TSOs’ ability to search for components of improvised explosive devices. According to TSA officials, threat reports have also indicated that terrorists rely on the routine nature of security measures in order to plan their attacks. To address this threat, the task force proposed a procedure that incorporated unpredictability into the screening process by requiring designated TSOs to randomly select passengers to receive additional search procedures. Following our 9-month review period, TSA continued to use threat information as the basis for proposed modifications to passenger checkpoint screening SOPs. In August 2006, TSA proposed modifications to passenger checkpoint screening SOPs after receiving threat information regarding an alleged terrorist plot to detonate liquid explosives onboard multiple aircraft en route from the United Kingdom to the United States. Regarding vulnerabilities to reported threats, based on the results of TSA’s own covert tests (undercover, unannounced tests), TSA’s Office of Inspection recommended SOP modifications to enhance the detection of explosives at the passenger screening checkpoint.

TSA officials also proposed modifications to passenger checkpoint screening SOPs based on their professional judgment regarding perceived threats to aviation security. For example, an FSD recommended changes to the screening of funeral urns based on a perceived threat. In some cases, proposed SOP modifications appeared to reflect threat information analyzed by TSA officials. For example, TSOs are provided with Threat in the Spotlight, a weekly report that identifies new threats to commercial aviation, examples of innovative ways in which passengers may conceal prohibited items, and pictures of items that may not appear to be prohibited items but actually are. TSOs are also provided relevant threat information during briefings that take place before and after their shifts. In addition, FSDs are provided classified intelligence summaries on a daily and weekly basis, as well as monthly reports of suspicious incidents that occurred at airports nationwide. TSA’s consideration of threat and vulnerability—through analysis of current documentation and by exercising professional judgment—is consistent with a risk-based decision-making approach. As we have reported previously, and DHS and TSA have advocated, a risk-based approach, as applied in the homeland security context, can help to more effectively and efficiently prepare defenses against acts of terrorism and other threats.

TSA headquarters and field staff also based proposed SOP modifications— specifically, 36 of the 92 proposed modifications—on experience in the airport environment. For example, TSA headquarters officials conduct reviews at airports to identify best practices and deficiencies in the checkpoint screening process. During one of these reviews, headquarters officials observed that TSOs were not fully complying with the pat-down procedure. After discussions with TSOs, TSA headquarters officials determined that the way in which TSOs were conducting the procedure was more effective. In addition, TSA senior leadership, after learning that small airports had staffing challenges that precluded them from ensuring that passengers are patted down by TSOs of the same gender, proposed that opposite-gender pat-down screening be allowed at small airports.

Passenger complaints and concerns shared with TSA also served as a basis for proposed modifications during our 9-month review period. Specifically, of the 92 proposed SOP modifications considered during this period, TSA staff and officials recommended 29 modifications based on complaints and concerns raised by passengers. For example, TSA headquarters staff recommended allowing passengers to hold their hair while being screened by the Explosives Trace Portal, after receiving complaints from passengers about eye injuries from hair blowing in their eyes and hair being caught in the doors of the portal.

TSA Balanced Security, Efficiency, and Customer Service when Deciding whether to Implement Proposed SOP Modifications

When deciding whether to implement proposed SOP modifications, TSA officials also made efforts to balance the impact of proposed modifications on security, efficiency, and customer service. TSA’s consideration of these factors reflects the agency’s mission to protect transportation systems while also ensuring the free movement of people and commerce. As previously discussed, TSA sought to improve the security of the commercial aviation system by modifying the SOP for conducting the pat-down search. (TSA identified the modified pat-down procedure as the “bulk-item” pat-down.) When deciding whether to implement the proposed modification, TSA officials considered not only the impact that the bulk- item pat-down procedure would have on security, but also the impact that the procedure would have on screening efficiency and customer service. For example, TSA officials determined that the bulk-item pat-down procedure would not significantly affect efficiency because it would only add a few seconds to the screening process. Following our 9-month review period, TSA continued to make efforts to balance security, efficiency, and customer service when deciding whether to implement proposed SOP modifications, as illustrated by TSA senior leadership’s deliberation on proposed SOP modifications in response to the alleged August 2006 liquid explosives terrorist plot. TSA modified the passenger checkpoint screening SOP four times between August 2006 and November 2006 in an effort to defend against the threat of terrorists’ use of liquid explosives onboard commercial aircraft. While the basis for these modifications was to mitigate risk, as shown in table 2, TSA senior leadership considered several other factors when deciding whether to implement the modifications.

As TSA senior leadership obtained more information about the particular threat posed by the liquid explosives through tests conducted by DHS’s Science and Technology Directorate and FBI, TSA relaxed the restrictions to allow passengers to carry liquids, gels, and aerosols onboard aircraft in 3-fluid-ounce bottles—and as of November 2006, 3.4-fluid-ounce bottles— that would easily fit in a quart-sized, clear plastic, zip-top bag. TSA senior leadership identified both benefits and drawbacks to this SOP modification, but determined that the balance of security, efficiency, and customer service that would result from these SOP changes was appropriate. As shown in table 2, TSA officials recognize that there are security drawbacks—or vulnerabilities—associated with allowing passengers to carry even small amounts of liquids and gels onboard aircraft. For example, two or more terrorists could combine small amounts of liquid explosives after they pass through the checkpoint to generate an amount large enough to possibly cause catastrophic damage to an aircraft. However, TSA officials stated that doing so would be logistically challenging given the physical harm that the specific explosives could cause to the person handling them, and that suspicion among travelers, law enforcement officials, and airport employees would likely be raised if an individual was seen combining the liquid contents of small containers stored in two or more quart-sized plastic bags. TSA officials stated that at the time of the modifications to the liquid, gels, and aerosols screening procedures, there was consensus among explosives detection experts, both domestically and abroad, regarding TSA’s assumptions about how the explosives could be used and the damage they could cause to an aircraft. TSA officials also stated that after reviewing the intelligence information related to the alleged August 2006 London terror plot— particularly with regard to the capability and intent of the terrorists—TSA determined that allowing small amounts of liquids, gels, and aerosols onboard aircraft posed an acceptable level of risk to the commercial aviation system. Moreover, TSA officials acknowledged that there are vulnerabilities with allowing passengers to carry liquids that are exempted from the 3.4-fluid-ounce limit—such as baby formula and medication— onboard aircraft.

TSA officials stated that the enhancements TSA is making to the various other layers of aviation security will help address the security vulnerabilities identified above. For example, TSA has increased explosives detection canine patrols, deployed Federal Air Marshals on additional international flights, increased random screening of passengers at boarding gates, and increased random screening of airport and TSA employees who pass through the checkpoint. TSA also plans to expand implementation of its Screening Passengers by Observation Technique (SPOT) to additional airports. SPOT involves specially trained TSOs observing the behavior of passengers and resolving any suspicious behavior through casual conversation with passengers and referring suspicious passengers to selectee screening. TSA intends for SPOT to provide a flexible, adaptable, risk-based layer of security that can be deployed to detect potentially high-risk passengers based on certain behavioral cues.

TSA’s Analysis of the Impact of Certain Proposed Screening Changes on Security and TSO Resources Could Be Strengthened

While professional judgment regarding risk factors, experience in the operating environment, and customer feedback have guided many of the decisions TSA leadership made about which screening procedures to implement, TSA also sought to use empirical data as a basis for evaluating the impact some screening changes could have on security and TSO resources. The TSA Assistant Secretary stated in December 2005 that TSA sought to make decisions about screening changes based on data and metrics—a practice he said TSA would continue. The use of data and metrics to inform TSA’s decision making regarding implementing proposed screening procedures is consistent with our prior work that has shown the importance of data collection and analyses to support agency decision making. Between October 2005 and January 2006, TSA’s Explosives Detection Improvement Task Force sought to collect data as part of an effort to test the impact of seven proposed procedures at selected airports, as noted earlier. These seven proposed procedures were selected because officials believed they would have a significant impact on how TSOs perform daily screening functions, TSO training, and customer acceptability. According to TSA’s chief of security operations, the purpose of testing these procedures in the airport environment was to ensure that TSA was “on the right path” in implementing them. These particular procedures were considered by senior TSA officials as especially important for enhancing the detection of explosives and for deterring terrorists from attempting to carry out an attack. According to TSA, some of the proposed procedures could also free up TSOs so that they could spend more time on procedures for detecting explosives and less time on procedures associated with low security risks, such as identifying small scissors in carry-on bags. The seven proposed procedures tested by the task force reflect both new procedures and modifications to existing procedures, as shown in table 3.

Our analysis of TSA’s data collection and data analysis for the seven procedures that were operationally tested identified several problems that affected TSA’s ability to determine whether these procedures, as designed and implemented by TSA, would have the intended effect—to enhance the detection of explosives during the passenger screening process or to free up resources so that explosives detection procedures could be implemented. Although the deterrence of persons intending to do harm is also an intended effect of some proposed SOP modifications, TSA officials said that it is difficult to assess the extent to which implementation of proposed procedures would deter terrorists. The Office of Management and Budget has also acknowledged the difficulty in measuring deterrence, particularly for procedures intended to prevent acts of terrorism. While we agree that measuring deterrence is difficult, opportunities exist for TSA to strengthen its analyses to help provide information on whether the proposed procedures would enhance detection or free up TSO resources, when intended.

Screening Passengers by Observation Technique. TSA officials stated that SPOT is intended to both deter terrorists and identify suspicious persons who intend to cause harm while on an aircraft. While we recognize that it is difficult to assess the extent to which terrorists are deterred by the presence of designated TSOs conducting behavioral observations at the checkpoint, we believe that there is an opportunity to assess whether SPOT contributes to enhancing TSA’s ability to detect suspicious persons that may intend to cause harm on an aircraft. One factor that may serve as an indicator that a person intends to do harm on an aircraft is whether that individual is carrying a prohibited item. TSA collected and assessed data at 14 airports for various time periods on the number of prohibited items found on passengers who were targeted under SPOT and referred to secondary screening or law enforcement officials. However, these data collection efforts, alone, did not enable TSA to determine whether the detection of prohibited items would be enhanced if SPOT were implemented because TSA had no means of comparing whether persons targeted by SPOT were more likely to carry prohibited items than persons not targeted by SPOT. To obtain this information, the task force would have had to collect data on the number of passengers not targeted by SPOT that had prohibited items on them. This information could be used to determine whether a greater percentage of passengers targeted under SPOT are found to have prohibited items than those passengers who are not targeted by SPOT, which could serve as one indicator of the extent to which SPOT would contribute to the detection of passengers intending to cause harm on an aircraft.

Although it has not yet done so, it may be possible for TSA to evaluate the impact of SPOT on identifying passengers carrying prohibited items. There is precedent in other federal agencies for evaluating the security benefits of similar procedures. For instance, U.S. Customs and Border Protection (CBP) within DHS developed the Compliance Examination (COMPEX) system to evaluate the effectiveness of its procedures for selecting international airline passengers for secondary screening. Specifically, COMPEX compares the percentage of targeted passengers on which prohibited items are found to the percentage of randomly selected passengers on which prohibited items are found. The premise is that targeting is considered to be effective if a greater percentage of targeted passengers are found to possess prohibited items than the percentage of randomly selected passengers, and the difference between the two percentages is statistically significant. CBP officials told us in May 2006 that they continue to use COMPEX to assess the effectiveness of their targeting of international airline passengers. When asked about using a method such as COMPEX to assess SPOT, TSA officials stated that CBP and TSA are seeking to identify different types of threats through their targeting programs. CBP, through its targeting efforts, is attempting to identify passengers with contraband and unauthorized aliens, whereas TSA, through SPOT, is attempting to identify potential high-risk passengers. Additionally, in commenting on a draft of this report, DHS stated that, according to TSA, the possession of a prohibited item is not a good measure of SPOT effectiveness because an individual may not intend to use a prohibited item to cause harm or hijack an aircraft. While it may be possible for a terrorist to cause harm or hijack an aircraft without using a prohibited item, as in the case of the September 11 terrorist attacks, other terrorist incidents and threat information identify that terrorists who carried out or planned to carry out an attack on a commercial aircraft intended to do so by using prohibited items, including explosives and weapons. Therefore, we continue to believe that comparing the percentage of individuals targeted and not targeted under SPOT on which prohibited items are found could be one of several potential indicators of the effectiveness of SPOT. Such a measure may be most useful with regard to the prohibited items that could be used to bring down or hijack an aircraft. TSA officials stated that the agency agrees in principle that measuring SPOT effectiveness, if possible, may provide valuable insights.

Unpredictable Screening Process, Bulk-Item Pat-Down Search, and IED Component Search. We found that the task force also could have strengthened its efforts to evaluate the security impact of other proposed procedures—specifically, USP, the bulk-item pat-down search, and the IED component search. For all three of these procedures, the task force did not collect any data during the operational testing that would help determine whether they would enhance detection capability. TSA officials told us that they did not collect these data because they had a limited amount of time to test the procedures because they had to make SOP modifications quickly as part of the agency’s efforts to focus on higher threats, such as explosives, and the TSA Assistant Secretary’s goal of implementing the SOP modifications before the 2005 Thanksgiving holiday travel season. Nevertheless, TSA officials acknowledged the importance of evaluating whether proposed screening procedures, including USP and the bulk-item pat-down, would enhance detection capability. TSA officials stated that covert testing has been used to assess TSOs’ ability to detect prohibited items, but covert testing was not implemented during operational testing of proposed procedures. Office of Inspection officials questioned whether covert testing could be used to test, exclusively, the security benefit of proposed procedures, because TSO proficiency and the capability of screening technology also factor into whether threat objects are detected during covert tests. Four of the five aviation security experts we interviewed acknowledged this limitation but stated that covert testing is the best way to assess the effectiveness of passenger checkpoint screening. In commenting on a draft of this report, DHS stated that, according to TSA, USP is intended to disrupt terrorists’ planning of an attack by introducing unpredictability into the passenger checkpoint screening process, and tools such as covert testing could not be used to measure the effectiveness of USP to this end. While we agree that covert testing may not be a useful tool to assess the impact USP has on disrupting terrorists’ plans and deterring terrorists from attempting to carry out an attack, we continue to believe that covert testing could have been used to assess whether USP would have helped to enhance detection capability during the passenger screening process, which TSA officials stated was another intended result of USP.

Although TSA did not collect data on the security impact of the USP and bulk-item pat-down procedures, the task force did collect data on the impact these procedures had on screening efficiency—the time required to perform procedures—and on the reaction of TSOs, FSDs, and passengers to the proposed procedures. These data indicated that the USP procedure took less time, on average, for TSOs to conduct than the procedure it replaced (the random continuous selectee screening process); the revised pat-down procedure took TSOs about 25 seconds to conduct; and that passengers generally did not complain about the way in which both procedures were conducted.

With respect to operational testing of the IED component search procedure, TSA was unable to collect any data during the testing period because no IEDs were detected by TSOs at the airports where the testing took place. As with the USP and bulk-item pat-down procedures, TSA could have conducted covert tests during the operational testing period to gather simulated data for the IED search procedure, in the absence of actual data.

Selectee Screening Changes and Threat Area Search. Recognizing that some of the proposed procedures intended to enhance detection would require additional TSO resources, TSA implemented several measures aimed collectively at freeing up TSOs’ time so that they could focus on conducting more procedures associated with higher threats— identifying explosives and suspicious persons. For example, TSA modified the selectee screening procedure and the procedure for searching carry-on items—the threat area search—in order to reduce screening time. During an informal pilot of these proposed procedures at 3 airports in November 2005, TSA determined that the proposed selectee screening procedure would reduce search time of each selectee passenger, on average, by about 1.17 minutes at these airports. TSA also determined through this study that the proposed threat area search, on average, took 1.83 minutes to conduct at the participating airports, as compared to the existing target object search that took, on average, 1.89 minutes, and the existing whole bag search that took, on average, 2.37 minutes.

Prohibited Items List Changes. Another measure TSA implemented to free up TSO resources to focus on higher threats involved changes to the list of items prohibited onboard aircraft. According to TSA, TSOs were spending a disproportionate amount of TSA’s limited screening resources searching for small scissors and small tools, even though, based on threat information and TSA officials’ professional judgment, such items no longer posed a significant security risk given the multiple layers of aviation security. TSA officials surmised that by not having to spend time and resources physically searching passengers’ bags for low-threat items, such as small scissors and tools, TSOs could focus their efforts on implementing more effective and robust screening procedures that can be targeted at screening for explosives.

To test its assumption that a disproportionate amount of TSO resources was being spent searching for small scissors and tools, TSA collected information from several sources. First, TSA reviewed data maintained in TSA’s Performance Management Information System (PMIS), which showed that during the third and fourth quarters of fiscal year 2005 (a 6-month period), TSOs confiscated a total of about 1.8 million sharp objects other than knives or box cutters. These sharp objects constituted 19 percent of all prohibited items confiscated at the checkpoint. Second, based on information provided by FSDs, TSOs, and other screening experts, TSA determined that scissors constituted a large majority of the total number of sharp objects found at passenger screening checkpoints. Third, TSA headquarters officials searched through confiscated items bins at 4 airports and found that most of the scissors that were confiscated had blades less than 4 inches in length. Based on these collective efforts, TSA concluded that a significant number of items found at the checkpoint were low-threat, easily identified items, such as small scissors and tools, and that a disproportionate amount of time was spent searching for these items—time that could have been spent searching for high-threat items, such as explosives. TSA also concluded that because TSOs can generally easily identify scissors, if small scissors were no longer on the prohibited items list, TSOs could avoid conducting time-consuming physical bag searches to locate and remove these items.

While we commend TSA’s efforts to supplement professional judgment with data and metrics in its decision to modify passenger checkpoint screening procedures, TSA did not conduct the necessary analysis of the data collected to determine the extent to which the removal of small scissors and tools from the prohibited items list could free up TSO resources. Specifically, TSA did not analyze the data on sharp objects confiscated at the checkpoint along with other relevant factors, such as the amount of time taken to search for scissors and the number of TSOs at the checkpoint conducting these searches, to determine the extent to which TSO resources could actually be freed up. Based on our analysis of TSA’s data for the 6-month period, where we considered these other relevant factors, we determined that TSOs spent, on average, less than 1 percent of their time—about 1 minute per day over the 6-month period— searching for the approximately 1.8 million sharp objects, other than knives and box cutters, that were found at passenger screening checkpoints between April 2005 and September 2005. If the average amount of time TSOs spent searching for sharp objects per day over a 6-month period was less than 1 minute per TSO, and sharp objects constituted just 19 percent of all prohibited items confiscated at checkpoints over this period, then it may not be accurate to assume that no longer requiring TSOs to search for small scissors and tools would significantly contribute to TSA’s efforts to free up TSO resources that could be used to implement other security measures.

To further support its assertion that significant TSO resources would be freed up as a result of removing small scissors and tools from the list of prohibited items, TSA officials cited the results of an informal study conducted in October 2005—which was intended to provide a general idea of the types of prohibited items TSOs were finding as a result of their searches and how long various types of searches were taking TSOs to conduct. Specifically, according to the study conducted at 9 airports over a 14-day period, TSA determined that 24 percent of items found during carry-on bag searches were scissors. However, based on data regarding the number of bags searched, removing scissors may not significantly contribute to TSA’s efforts to free up TSO resources.

TSA conducted additional informal studies 30, 60, and 90 days after the prohibited items list change went into effect to determine whether the change had resulted in reductions in the percentage of carry-on bags that were searched and overall screening time. However, we identified limitations in TSA’s methodology for conducting these studies. In February 2007, a TSA official stated that some FSDs interviewed several TSOs after the prohibited items list change went into effect, and these TSOs reported that the change did save screening time. However, TSA could not identify how many TSOs were interviewed, at which airports the TSOs were located, and how the TSOs were selected for the interview; nor did TSA document the results of these interviews. TSA also did not use random selection or representative sampling when determining which TSOs should be interviewed. Therefore, the interview results cannot be generalized.

TSA officials acknowledged that they could have made some improvements in the various analyses they conducted on the prohibited items list change. However, they stated that they had to make SOP modifications quickly as part of the agency’s efforts to focus on higher threats, such as explosives, and the TSA Assistant Secretary’s goal of implementing the SOP modifications before the 2005 Thanksgiving holiday travel season. Additionally, officials stated that they continue to view their decision to remove small scissors and tools from the prohibited items list as sound, particularly because they believe small scissors and tools do not pose a significant threat to aviation security. TSA officials also stated that they believe the prohibited items list change would free up resources based on various sources of information, including the professional judgment of TSA airport staff, and their analysis of PMIS data on prohibited items confiscated at checkpoints. The TSA Assistant Secretary told us that even if TSA determined that the proposed SOP modifications would not free up existing TSO resources to conduct explosives detection procedures, he would have implemented the modifications anyway considering the added security benefit of the explosives detection procedures. Additionally, a TSA headquarters official responsible for airport security operations stated that to help strengthen the agency’s analysis of future proposed SOP changes, the agency plans to provide the Explosives Detection Improvement Task Force with the necessary resources to help improve its data collection and analysis.

An additional measure intended to free up TSO resources involved changes to CAPPS rules. TSA’s assumption is that these changes could allow TSOs who were normally assigned to selectee screening duties to be reassigned to new procedures, such as USP, which may require new screening positions. (Both USP and SPOT require TSO positions: USP requires one screening position for every two screening lanes, while SPOT typically uses more than one screening position per ticket checker at the checkpoint.)

According to FSDs we interviewed, the changes made to the prohibited items list and the CAPPS rules had not freed up existing TSO resources, as intended. Specifically, as of August 2006, 13 of 19 FSDs we interviewed at airports that tested USP or SPOT said that TSO resources were not freed up as a result of these changes. In addition, 9 of the 19 FSDs said that in order to operationally test USP or SPOT, TSOs had to work overtime, switch from other functions (such as checked baggage screening), or a screening lane had to be closed. TSA’s Explosives Detection Improvement Task Force reported that nearly all of the FSDs at airports participating in operational testing of USP believed that the procedure had security value, though the task force also reported that 1 FSD dropped out of the operational testing program for USP due to insufficient staffing resources and another could only implement the procedure during off-peak travel periods. Additionally, most of the FSDs we interviewed stated that the changes to the prohibited items list and CAPPS rules did not free up TSOs, as intended, to better enable TSOs to take required explosives detection training. Specifically, as of August 2006, of the 19 FSDs we interviewed at airports that implemented USP and SPOT, 13 said that they did not experience more time to conduct explosives training as a result of changes to the prohibited items list and CAPPS rules. Three of the 13 FSDs said that they used overtime to enable TSOs to take the explosives training. As previously stated, the TSA Assistant Secretary stated that even if existing TSO resources are not freed up to conduct explosives detection procedures, longer lines and wait times at airport checkpoints are an acceptable consequence, considering their added security benefit. With regard to explosives training, he stated that it is acceptable for FSDs to use overtime or other methods to ensure that all TSOs participated in the required explosives detection training. He further noted that even if one screening change does not free up TSO resources, all of the changes intended to accomplish this—when taken together—should ultimately help to redirect TSO resources to where they are most needed.

TSA’s efforts to add data and metrics to its tool kit for evaluating the impact of screener changes are a good way to supplement the use of professional judgment and input from other experts and sources in making decisions about modifying screening procedures. However, TSA’s methods for data collection and analysis could be improved. We recognize the challenges TSA faces in evaluating the effectiveness of proposed procedures, particularly when faced with time pressures to implement procedures. However, by attempting to evaluate the potential impact of screening changes on security and resource availability, TSA could help support its decision making on how best to allocate limited TSO resources and ensure that the ability to detect explosives and other high-threat objects during the passenger screening process is enhanced.

Documentation of the Reasoning behind Proposed SOP Modifications Was Incomplete

While we were able to assess TSA’s reasoning behind certain proposed SOP modifications considered during our review period, our analysis was limited because TSA did not maintain complete documentation of proposed SOP modifications. Documentation of the reasoning behind decisions to implement or reject proposed modifications was maintained in various formats, including spreadsheets developed by TSA officials, internal electronic mail discussions among TSA officials, internal memorandums, briefing slides, and reports generated based on the results of operational testing. TSA did improve its documentation of the proposed SOP modifications that were considered during the latter part of our 9-month review period. Specifically, the documentation for the SOP modifications proposed under the Explosives Detection Improvement Task Force provided more details regarding the basis of the proposed modifications and the reasoning behind decisions to implement or reject the proposed modifications.

Of the 92 proposed SOP modifications considered during our 9-month review period that TSA documented, TSA provided the basis for 72. More specifically, TSA documented the basis—that is, the information, experience, or event that encouraged TSA officials to propose an SOP modification—for 35 of the 48 that were implemented and for 37 of the 44 that were not implemented. However, TSA only documented the reasoning behind TSA senior leadership’s decisions to implement or not implement proposed SOP modifications for 43 of 92 proposed modifications. According to TSA officials, documentation that explains the basis for recommending proposed modifications can also be used to explain TSA’s reasoning behind its decisions to implement proposed modifications. However, the basis on which an SOP modification was proposed cannot always be used to explain TSA senior leadership’s decisions not to implement a proposed modification. In these cases, additional documentation would be needed to understand TSA’s decision making. However, TSA only documented the reasoning behind its decisions for about half (26 of 44) of the proposed modifications that were not implemented. TSA officials told us that they did not intend to document all SOP modifications that were proposed during our review period. Officials stated that, in some cases, the reasoning behind TSA’s decision to implement or not implement a proposed SOP modification is obvious and documentation is not needed. TSA officials acknowledged that it is beneficial to maintain documentation on the reasoning behind decisions to implement or reject proposed SOP modifications deemed significant, particularly given the organizational restructuring and staff turnover within TSA. However, TSA officials could not identify which of the 92 proposed SOP modifications they consider to be significant because they do not categorize proposed modifications in this way.

Our standards for governmental internal controls and associated guidance suggest that agencies should document key decisions in a way that is complete and accurate, and that allows decisions to be traced from initiation, through processing, to after completion. These standards further state that documentation of key decisions should be readily available for review. Without documenting this type of information, TSA cannot always justify significant modifications to passenger checkpoint screening procedures to internal or external stakeholders, including Congress and the traveling public. In addition, considering the ongoing personnel changes, without sufficient documentation, future decision makers in TSA may not know on what basis the agency historically made decisions to develop new or revise existing screening procedures.

Following our 9-month review period, TSA continued to make efforts to improve documentation of agency decision making, as evidenced by decisions regarding the August 2006 and September 2006 SOP modifications related to the screening of liquids and gels. For example, TSA senior leadership evaluated the actions taken by the agency between August 7 and August 13, 2006, in response to the alleged liquid explosives terrorist plot, in order to identify lessons learned and improve the agency’s reaction to future security incidents. As a result of this evaluation, as shown in table 4, TSA made several observations and recommendations for improving documentation of agency decision making when considering modifications to screening procedures.

Documentation of TSA’s decisions regarding the September 26, 2006, modifications to the liquid screening procedures showed that TSA had begun implementing the recommendations in table 4. TSA’s documentation identified the various proposed liquid screening procedures that were considered by TSA, the benefits and drawbacks of each proposal, and the rationale behind TSA’s final decision regarding which proposal to implement. The documentation also tracked the timing of TSA’s deliberations of each of the proposed liquid screening procedures. However, the documentation of TSA’s decisions was not always presented in a standard format, nor was the origin and use of supporting documentation always identified. TSA officials acknowledged that documentation of the September 2006 SOP modifications could have been improved and stated that efforts to improve documentation, through implementation of the recommendations in table 4, will continue to be a high priority.

TSA Has Several Methods in Place to Monitor TSO Compliance with Passenger Checkpoint Screening SOPs

A New Performance Accountability System Helps TSA Monitor TSO Compliance with SOPs

TSA implemented a performance accountability system in part to strengthen its monitoring of TSO compliance with passenger checkpoint screening SOPs. Specifically, in April 2006, TSA implemented the Performance Accountability and Standards System (PASS) to assess the performance of all TSA employees, including TSOs. According to TSA officials, PASS was developed in response to our 2003 report that recommended that TSA establish a performance management system that makes meaningful distinctions in employee performance, and in response to input from TSA airport staff on how to improve passenger and checked baggage screening measures. With regard to TSOs, PASS is not intended solely to measure TSO compliance with SOPs. Rather, PASS will be used by TSA to assess agency personnel at all levels on various competencies, including training and development, readiness for duty, management skills, and technical proficiency.

There are three elements of the TSO technical proficiency component of PASS that are intended to measure TSO compliance with passenger checkpoint screening procedures: (1) quarterly observations conducted by FSD management staff of TSOs’ ability to perform particular screening functions in the operational environment, such as pat-down searches and use of the hand-held metal detector, to ensure they are complying with checkpoint screening SOPs; (2) quarterly quizzes given to TSOs to assess their knowledge of the SOPs; and (3) an annual, multipart knowledge and skills assessment. While the first two elements are newly developed, the third element—the knowledge and skills assessment—is part of the annual TSO recertification program that is required by the Aviation and Transportation Security Act (ATSA) and has been in place since October 2003. Collectively, these three elements of PASS are intended to provide a systematic method for monitoring whether TSOs are screening passengers and their carry-on items according to SOPs. TSA’s implementation of PASS is consistent with our internal control standards, which state that agencies should ensure that policies and procedures are applied properly.

The first component of PASS (quarterly observations) is conducted by screening supervisors or screening managers, using a standard checklist developed by TSA headquarters, with input from TSA airport staff. There is one checklist used for each screening function, and TSOs are evaluated on one screening function per quarter. For example, the hand-held metal detector skills observation checklist includes 37 tasks to be observed, such as whether the TSO conducted a pat-down search to resolve any suspect areas. The second component of PASS (quarterly quizzes) consists of multiple-choice questions on the standard operating procedures. For example, one of the questions on the PASS quiz is “What is the correct place to start an HHMD outline on an individual: (a) top of the head, (b) top of the feet, or (c) top of the shoulder?”

The third component of PASS is the annual knowledge and skills assessment, a component of the annual recertification program that evaluates the technical proficiency of TSOs. This assessment is composed of three modules: (1) knowledge of standard operating procedures, (2) recognition of threat objects on an X-ray image, and (3) demonstration of screening functions. According to TSA officials, while recertification testing is not a direct measure of operational compliance with passenger checkpoint screening SOPs, recertification testing, particularly module 1 and module 3, is an indicator of whether TSOs are capable of complying with SOPs. TSA officials stated that if a TSO does not have knowledge of SOPs and if the TSO cannot demonstrate basic screening functions as outlined in the SOPs, then the TSO will likely not be able to comply with SOPs when performing in the operating environment. Table 5 provides a summary of each of these modules.

FSDs we interviewed reported that they have faced resource challenges in implementing PASS. Specifically, as of July 2006, 9 of 24 FSDs we interviewed said they experienced difficulties in implementing PASS due to lack of available staff to conduct the compliance-related evaluations. TSA officials stated that they have automated many of the data-entry functions of PASS to relieve the field of the burden of manually entering this information into the PASS online system. For example, all scores related to the quarterly quiz and skill observation components are automatically uploaded, and PASS is linked to TSA’s online learning center database to eliminate the need to manually enter TSOs’ learning history. In addition, the TSA Assistant Secretary said that FSDs were given the option of delaying implementation of PASS if they were experiencing resource challenges.

TSA Uses Local and National Covert Testing, in Part, to Assess TSO Compliance with SOPs

TSA also conducts local and national covert tests, which are used to evaluate, in part, the extent to which noncompliance with the SOPs affects TSOs’ ability to detect simulated threat items hidden in accessible property or concealed on a person. TSA first issued guidance on its local covert testing program—known as Screener Training Exercises and Assessments (STEA)—in February 2004. STEA testing is conducted by FSD staff at airports, who determine the frequency at which STEA tests are conducted as well as which type of STEA tests are conducted. According to the STEA results reported by TSA between March 2004 and February 2006, TSOs’ noncompliance with the SOP accounted for some of the STEA test failures. TSOs’ lack of proficiency in skills or procedures, which may affect TSOs’ ability to comply with procedures, was also cited as the reason for some of the STEA test failures. TSOs who fail STEA tests are required to take remedial training to help them address the reasons for their failure.

FSDs we interviewed reported that they have faced resource challenges in conducting STEA tests. Specifically, even though all 24 FSDs we interviewed as of July 2006 said that they have conducted STEA tests, 10 of these FSDs said that the lack of available staff made it difficult to conduct these tests. When asked how they planned to address FSDs’ concerns regarding a lack of available staff to complete STEA tests, TSA headquarters officials told us that they are considering resource alternatives for implementing the STEA program, but could not provide us with the specific details of these plans. Until the resource limitations that have restricted TSA’s use of its compliance monitoring tools have been fully addressed, TSA may not have assurance that TSOs are screening passengers according to the SOP.

As previously discussed, TSA’s Office of Inspection initiated its national covert testing program in September 2002. National covert tests are conducted by TSA headquarters-based inspectors who carry simulated threat objects hidden in accessible property or concealed on their person through airport checkpoints, and in cases where TSOs fail to detect threat objects, the inspectors identify the reasons for failure. During September 2005, TSA implemented a revised covert testing program to focus more on catastrophic threats—threats that can bring down or destroy an aircraft. According to Office of Inspection officials, TSOs may fail to detect threat objects during covert testing for various reasons, including limitations in screening technology, lack of training, limitations in the procedures TSOs must follow to conduct passenger and bag searches, and TSOs’ noncompliance with screening checkpoint SOPs. Office of Inspection officials also said that one test could be failed due to multiple factors, and that it is difficult to determine the extent to which any one factor contributed to the failure. TSOs who fail national covert tests, like those who fail STEA tests, are also required to take remedial training to help them address the reasons for failure.

Conclusions

The alleged August 2006 terrorist plot to detonate liquid explosives onboard multiple U.S.-bound aircraft highlighted the need for TSA to continuously reassess and revise, when deemed appropriate, existing passenger checkpoint screening procedures to address threats against the commercial aviation system. In doing so, TSA faces the challenge of securing the aviation system while facilitating the free movement of people. Passenger screening procedures are only one element that affects the effectiveness and efficiency of the passenger checkpoint screening system. Securing the passenger checkpoint screening system also involves the TSOs who are responsible for conducting the screening of airline passengers and their carry-on items, and the technology used to screen passengers and their carry-on items.

We believe that TSA has implemented a reasonable approach to modifying passenger checkpoint screening procedures through its consideration of risk factors (threat and vulnerability information), day-to-day experience of TSA airport staff, and complaints and concerns raised by passengers and by making efforts to balance security, efficiency, and customer service. We are also encouraged by TSA’s efforts to conduct operational testing and use data and metrics to support its decisions to modify screening procedures. We acknowledge the difficulties in assessing the impact of proposed screening procedures, particularly with regard to the extent to which proposed procedures would deter terrorists from attempting to carry out an attack onboard a commercial aircraft. However, there are existing methods, such as covert testing and CBP’s COMPEX—a method that evaluates the effectiveness of CBP’s procedures for selecting international airline passengers for secondary screening—that could be used by TSA to assess whether proposed screening procedures enhance detection capability. It is also important for TSA to fully assess available data to determine the extent to which TSO resources would be freed up to perform higher-priority procedures, when this is the intended effect. Without collecting the necessary data or conducting the necessary analysis that would enable the agency to assess whether proposed SOP modifications would have the intended effect, it may be difficult for TSA to determine how best to improve TSOs’ ability to detect explosives and other high-threat items and to allocate limited TSO resources. With such data and analysis, TSA would be in a better position to justify its SOP modifications and to have a better understanding of how the changes affect TSO resources. Additionally, because TSA did not always document the basis on which SOP modifications were proposed or the reasoning behind decisions to implement or not implement proposed modifications, TSA may not be able to justify SOP modifications to Congress and the traveling public. While we are encouraged that TSA’s documentation of its decisions regarding the SOP modifications made in response to the alleged August 2006 liquid explosives terrorist plot was improved compared to earlier documentation, it is important for TSA to continue to work to strengthen its documentation efforts. Such improvements would enable TSA officials responsible for making SOP decisions in the future to understand how significant SOP decisions were made historically— a particular concern considering the restructuring and staff turnover experienced by TSA.

As shown by TSA’s covert testing results, the effectiveness of passenger checkpoint screening relies, in part, on TSOs’ compliance with screening procedures. We are, therefore, encouraged by TSA’s efforts to strengthen its monitoring of TSO compliance with passenger screening procedures. We believe that TSA has implemented a reasonable process for monitoring TSO compliance and that this effort should assist TSA in providing reasonable assurance that TSOs are screening passengers and their carry-on items according to screening procedures. Given the resource challenges FSDs identified in implementing the various methods for monitoring TSO compliance, it will be important for TSA to take steps, such as automating PASS data entry functions, to address such challenges.

Recommendations for Executive Action

To help strengthen TSA’s evaluation of proposed modifications to passenger checkpoint screening SOPs and TSA’s ability to justify its decisions to implement or not implement proposed SOP modifications, in the March 2007 report that contained sensitive security information, we recommended that the Secretary of Homeland Security direct the Assistant Secretary of Homeland Security for TSA to take the following two actions: when operationally testing proposed SOP modifications, develop sound evaluation methods, when possible, that can be used to assist TSA in determining whether proposed procedures would achieve their intended result, such as enhancing TSA’s ability to detect prohibited items and suspicious persons and freeing up existing TSO resources that could be used to implement proposed procedures, and for future proposed SOP modifications that TSA senior leadership determines are significant, generate and maintain documentation to include, at minimum, the source, intended purpose, and reasoning behind decisions to implement or not implement proposed modifications.

Agency Comments and Our Evaluation

On March 6, 2007, we received written comments on the draft report, which are reproduced in full in appendix III. DHS generally concurred with our recommendations and outlined actions TSA plans to take to implement the recommendations.

DHS stated that it appreciates GAO’s conclusion that TSA has implemented a reasonable approach to modifying passenger checkpoint screening procedures through its assessment of risk factors, the expertise of TSA employees, and input from the traveling public and other stakeholders, as well as TSA’s efforts to balance security, operational efficiency, and customer service while evaluating proposed changes.

With regard to our recommendation to develop sound evaluation methods, when possible, to help determine whether proposed SOP modifications would achieve their intended result, DHS stated that TSA plans to make better use of generally accepted research design principles and techniques when operationally testing proposed SOP modifications. For example, TSA will consider using random selection, representative sampling, and control groups in order to isolate the impact of proposed SOP modifications from the impact of other variables. DHS also stated that TSA’s Office of Security Operations is working with subject matter experts to ensure that operational tests are well designed and executed, and produce results that are scientifically valid and reliable. As discussed in this report, employing sound evaluation methods for operationally testing proposed SOP modifications will enable TSA to have better assurance that new passenger checkpoint screening procedures will achieve their intended purpose, which may include improved allocation of limited TSO resources and enhancing detection of explosives and other high-threat objects during the passenger screening process. However, DHS stated, and we agree, that the need to make immediate SOP modifications in response to imminent terrorist threats may preclude operational testing of some proposed modifications.

Concerning our recommendation regarding improved documentation of proposed SOP modifications, DHS stated that TSA intends to document the source, intent, and reasoning behind decisions to implement or reject proposed SOP modifications that TSA senior leadership deems significant. Documenting this type of information will enable TSA to justify significant modifications to passenger checkpoint screening procedures to internal and external stakeholders, including Congress and the traveling public. In addition, considering the ongoing personnel changes TSA has experienced, such documentation should enable future decision makers in TSA to understand on what basis the agency historically made decisions to develop new or revise existing screening procedures.

In addition to commenting on our recommendations, DHS provided comments on some of our findings, which we considered and incorporated in the report where appropriate. One of DHS’s comments pertained to TSA’s evaluation of the prohibited items list change. Specifically, while TSA agrees that the agency could have conducted a more methodologically sound evaluation of the impact of the prohibited items list change, TSA disagrees with our assessment that the prohibited items list change may not have significantly contributed to TSA’s efforts to free up TSO resources to focus on detection of high-threat items, such as explosives. As we identified in this report, based on interviews with FSDs, airport visits to determine the types of items confiscated at checkpoints, and a study to determine the amount of time taken to conduct bag searches and the number of sharp objects collected as a result of these searches, TSA concluded that the prohibited items list change would free up TSO resources. DHS also stated that interviews with TSOs following the prohibited items list change confirmed that the change had freed up TSO resources. However, based on our analysis of the data TSA collected both prior to and following the prohibited items list change, we continue to believe that TSA did not conduct the necessary analysis to determine the extent to which the removal of small scissors and tools from the prohibited items list would free up TSA resources.

As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 21 days from the date of this report. At that time, we will send copies of the report to the Secretary of the Department of Homeland Security, the TSA Assistant Secretary, and interested congressional committees as appropriate. We will also make copies available to others on request.

If you or your staff have any questions about this report, please contact me at (202) 512-3404 or berrickc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope and Methodology

To assess the Transportation Security Administration’s (TSA) process for modifying passenger checkpoint screening procedures and how TSA monitors compliance with these procedures, we addressed the following questions: (1) How and on what basis did TSA modify passenger screening procedures and what factors guided the decisions to do so? (2) How does TSA determine whether TSOs are complying with the standard procedures for screening passengers and their carry-on items?

To address how TSA modified passenger screening procedures and what factors guided the decisions to do so, we obtained and analyzed documentation of proposed standard operating procedures (SOP) changes considered between April 2005 and September 2005, as well as threat assessments and operational studies that supported SOP modifications. The documentation included a list of proposed changes considered, as well as the source, the intended purpose, and in some cases the basis for recommending the SOP modification—that is, the information, experience, or event that encouraged TSA officials to propose the modifications—and the reasoning behind decisions to implement or reject proposed SOP modifications. We also obtained documentation of the proposed SOP changes considered by TSA’s Explosives Detection Improvement Task Force, which was the deliberating body for proposed changes that were considered between October 2005 and December 2005. We also reviewed and analyzed similar documentation for proposed SOP modifications considered between August 2006 and November 2006 in response to the alleged terrorist plot to detonate liquid explosives onboard multiple aircraft en route from the United Kingdom to the United States. We included modifications to passenger checkpoint screening procedures related to this particular event because they provided the most recent information available of TSA’s approach to modifying screening procedures in response to an immediate perceived threat to civil aviation. The documentation included notes from internal meetings, slides for internal and external briefings on proposed SOP modifications, data on customer complaints and screening efficiency, and the results of liquid explosives testing conducted by the Department of Homeland Security (DHS) Science and Technology Directorate and the Federal Bureau of Investigation (FBI). We also obtained each revision of the passenger checkpoint screening SOP that was generated between April 2005 and December 2005 and August 2006 and November 2006, as well as accompanying documentation that highlighted all of the changes made in each revision. In addition, we met with TSA headquarters officials who were involved in the process for determining whether proposed passenger checkpoint screening procedures should be implemented. We also met with officials in the DHS Science and Technology Directorate as well as the FBI to discuss the methodology and results of their liquid explosives tests, which were used to support TSA’s decisions to modify the SOP in September 2006. We also met with TSA Office of Inspection and DHS Office of Inspector General staff to discuss their covert testing at passenger checkpoints and the recommended changes to the passenger checkpoint screening SOP that were generated based on testing results. We also obtained and analyzed data and information collected by TSA on the proposed procedures that were evaluated in the operational environment. In addition, we met or conducted phone interviews with Federal Security Directors (FSD) and their management staff, including Assistant FSDs and Screening Managers, and Transportation Security Officers (TSO) with passenger screening responsibilities, at 25 commercial airports to gain their perspectives on TSA’s approach to revising the passenger checkpoint screening SOP. We also met with officials from four aviation associations—the American Association of Airport Executives, Airports Council International, the Air Transport Association, and the Regional Airline Association—to gain their perspectives on this objective. Finally, we met with five aviation security experts to obtain their views on methods for assessing the impact of proposed passenger checkpoint screening procedures. We selected these experts based on their depth of experience in the field of aviation security, employment history, and their recognition in the aviation security community. However, the views of these experts may not necessarily represent the general view of other experts in the field of aviation security. We compared TSA’s approach to revising its passenger checkpoint screening SOP with the Comptroller General’s standards for internal control in the federal government and risk management guidance.

To address how TSA determines whether TSOs are complying with the standard procedures for screening passengers and their carry-on items, we obtained documentation of compliance-related initiatives, including guidance, checklists, and SOP quizzes used to assess TSO compliance under the Performance Accountability and Standards System (PASS), and guidance provided to FSDs for developing local compliance audit programs. We also obtained the fiscal year 2005 recertification and Screener Training Exercises and Assessments (STEA) test results, which were used, in part, to assess TSO compliance with and knowledge of the passenger checkpoint screening SOP. In addition, we reviewed the results of covert testing conducted by TSA’s Office of Inspection, which were also used, in part, to assess TSO compliance with the passenger checkpoint screening SOP. We assessed the reliability of the compliance-related data we received from TSA, and found the data to be sufficiently reliable for our purposes. In addition, we interviewed TSA headquarters officials who were responsible for overseeing efforts to monitor TSO compliance with standard operating procedures. This included officials in the Office of Security Operations, Office of Human Capital, and the Office of Operational Process and Technology. Our audit work also included visits to or phone conferences with 25 airports, where we interviewed FSDs, members of their management teams, and Transportation Security Officers with passenger screening responsibilities. However, the perspectives of these FSDs and their staff cannot be generalized across all airports. In July 2006, we submitted two sets of follow-up questions to FSD staff, related to their experiences with implementing PASS and STEA tests. We also obtained documentation of local compliance audit programs from the FSD staff at several of these airports. We compared TSA’s approach for monitoring TSO compliance with the Comptroller General’s standards for internal control in the federal government.

As previously mentioned, we conducted site visits and/or phone interviews at 25 airports (8 category X airports, 7 category I airports, 4 category II airports, 4 category III airports, and 2 category IV airports) to discuss issues related to TSA’s approach to revising the passenger checkpoint screening SOP, and the agency’s approach to monitoring TSO compliance with the SOP. We visited 7 of these airports during the design phase of our study. These airports were selected based on variations in size and geographic location, and whether they were operationally testing any proposed passenger checkpoint screening procedures or passenger screening technology. We also selected 2 airports that participated in the Screening Partnership Program.

After visiting the 7 airports during the design phase of our review, we selected an additional 15 airports to visit based on variations in size, geographic distribution, and performance on compliance-related assessments. Specifically, we obtained and analyzed fiscal year 2005 Screener Training Exercise and Assessments results and fiscal year 2005 recertification testing results to identify airports across a range of STEA and recertification scores. Additionally, we visited 3 additional airports that operationally tested the proposed Unpredictable Screening Process (USP) and the Screening Passengers by Observation Technique (SPOT) procedure.

In July 2006, we received from 19 FSDs answers to follow-up questions on their experiences with implementing pilot testing of SPOT or USP. This included 14 FSDs that were not part of our initial rounds of interviews. Nine of these 14 FSDs were from airports that participated in SPOT pilots. The remaining 5 of 14 FSDs that were not part of our initial rounds of interviews were from airports that were participants in USP pilots.

We conducted our work from March 2005 through March 2007 in accordance with generally accepted government auditing standards.

Appendix II: Sources of SOP Changes

Of the 92 proposed screening changes considered by TSA between April 2005 and December 2005, 63 were submitted by TSA field staff, including Federal Security Directors and Transportation Security Officers. Thirty proposed screening changes were submitted by TSA headquarters officials. Last, TSA senior leadership, such as the TSA Assistant Secretary, recommended 5 of the 92 proposed screening changes considered during this time period. One SOP modification was also proposed through a congressional inquiry. TSA’s solicitation of input from both field and headquarters officials regarding changes to the passenger checkpoint screening SOP was consistent with internal control standards, which suggest that there be mechanisms in place for employees to recommend improvements in operations.

The FSDs with whom we met most frequently identified periodic conference calls with the Assistant Secretary, the SOP Question and Answer mailbox, or electronic mail to Security Operations officials as the mechanisms by which they recommended changes to the SOP. The TSOs with whom we met identified their chain of command and the SOP Question and Answer mailbox as the primary mechanisms by which they submitted suggestions for new or revised procedures. According to TSA officials, the SOP mailbox entails FSDs and their staff, including TSOs, submitting suggestions, questions, or comments to TSA’s Security Operations division via electronic mail, either directly or through their supervisors. Submissions are then compiled and reviewed by a single Security Operations official, who generates responses to the questions that have clear answers. However, for submissions for which the appropriate response is not obvious or for submissions that include a suggestion to revise the SOP, this official forwards the submissions to other Security Operations officials for further deliberation. SOP mailbox responses are provided to all TSA airport officials. If TSA headquarters revised a screening procedure based on a mailbox submission, the revision is noted in the mailbox response.

Thirty of the screening changes considered by TSA between April 2005 and December 2005 were proposed by TSA headquarters officials, including Security Operations officials, who are responsible for overseeing implementation of checkpoint screening. According to Security Operations officials, they recommended changes to checkpoint screening procedures based on communications with TSA field officials and airport optimization reviews. Security Operations officials conduct optimization reviews to identify best practices and deficiencies in the checkpoint screening and checked baggage screening processes. As part of these reviews, Security Operations officials may also assess screening efficiency and whether TSOs are implementing screening procedures correctly.

Other TSA headquarters divisions also suggested changes to passenger checkpoint screening procedures. For example, the Office of Law Enforcement recommended that there be an alternative screening procedure for law enforcement officials who are escorting prisoners or protectees. Previously, all armed law enforcement officers were required to sign a logbook at the screening checkpoint, prior to entering the sterile area of the airport. The officials in the Office of Passengers with Disabilities also recommended changes to checkpoint screening procedures. For example, in the interest of disabled passengers, they suggested that TSOs be required to refasten all wheelchair straps and buckles undone during the screening process.

Last, TSA senior leadership suggested 5 of the 92 procedural changes considered by TSA between April 2005 and December 2005. TSA senior leadership also proposed a procedure that would allow TSOs to conduct the pat-down procedure on passengers of the opposite gender at airports with a disproportionate ratio of male and female TSOs.

Appendix III: Comments from the Department of Homeland Security

Appendix IV: GAO Contact and Staff Acknowledgments

Acknowledgments

In addition to the person named above, Maria Strudwick, Assistant Director; David Alexander; Christopher W. Backley; Amy Bernstein; Kristy Brown; Yvette Gutierrez-Thomas; Katherine N. Haeberle; Robert D. Herring; Richard Hung; Christopher Jones, Stanley Kostyla; and Laina Poon made key contributions to this report.

GAO Related Products

Aviation Security: TSA's Staffing Allocation Model Is Useful for Allocating Staff among Airports, but Its Assumptions Should Be Systematically Reassessed. GAO-07-299. Washington, D.C.: February 28, 2007 Aviation Security: Progress Made in Systematic Planning to Guide Key Investment Decisions, but More Work Remains. GAO-07-448T. Washington, D.C.: February 13, 2007.

Homeland Security: Progress Has Been Made to Address the Vulnerabilities Exposed by 9/11, but Continued Federal Action Is Needed to Further Mitigate Security Risks. GAO-07-375. Washington, D.C.: January 24, 2007.

Aviation Security: TSA Oversight of Checked Baggage Screening Procedures Could Be Strengthened GAO-06-869. Washington, D.C.: July 28, 2006.

Aviation Security: TSA Has Strengthened Efforts to Plan for the Optimal Deployment of Checked Baggage Screening Systems, but Funding Uncertainties Remain GAO-06-875T. Washington, D.C.: June 29, 2006.

Aviation Security: Management Challenges Remain for the Transportation Security Administration’s Secure Flight Program. GAO-06-864T. Washington, D.C.: June 14, 2006.

Aviation Security: Further Study of Safety and Effectiveness and Better Management Controls Needed if Air Carriers Resume Interest in Deploying Less-than-Lethal Weapons. GAO-06-475. Washington, D.C.: May 26, 2006.

Aviation Security: Enhancements Made in Passenger and Checked Baggage Screening, but Challenges Remain. GAO-06-371T. Washington, D.C.: April 4, 2006.

Aviation Security: Transportation Security Administration Has Made Progress in Managing a Federal Security Workforce and Ensuring Security at U.S. Airports, but Challenges Remain. GAO-06-597T. Washington, D.C.: April 4, 2006.

Aviation Security: Progress Made to Set Up Program Using Private- Sector Airport Screeners, but More Work Remains. GAO-06-166. Washington, D.C.: March 31, 2006.

Aviation Security: Significant Management Challenges May Adversely Affect Implementation of the Transportation Security Administration’s Secure Flight Program. GAO-06-374T. Washington, D.C.: February 9, 2006.

Aviation Security: Federal Air Marshal Service Could Benefit from Improved Planning and Controls. GAO-06-203. Washington, D.C.: November 28, 2005.

Aviation Security: Federal Action Needed to Strengthen Domestic Air Cargo Security. GAO-06-76. Washington, D.C.: October 17, 2005.

Transportation Security Administration: More Clarity on the Authority of Federal Security Directors Is Needed. GAO-05-935. Washington, D.C.: September 23, 2005.

Aviation Security: Flight and Cabin Crew Member Security Training Strengthened, but Better Planning and Internal Controls Needed. GAO-05-781. Washington, D.C.: September 6, 2005.

Aviation Security: Transportation Security Administration Did Not Fully Disclose Uses of Personal Information during Secure Flight Program Testing in Initial Privacy Notes, but Has Recently Taken Steps to More Fully Inform the Public. GAO-05-864R. Washington, D.C.: July 22, 2005.

Aviation Security: Better Planning Needed to Optimize Deployment of Checked Baggage Screening Systems. GAO-05-896T. Washington, D.C.: July 13, 2005.

Aviation Security: Screener Training and Performance Measurement Strengthened, but More Work Remains. GAO-05-457. Washington, D.C.: May 2, 2005.

Aviation Security: Secure Flight Development and Testing Under Way, but Risks Should Be Managed as System Is Further Developed. GAO-05-356. Washington, D.C.: March 28, 2005.

Aviation Security: Systematic Planning Needed to Optimize the Deployment of Checked Baggage Screening Systems. GAO-05-365. Washington, D.C.: March 15, 2005.

Aviation Security: Measures for Testing the Effect of Using Commercial Data for the Secure Flight Program. GAO-05-324. Washington, D.C.: February 23, 2005.

Transportation Security: Systematic Planning Needed to Optimize Resources. GAO-05-357T. Washington, D.C.: February 15, 2005.

Aviation Security: Preliminary Observations on TSA’s Progress to Allow Airports to Use Private Passenger and Baggage Screening Services. GAO-05-126. Washington, D.C.: November 19, 2004.

General Aviation Security: Increased Federal Oversight Is Needed, but Continued Partnership with the Private Sector Is Critical to Long-Term Success. GAO-05-144. Washington, D.C.: November 10, 2004.

Aviation Security: Further Steps Needed to Strengthen the Security of Commercial Airport Perimeters and Access Controls. GAO-04-728. Washington, D.C.: June 4, 2004.

Transportation Security Administration: High-Level Attention Needed to Strengthen Acquisition Function. GAO-04-544. Washington, D.C.: May 28, 2004.

Aviation Security: Challenges in Using Biometric Technologies. GAO-04-785T. Washington, D.C.: May 19, 2004.

Nonproliferation: Further Improvements Needed in U.S. Efforts to Counter Threats from Man-Portable Air Defense Systems. GAO-04-519. Washington, D.C.: May 13, 2004.

Aviation Security: Private Screening Contractors Have Little Flexibility to Implement Innovative Approaches. GAO-04-505T. Washington, D.C.: April 22, 2004.

Aviation Security: Improvement Still Needed in Federal Aviation Security Efforts. GAO-04-592T. Washington, D.C.: March 30, 2004.

Aviation Security: Challenges Delay Implementation of Computer- Assisted Passenger Prescreening System. GAO-04-504T. Washington, D.C.: March 17, 2004.

Aviation Security: Factors Could Limit the Effectiveness of the Transportation Security Administration’s Efforts to Secure Aerial Advertising Operations. GAO-04-499R. Washington, D.C.: March 5, 2004.

Aviation Security: Computer-Assisted Passenger Prescreening System Faces Significant Implementation Challenges. GAO-04-385. Washington, D.C.: February 13, 2004.

Aviation Security: Challenges Exist in Stabilizing and Enhancing Passenger and Baggage Screening Operations. GAO-04-440T. Washington, D.C.: February 12, 2004.

The Department of Homeland Security Needs to Fully Adopt a Knowledge-based Approach to Its Counter-MANPADS Development Program. GAO-04-341R. Washington, D.C.: January 30, 2004.

Aviation Security: Efforts to Measure Effectiveness and Strengthen Security Programs. GAO-04-285T. Washington, D.C.: November 20, 2003.

Aviation Security: Federal Air Marshal Service Is Addressing Challenges of Its Expanded Mission and Workforce, but Additional Actions Needed. GAO-04-242. Washington, D.C.: November 19, 2003.

Aviation Security: Efforts to Measure Effectiveness and Address Challenges. GAO-04-232T. Washington, D.C.: November 5, 2003.

Airport Passenger Screening: Preliminary Observations on Progress Made and Challenges Remaining. GAO-03-1173. Washington, D.C.: September 24, 2003.

Aviation Security: Progress since September 11, 2001, and the Challenges Ahead. GAO-03-1150T. Washington, D.C.: September 9, 2003.

Transportation Security: Federal Action Needed to Enhance Security Efforts. GAO-03-1154T. Washington, D.C.: September 9, 2003.

Transportation Security: Federal Action Needed to Help Address Security Challenges. GAO-03-843. Washington, D.C.: June 30, 2003.

Federal Aviation Administration: Reauthorization Provides Opportunities to Address Key Agency Challenges. GAO-03-653T. Washington, D.C.: April 10, 2003.

Transportation Security: Post-September 11th Initiatives and Long- Term Challenges. GAO-03-616T. Washington, D.C.: April 1, 2003.

Airport Finance: Past Funding Levels May Not Be Sufficient to Cover Airports’ Planned Capital Development. GAO-03-497T. Washington, D.C.: February 25, 2003.

Transportation Security Administration: Actions and Plans to Build a Results-Oriented Culture. GAO-03-190. Washington, D.C.: January 17, 2003.

Aviation Safety: Undeclared Air Shipments of Dangerous Goods and DOT’s Enforcement Approach. GAO-03-22. Washington, D.C.: January 10, 2003.

Aviation Security: Vulnerabilities and Potential Improvements for the Air Cargo System. GAO-03-344. Washington, D.C.: December 20, 2002.

Aviation Security: Registered Traveler Program Policy and Implementation Issues. GAO-03-253. Washington, D.C.: November 22, 2002.

Airport Finance: Using Airport Grant Funds for Security Projects Has Affected Some Development Projects. GAO-03-27. Washington, D.C.: October 15, 2002.

Commercial Aviation: Financial Condition and Industry Responses Affect Competition. GAO-03-171T. Washington, D.C.: October 2, 2002.

Aviation Security: Transportation Security Administration Faces Immediate and Long-Term Challenges. GAO-02-971T. Washington, D.C.: July 25, 2002.

Aviation Security: Information Concerning the Arming of Commercial Pilots. GAO-02-822R. Washington, D.C.: June 28, 2002.

Aviation Security: Vulnerabilities in, and Alternatives for, Preboard Screening Security Operations. GAO-01-1171T. Washington, D.C.: September 25, 2001.

Aviation Security: Weaknesses in Airport Security and Options for Assigning Screening Responsibilities. GAO-01-1165T. Washington, D.C.: September 21, 2001.

Homeland Security: A Framework for Addressing the Nation’s Efforts. GAO-01-1158T. Washington, D.C.: September 21, 2001.

Aviation Security: Terrorist Acts Demonstrate Urgent Need to Improve Security at the Nation’s Airports. GAO-01-1162T. Washington, D.C.: September 20, 2001.

Aviation Security: Terrorist Acts Illustrate Severe Weaknesses in Aviation Security. GAO-01-1166T. Washington, D.C.: September 20, 2001.

This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.