Aviation Security: TSA Should Take Additional Action to Obtain Stakeholder Input when Modifying the Prohibited Items List
Published: 2015-02-04T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: As part of its responsibilities for securing civil aviation, TSA ensures that all passengers and their accessible property are screened and prohibits individuals from carrying onto aircraft items that it determines to be a threat.TSA maintains a public list of such items, known as the Prohibited Items List, and updates it as necessary. In March 2013, TSA announced it would modify the PIL to allow small knives and certain sporting equipment onto aircraft, stating the change would result in more efficient security screening. However, several aviation industry groups opposed the decision, leading TSA to reverse its decision to implement the change. GAO was asked to review TSA's procedures for modifying the PIL.
This report examines, among other issues, (1) on what basis TSA modifies the PIL and the extent to which TSA assessed risk when considering recent modifications to the PIL, and (2) the extent to which TSA involved stakeholders when considering these modifications. GAO reviewed TSA's standard operating procedures, risk assessment, documentation of its decisions and stakeholder outreach, and interviewed TSA officials at six airports. This is a public version of a report with Sensitive Security Information that GAO issued in December 2014. Information TSA deemed sensitive has been redacted.
section_title: What GAO Found
paragraphs: Transportation Security Administration (TSA) officials stated that TSA considers four factors when determining whether to make modifications to the Prohibited Items List (PIL), but the agency did not fully assess risk when considering its recent proposed PIL modifications, as GAO has previously recommended. TSA generally considers the following four factors when determining whether to modify the PIL: (1) the security risks posed by each carry-on item, (2) opportunities to improve screening operations and passenger experience, (3) harmonization with international security standards and practices, and (4) stakeholder perspectives. While TSA considered these four factors when making its March 5, 2013, decision to allow small knives and certain sporting equipment on aircraft, TSA officials also reasoned that the proposed change could help screening personnel focus less on lower-threat items, such as small knives, and more on higher-threat items, such as explosives, thereby potentially increasing security for passengers. However, TSA did not conduct sufficient analysis to show that the increased risk of allowing small knives on aircraft—as determined in its risk assessment—would be offset by a resulting reduction in risk from improved screening for explosives. GAO has previously recommended that TSA strengthen its evaluation methods for operationally testing proposed modifications to checkpoint screening procedures, including changes to the PIL. However, TSA has not consistently implemented this recommendation. Conducting additional risk analysis would have allowed TSA to actually measure whether airport screeners would be better able to identify explosives if they no longer had to screen for small knives. GAO continues to believe that TSA should develop and apply sound evaluation methods when considering modifications to the PIL, as GAO recommended in April 2007.
TSA did not effectively solicit feedback on its 2013 PIL decision from relevant external stakeholders, some of whom subsequently expressed strong opposition to the decision to remove small knives from the PIL. For example, prior to announcing its decision, TSA did not coordinate with or obtain input from the Aviation Security Advisory Committee, which is TSA's primary external advisory group for aviation security matters and whose membership includes various airline industry associations. Some relevant stakeholders, such as flight attendant groups—from whom TSA did not adequately solicit feedback—subsequently expressed strong opposition to the proposal, which contributed to TSA reversing its decision to implement the change after having already trained screening personnel for its implementation. Having a defined process and associated procedures in place to communicate with relevant stakeholders earlier in the decision-making process could allow TSA to ensure appropriate consideration of their perspectives in the decision-making process. Use of a defined process and associated procedures could also allow TSA to better avoid rescission of any future changes after investing resources in training screening personnel and informing the general public of the change—as happened in the case of TSA's 2013 PIL decision.
section_title: What GAO Recommends
paragraphs: GAO recommends that TSA establish a formal process for soliciting input from relevant external stakeholders on proposed modifications to the PIL before making changes to it. DHS agreed with the recommendation.

section_title: Letter
paragraphs: The Transportation Security Administration (TSA), within the Department  of Homeland Security, is responsible for, among other things, securing  the nation’s civil aviation system while also facilitating the movement of  passengers and commerce. One measure TSA uses to secure the  aviation environment is to prohibit individuals from carrying into the sterile  area of an airport or onto a commercial passenger aircraft any item that   To  TSA has determined to be a threat to the aircraft and its passengers.help implement this measure, TSA maintains a list—the Prohibited Items  List (PIL)—that identifies for the traveling public those items that may not  be carried through an airport security checkpoint or on board an aircraft,  and trains transportation security officers (TSO) to screen passengers  entering the checkpoints for such items. TSA ensures the PIL is  effectively carried out and implemented at the checkpoint by TSOs who  are trained on the use of various technologies and procedures to screen  passengers and their accessible property for these prohibited items at  airport checkpoints.
Since assuming responsibility for aviation security from the Federal  Aviation Administration following the terrorist attacks of September 11,  2001, TSA has enforced and made several modifications to the PIL. For  example, in December 2005, TSA began permitting passengers to carry  small scissors and tools onto aircraft based on its review of threat  information that indicated these items do not pose a high risk to the  aviation system, among other considerations. At the time, some industry  representatives raised concerns that allowing sharp objects onto planes  would put flight crews at risk. Stakeholder concerns over TSA’s decision  to allow these objects onto aircraft resulted in Congress asking the TSA  Administrator to testify on the agency’s rationale for the change, and GAO  to assess the impact of the change on public safety and screening  operations. We reported on this assessment in April 2007. Most recently,  on March 5, 2013, as part of efforts to implement a risk-based security  approach to passenger screening, TSA announced that it would further  modify the PIL to allow passengers to carry small knives and certain  types of sporting equipment onto aircraft. Several industry associations,  including flight attendant associations, actively opposed the change, and  on March 21, 2014, 133 Members of Congress signed a letter asking the  TSA Administrator not to go through with implementing the change. On  April 22, 2013, 3 days before the PIL change was to take effect, TSA  postponed its implementation so it could obtain further input from various  industry representatives and other stakeholder groups. On June 5, 2013,  TSA announced that, after extensive engagement with various aviation  stakeholder groups, it would not be implementing the proposed change  and would continue to maintain and enforce the ban on these items as  included on the current PIL.
In a 2007 report on TSA’s changes to passenger screening procedures,  we found that TSA had a reasonable approach to modifying checkpoint  screening procedures, which include screening for prohibited items  (referred to in this report as the PIL), but that TSA could strengthen its  evaluation of proposed modifications to better justify its decisions. example, TSA stated that the purpose of its December 2005 decision to  allow passengers to carry small scissors and tools onto aircraft was to  shift TSO focus from items considered by TSA to pose a low threat (small  scissors and tools) to items considered to pose a high threat, such as  improvised explosive devices (IED), as well as better allocate TSA resources to implement other security measures that target IEDs.
See GAO, Aviation Security: Risk, Experience, and Customer Concerns Drive Changes  to Airline Passenger Screening Procedures, but Evaluation and Documentation of  Proposed Changes Could Be Improved, GAO-07-634 (Washington, D.C.: April 2007). In  this report, references to the PIL include the checkpoint screening procedures TSOs  implement to identify and screen for prohibited items as well as the interpretive rule  published on occasion in the Federal Register and the list of prohibited items maintained  on TSA’s website—http://www.tsa.gov/traveler-information/prohibited-items—through  which the traveling public is informed of those items that may or may not be carried  through a screening checkpoint and on board an aircraft.
However, we reported that TSA did not conduct the necessary analysis of  the data it had collected to determine the extent to which the removal of  small scissors and tools from the PIL could free up TSO resources to  screen for higher-risk items, nor did TSA analyze other relevant factors  such as the amount of time taken to search for small scissors and tools  and the number of TSOs conducting these searches. As a result, we  recommended that TSA, when operationally testing proposed  modifications to its checkpoint screening procedures, such as the PIL,  develop sound evaluation methods to assist it in determining whether  proposed procedures would achieve their intended result, such as  enhancing the agency’s ability to better detect certain prohibited items,  such as IEDs, and free up existing TSO resources. TSA concurred with  the recommendation but has not fully implemented it.
Given the importance of TSA’s PIL for aviation security and the safe and  efficient movement of passengers and security operations, and in light of  TSA’s 2013 decision not to remove small knives and certain types of  sporting equipment from the PIL, you asked us to assess the manner in  which TSA considers making changes to the PIL. Accordingly, this report  addresses the following questions: (1) On what basis does TSA modify  the PIL, and to what extent did TSA assess risk and involve stakeholders  when assessing whether to remove small knives and certain sporting  equipment from the PIL? (2) How are transportation security officers  trained to implement modifications to the PIL at the checkpoint?
This report is a public version of the prior sensitive report that we  provided to you. TSA deemed some of the information in the prior report  as Sensitive Security Information, which must be protected from public  disclosure. Therefore, this report omits sensitive information about TSA’s  risk assessment of various prohibited items. Although the information  provided in this report is more limited and less specific, it addresses the  same questions as the sensitive report. Also, the overall methodology  used for both reports is the same.
To determine the basis upon which TSA decides to modify the PIL, we  reviewed and assessed TSA’s criteria and process for making changes to  the list, including the factors TSA uses to guide its decision making. We  obtained documentation and interviewed agency officials about proposed  modifications, including how TSA considered risk information, stakeholder  perspectives, and impact on the efficiency of airport operations and  customer service during its process for considering changes to the PIL.  Specifically, to assess the extent to which TSA considered risk in its most  recent proposal to modify the PIL to allow small knives and certain types  of sporting equipment aboard aircraft, we reviewed documentation— including decision memos—detailing the risk assessment conducted by  TSA that led to its March 5, 2013, decision to remove small knives from  the PIL. For the purposes of this report we will refer to this proposed PIL  change as the “small knives” PIL change. We evaluated TSA’s risk  assessment of the PIL detailed in this documentation against Department  of Homeland Security (DHS) guidance on managing and assessing risk to  determine the extent to which TSA’s risk assessment aligned with  established risk management principles. We also interviewed TSA  officials from the Office of Security Operations and the Office of the Chief  Risk Officer to discuss details of how they conducted the risk assessment  that resulted in the small knives decision. In addition, to aid our  understanding of how TSA modifies checkpoint screening procedures— such as the PIL—or to understand how it trains TSOs when making  modifications, we obtained and reviewed TSA’s standard operating  procedures related to passenger checkpoint screening to determine   Last, to obtain the  TSA’s process for screening for prohibited items.perspectives of TSA airport personnel on PIL modifications, including  their involvement in the modification process, and the impact of PIL  modifications on field operations, we visited six airports of varying sizes to  include two category X airports and one each from categories I through  IV.airports, this variance in size allowed us to gain perspectives from airports  with different levels of passenger throughput. During our visits, we  interviewed federal security directors, TSA security managers,  supervisors, TSOs, and airport officials regarding their views on the  impacts of TSA’s proposed modifications of the PIL on airport security.
Specifically, we contacted (1) Airlines for America (representing airline executives), (2)  Airports Council International-North America (representing regional and state government  owners/operators of commercial airports), (3) the Association of Flight Attendants, (4) the  Airline Pilots Association (ALPA), (5) FlyersRights.org (a fliers’ rights consumer group), (6)  the Federal Law Enforcement Officers Association (FLEOA), and (7) the Federal Flight  Deck Officers Association (FFDOA). stakeholders. With respect to the decision to allow small knives on  aircraft, we also evaluated TSA actions on stakeholder coordination  against its internal decision memos citing the importance of, and need for,  full coordination with certain aviation stakeholder groups.
To determine how TSOs are trained to implement modifications to the PIL  once they are approved, we obtained documentation and interviewed  agency officials from TSA’s Office of Training and Workforce Engagement  about the steps TSA takes to ensure that TSOs are informed of  modifications to the PIL and trained to implement them at passenger  checkpoints. We reviewed training materials, including briefing  documents and training aids, used to prepare TSOs to implement the  small knives decision. During our airport visits, we also obtained the  perspectives of TSA personnel at the airports on how they are prepared  to implement changes to the PIL. Because we did not select TSA  employees and airports at random for our interviews, the views obtained  during these visits are not generalizable to TSA operations at all airports  nationwide. However, the information we obtained from these interviews  provided insight into the views various TSA employee groups have of the  training programs TSA provides for TSOs leading up to a change to the  PIL.
We conducted this performance audit from January 2014 to February  2015, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: TSA Authority
paragraphs: As part of its responsibilities for civil aviation security, TSA enforces law  and regulations requiring that passengers be screened to ensure that  potential weapons, explosives, and incendiaries are not carried into an  airport sterile area or on board a passenger aircraft. To provide the  general public with guidance on the types of property TSA policy prohibits  from being brought into airport sterile areas and on board aircraft, TSA  publishes, and on occasion has updated, an interpretive rule in the  Federal Register—known as the PIL—that, among other things, lists  items prohibited from being carried on a passenger’s person or in the  passenger’s accessible property into airport sterile areas and into the  cabins of passenger aircraft. TSA also maintains a current list of  prohibited items on its public website. The list is not intended to be  exhaustive, and TSOs may exercise discretion, informed by the  categories and examples included in the PIL and their standard operating  procedures, to prohibit an individual from carrying an item through the  checkpoint if in the screener’s determination the item could pose a threat  to transportation (i.e., whether it is or could be a weapon, explosive, or  incendiary) regardless of whether it is or is not on the PIL. TSA has  divided prohibited items into nine categories. Table 1 provides a  description of the items included in the nine categories. Individuals are  prohibited from carrying these items into an airport sterile area or on  board an aircraft either in their carry-on bags or on their person.
section_title: Passenger Checkpoint Screening
paragraphs: At passenger screening checkpoints, TSOs inspect individuals and  property as part of the passenger screening process to deter and prevent  the carriage of any unauthorized explosive, incendiary, weapon, or other  items included on the PIL into the sterile area or on board an aircraft.shown in figure 1, TSOs use the following methods, among others, to  screen passengers:
X-ray screening of property,
Advanced imaging technology scanners (often referred to by the  public as body scanners) or walk-through metal detector screening of  individuals, pat-down screening of individuals, physical search of property, trace detection for explosives, and behavioral observation.
TSA has developed checkpoint screening standard operating procedures  that establish the process and standards by which TSOs are to screen  passengers and their carry-on items at the screening checkpoint.  According to TSA standard operating procedures, passengers may be  screened through the use of a walk-through metal detector, advanced  imaging technology scanner, or a pat-down. Passengers are also  generally required to divest their property, including the removal of shoes  and outer garments, and empty their pockets. During this screening  process, TSOs look for any prohibited or dangerous items on a  passenger or among the passenger’s property.
Ordinarily, passenger screening at the checkpoint begins when the  individual divests and places his or her accessible property on the X-ray  conveyor belt or hands such property to a TSO. A TSO then reviews  images of the property running through the X-ray machine and looks for  signs of prohibited items. The passengers themselves are typically  screened via a walk-through metal detector or an advanced imaging  technology scanner, and passengers have the option to request  screening by a pat-down if they do not wish to be screened via the  advanced imaging technology scanner. TSA uses additional screening  techniques on a random basis to provide an additional layer of security.  These additional screening techniques, referred to as an Unpredictable  Screening Process, are prompted automatically by the walk-through  metal detector, which picks out a certain percentage of passengers at  random to be selected for additional screening. For example, TSA uses  explosives trace detection (ETD) to swab the hands or property of  passengers on a random basis to screen for explosives.
section_title: Modifications to the Prohibited Items List
paragraphs: According to TSA officials, because of statutory and other considerations,  TSA has revised the PIL six times since its inception in February 2003  (see table 2). In general, TSA modifies the PIL as necessary when  circumstances prompt the agency to revise the items listed as prohibited  from being carried into an airport sterile area or on board an aircraft. For  example, in 2005, TSA modified the PIL in response to a statutory  requirement to prohibit passengers from carrying any type of lighter on  their person or in their accessible property on board aircraft. Later that  year, TSA also modified the PIL to allow passengers with ostomates to  carry small ostomy scissors with them onto aircraft because the agency  had heard from persons with ostomies that they avoid flying, in part,  because they are not allowed to carry the scissors they need onto the  aircraft. In 2006, TSA further modified its policy with respect to permitted  and prohibited items in response to a specific terrorist threat by initially  prohibiting the carriage of liquids, gels, and aerosols on board an aircraft,  and subsequently permitting passengers to carry limited amounts of  liquids, gels, and aerosols on board an aircraft in a manner prescribed by  the agency.
See TSA Security Directive 1544-14-02 (Feb. 6, 2014) and TSA Emergency  Amendment 1546-14-01 (Feb. 6, 2014) (imposing additional security requirements on U.S.  and foreign air carrier operations, respectively, to and from the Russian Federation). The  security directive and emergency amendment did permit the carriage of medication in  liquid, gel, or aerosol form. such measures are carried out, TSA generally accomplishes this by  issuing security directives and emergency amendments and, as  circumstances permit, in coordination and consultation with host  governments, the International Civil Aviation Organization (ICAO), and  other affected parties.
section_title: TSA Considers Several Factors when Modifying the PIL, but Did Not Fully Assess Risk or Stakeholder Perspectives when Evaluating Its Small Knives Proposal
subsections: section_title: TSA Considers Several Factors when Evaluating Potential PIL Modifications
paragraphs: TSA officials told us that when evaluating whether or not to change the  PIL, they generally consider the following four factors: (1) the security  risks posed by each item on the current PIL or potential item to be added,  (2) opportunities a potential change may have to improving checkpoint  screening and passenger experience, (3) harmonization with international  aviation security standards and recommended practices published by  ICAO, and (4) stakeholder perspectives on the change. For example, as  part of a broader set of potential changes related to adopting a risk-based  security approach to passenger screening, TSA formed a working group  in 2011 to conduct a risk-based review of the PIL; assessed the individual  risk posed by each PIL item; and then considered how removing a  particular item, or set of items would present opportunities, constraints,  and challenges for TSA security operations at the checkpoint. TSA  officials stated they then considered how any changes would affect TSA  personnel costs and passenger experience such as likely screening  throughput time if TSA personnel no longer had to screen for particular  items. TSA then evaluated how interested parties such as Congress,  airlines, and flight attendants would respond to permitting particular items  on board an aircraft. TSA also considered ICAO guidance on prohibited  items and took into account whether any changes it made to the PIL  would further align TSA’s guidelines for prohibiting items with ICAO  standards and recommended practices.
TSA officials told us that TSA does not have policies that require a  specific process to be followed or a specific set of criteria to be used  when evaluating potential modifications to the PIL, since the  circumstances for each potential PIL change are unique. Officials stated  the steps they take when considering a modification often vary depending  on the nature of the proposed revision. In its 2011 review of the PIL,  TSA’s working group addressed these factors as follows:  Impacts on security risk: The working group evaluated the risk to  transportation security presented by each prohibited item by assessing  the likelihood of an adversary successfully using the item to achieve  different terrorist objectives. TSA assigned risk ratings of high, medium,  low, or none to each item on the PIL for each terrorist objective. TSA  assessed the levels of risk posed by small knives for each terrorist  objective.
Impacts on screening operations: The working group also considered  how the removal of small knives would affect checkpoint screening  operations. For example, TSA estimated, using historical data prior to  2009, that approximately half of all nonfirearm, nonincendiary voluntarily  abandoned property (VAP) left behind at the checkpoint consisted of  small knives with blades shorter than 2.36 inches. TSA concluded that TSOs spent a disproportionate amount of their time searching for these  items. TSA reasoned that removing small knives from the PIL would have  a positive impact on screening operations since TSOs would no longer  have to detect and deal with small knives at the checkpoint, reducing  direct and indirect personnel costs, increasing passenger throughput, and  reducing distractions to TSOs. TSA also concluded that not requiring  TSOs to screen for small knives would in turn improve their ability to  screen for higher-threat items, such as IEDs, and thus reduce risk overall.  For example, the TSA risk assessment cited a research study focused on  how success rates for screening items vary based on what screeners look  for. TSA cited the study in support of its assertion that TSOs would be  more successful identifying IEDs if they did not have to screen for small  knives.
Harmonization with international standards and guidance: TSA also  considered the harmonization of the PIL with ICAO standards and  recommended practices. TSA concluded that making certain changes to  the PIL, such as removing small knives, could better harmonize its  policies with ICAO guidance. Specifically, ICAO guidance provides that  member states should consider prohibiting knives with blades of more  than 6 centimeters (approximately 2.36 inches) from being carried on  board aircraft. TSA concluded that there would be operational and policy  benefits from harmonizing the PIL with ICAO guidance because greater  harmony among the various countries promotes greater cooperation on  all security issues. Further, TSA asserted that inconsistencies between  the PIL and the ICAO guidance could create confusion for passengers  when items were allowed onto aircraft in one country, but prohibited in  another.
Stakeholder perspectives: The TSA working group also noted the need  to coordinate with stakeholders on some of the options for modifying the  PIL, as these options were likely to cause concern among some of these  groups, if implemented. For example, for the working group’s proposed  recommendation to remove small knives from the PIL, TSA officials noted  past concerns from stakeholders over the prospect of allowing small  knives or other items on board aircraft and stated that coordination and  collaboration with key stakeholders would be a critical success factor for  implementation. They also noted that stakeholder support would be  greatly enhanced by a unified approach to communicating to stakeholder  groups that TSA planned to shift its resource focus from finding small  knives to other efforts that would result in better security.
Although TSA recognized that allowing small knives on planes would  raise the potential risk of other terrorist aircraft scenarios, TSA concluded  the change would not raise the overall risk of catastrophic aircraft  destruction. However, rather than make an immediate decision about  changing the PIL, TSA elected to suspend working group activities and  delay making any decisions while it focused greater attention and TSA  resources on other emerging risk-based security initiatives, such as the  Known Crewmember and expedited passenger screening programs.  TSA resumed working group evaluations of the PIL in July 2012.
section_title: TSA Did Not Fully Assess Risk Prior to Its Decision to Remove Small Knives from the PIL
paragraphs: As previously discussed, TSA used its risk assessment to conclude that  overall risk to aviation security would be lowered by allowing small knives  onto aircraft because security screeners would be able to better focus on  identifying higher-risk items, such as IEDs. However, TSA did not conduct  sufficient analysis to show that removing small knives would ultimately  reduce risk and improve checkpoint screening.
TSA’s reasoning for its decision to remove small knives from the PIL was  to further align the PIL with ICAO guidance on prohibited items, decrease  time spent rescreening or searching bags for these items, and better  enable its TSOs to focus more attention on higher-threat items, such as  IEDs, thereby potentially increasing security. DHS guidance for managing  and assessing risk states that risk assessments should evaluate all the   In its risk assessment,  risk scenarios considered by the assessment.TSA assessed the risk posed by small knives for each terrorist objective;  however, it did not complete data collection or an evaluation to determine  whether TSOs would actually be better able to identify high-risk items,  such as IEDs, if they were not looking for small knives. Furthermore, the  research cited by TSA did not evaluate a situation where screeners had  to differentiate between knives with blades greater or less than 2.36  inches in length, as proposed by TSA. Without conducting a more valid  evaluation of the actual proposed change, TSA could not sufficiently  evaluate whether the added risk of allowing small knives onto aircraft  would be offset by a reduction in risk achieved through improved  screening for IEDs. Such an analysis would have allowed TSA to actually  measure whether airport screeners would be better able to identify  explosives if they no longer had to screen for small knives, and better  determine whether the added risk of allowing small knives onto aircraft  would be offset by potential efficiencies in screening for explosives.
Moreover, 25 of 35 TSOs (including supervisory TSOs) and 8 of the 10  Transportation Security Managers we interviewed during visits to six  airports did not agree that allowing small knives on planes would have  helped them better screen for IEDs, as TSA concluded in its risk  assessment. Four TSOs and 1 supervisory TSO we interviewed noted  that the exact size of a knife is difficult to ascertain on an X-ray.  Therefore, these 4 TSOs and the supervisor believed they would have to  open bags in many instances and physically measure the knife to make  sure it conformed to TSA’s definition of a permissible knife, which,  according to TSA’s definition, was a nonfixed blade less than 2.36 inches  and not exceeding a 0.5 inch in width with no locking mechanism, and no  molded grip or nonslip handle. TSA officials told us that the training  provided to TSOs specified that each TSO was expected to use his or her  judgment in determining, based on the X-ray image, whether a knife was  permissible or not.
We previously recommended in 2007 that TSA strengthen its evaluation  of proposed modifications to the PIL and other checkpoint screening  procedures to better justify its decisions. Specifically, in April 2007, we  found that TSA did not conduct the necessary analysis to support its 2005  decision to remove small scissors (4 inches or less) and certain tools (7  inches or less) from the PIL. As with TSA’s more recent rationale for  removing small knives from the PIL, TSA stated that the reason for its  decision to remove small scissors and tools was to shift TSO focus from  items considered by TSA to pose a low threat to items considered to pose  a high threat, such as IEDs, as well as to better allocate TSA resources to  implement other security measures that target IEDs. However, we found  that TSA did not conduct the necessary analysis to determine the extent  to which removing small scissors and tools from the PIL could improve  TSO performance in detecting higher-threat items, nor did TSA analyze  other relevant factors such as the amount of time taken to search for  small scissors and tools and the number of TSOs conducting these  searches. As a result, we recommended that TSA, when operationally  testing proposed modifications to its checkpoint screening procedures,  such as the PIL, develop sound evaluation methods to assist it in  determining whether proposed procedures would achieve their intended  result, such as enhancing the agency’s ability to better detect prohibited  items, and free up existing TSO resources.
TSA conducted one evaluation on proposed X-ray screening procedures and one test  on a proposed ETD procedure. Regarding the X-ray procedure change, TSA collected and  analyzed the necessary data to determine whether the X-ray screening procedures would  improve passenger throughput. However, in its evaluation of ETD devices, TSA was not  able to provide documentation that explained the intended purpose of the proposed ETD  procedure, the type of data TSA planned to collect, or how the data would be used. changes to standard operating procedures, as we recommended in April  2007. Without sound evaluation methods, TSA will be limited in its ability  to determine whether proposed modifications to standard operating  procedures—such as the PIL—will result in the intended risk reduction,  for example, by enhancing the agency’s ability to better detect IEDs and  other high-risk items.
section_title: TSA Did Not Adequately Obtain Stakeholder Perspectives when Evaluating Its Small Knives Proposal
paragraphs: TSA consulted both internal and external stakeholders during  development of its decision to remove small knives from the PIL, but it did  not adequately consult with several external aviation stakeholder groups.  Some of these groups later raised strong objections after TSA publicly  announced the change. GAO’s Standards for Internal Control in the  Federal Government states that an organization’s management should  ensure there are adequate means of communicating with, and obtaining  information from, external stakeholders that may have a significant impact  on the agency’s operations and its achievement of organizational goals.  These internal control standards further state that management is  responsible for developing detailed policies, procedures, and practices to  fit its agency’s operations and to ensure that they are built into, and an  integral part of, operations. Moreover, TSA’s risk assessment and other  planning documents leading up to its proposal to remove small knives  from the PIL called for full coordination and collaboration with key external  stakeholders who might have reservations about the change before  moving forward with any revisions to the PIL.
In coordinating with stakeholders, TSA primarily consulted with internal  groups who, according to TSA, were generally supportive overall of the  proposed revision. Specifically, TSA’s efforts to coordinate with internal  groups included the following:
The TSA “Idea Factory”: Provides for online comments from TSA  personnel with “likes” and “dislikes” similar to those on Facebook.  According to TSA, results received from April 2011 through December  2012 indicated that some TSA personnel, including screeners,  thought that removing small knives would be a good idea and would  improve their ability to screen for IEDs.
TSA National Advisory Council: An internal employee advisory  committee representing TSA employees at various levels, including  management, supervisors, and TSOs. An ad hoc subcommittee of this  council reviewed the PIL in July 2012 and recommended removing  small knives.
Administrator Pistole’s informal discussions with TSOs: According to  TSA, Administrator Pistole visited several airports, starting with a town  hall meeting at Charlotte Douglas International Airport in December  2012, to gather input on the small knives proposal. TSA officials  stated that, during these meetings, TSOs were supportive of the  knives proposal and thought it would improve their ability to screen for  explosive devices.
Federal Air Marshal Service (FAMS): In February 2013, FAMS  provided TSA management with comments that led to the small  knives decision being more restrictive than TSA executives had  originally considered. Specifically, the decision no longer allowed fixed  or locking blades, or tactical or combat knives, regardless of length.  FAMS officials stated they were generally opposed to allowing small  knives on aircraft, but their concerns were mitigated by TSA  management’s revision of the proposal.
TSA also reached out to some external stakeholder groups who,  according to TSA, were also supportive of the decision to eliminate small  knives, including the Airline Pilots Association, Families of September 11,  and DHS’s Homeland Security Advisory Council (HSAC). In addition,  the TSA Administrator discussed possible changes to the Prohibited  Items List in various appearances before Congress from 2010 to 2012  where he expressed the belief that screening personnel should  concentrate on items that can cause catastrophic destruction of an  aircraft. However, TSA did not discuss the proposal and solicit feedback  from other relevant external stakeholders prior to its announcement. For  example, TSA did not coordinate with or obtain input from the Aviation  Security Advisory Committee (ASAC), which is its primary external  advisory group for aviation security matters and whose membership  includes various airline industry associations.stakeholders—from whom TSA did not adequately solicit feedback— subsequently expressed strong opposition to the proposal, which  contributed to TSA reversing its decision to implement the proposal. For  example, TSA did not adequately consult with flight attendant groups  during development of the small knives proposal, including the  Association of Flight Attendants—an ASAC member—and the Coalition of  Flight Attendant Unions. Specifically, in a November 30, 2012, phone call  primarily regarding another matter, TSA informed the AFA president that  it was also planning to modify the PIL to remove small knives. AFA  officials disagreed with this decision. However, this conversation occurred  after TSA had developed the proposal for the decision over the preceding  months. Shortly after this meeting, the TSA Administrator approved the  decision to remove small knives from the PIL, which was followed by the  March 5, 2013, public announcement of the decision.
In response to feedback received after its March 5, 2013, public  announcement of the small knives decision, TSA conducted a classified  briefing with the ASAC. TSA officials met with the ASAC on April 22,  2013, more than a month after TSA’s March 5, 2013, public  announcement of its proposed change and just prior to its planned  implementation date of April 25, 2013, and briefed ASAC members on the  announced change. Immediately following this meeting, and on the basis  of input received by ASAC members and other stakeholders, the TSA  Administrator announced a delay in implementation of the change to  allow the agency additional time to more fully coordinate with various  external stakeholders groups and incorporate additional input on the  change. Following the ASAC briefing and announcement of the delay,  TSA held similar briefings with other stakeholder groups including the  Victims of Pan Am Flight 103 and the National Air Disaster  Alliance/Foundation. On June 5, 2013, the TSA Administrator announced  that on the basis of extensive engagement with the ASAC and other  stakeholder groups, including law enforcement officials and passenger  advocates, TSA would continue to enforce the current PIL and not go  forward with the decision to remove small knives from the list.
As described earlier, TSA management officials stated that they do not  have a formal policy or a specific process for evaluating PIL  modifications; this also means that they have no specific requirements for  coordinating with stakeholders during development of potential revisions  to the PIL. TSA officials stated that if some of the steps for stakeholder  coordination defined in other TSA processes for emergency amendments  and security directives had been in place for PIL changes—such as  obtaining key stakeholder input when developing a security policy  change—they may have helped to ensure better stakeholder coordination  during consideration of the knives change. For example, TSA officials  stated that, in hindsight, meeting with the ASAC and having more in- depth discussions with flight attendants during internal deliberations over  modifying the PIL would have improved their efforts to fully coordinate  and ensure they appropriately obtained and considered all key  stakeholder perspectives. TSA officials also stated that they would have  benefited from broader engagement earlier in the process with external  groups, such as the ASAC and flight attendants. In the case of the small  knives decision, the officials added that this broader and more timely  engagement could have provided additional insight into the breadth and  depth of potential concerns associated with removing certain items from  the PIL.
Clear processes outlining the appropriate types of stakeholders to  consult—including when in the process stakeholders should be  consulted—could help ensure that TSA’s process for determining PIL  changes is effective and efficient. For example, having clearly defined  processes for stakeholder coordination could ensure that TSA fully  obtains and considers stakeholder views—consistent with internal control  standards and TSA’s planning documents—that could help mitigate  potential inefficiencies resulting from reversing policy decisions. Going  forward, a formal process to ensure the solicitation of input from relevant  external stakeholders on proposed changes to the PIL, including when in  the PIL modification process TSA officials are to coordinate with such  stakeholders, would help provide reasonable assurance that TSA has a  more complete understanding of stakeholder perspectives earlier in the  decision-making process. This could help avoid rescission of those  changes after investing resources in training TSOs and informing the  general public of the change, as was the case with the proposed change  to remove small knives from the PIL.
section_title: TSA Utilizes a Variety of Approaches for Preparing TSOs to Implement Revisions to the PIL at Passenger Screening Checkpoints
subsections: section_title: TSA Customizes TSO Training to Fit the Unique Nature and Complexity of Each PIL Revision
paragraphs: According to TSA personnel from the Office of Training Workforce and  Engagement (OTWE), TSA evaluates on a case-by-case basis what  training tools it will use to ensure TSOs are adequately trained to  implement a change to the PIL. However, TSA typically provides TSOs  with one or more of the following methods to prepare and train them to  implement a PIL change:
Online training—This type of training is web-based and may be  completed by the TSOs either individually or as a group. This training  may include test questions to assess the TSOs’ mastery of the  material.
Instructor-led classroom training—Training personnel conduct formal  classroom training with multiple TSOs.
Informational briefings, bulletins, and memos—These include oral  briefings by TSA trainers or supervisors in addition to notifications  TSA headquarters sends to field personnel. These methods may be  used to notify the field personnel of standard operating procedure  changes or other matters. Trainers conduct briefings at the beginning  of a TSO shift or may do so at another designated time, such as  following a formal training session. The notifications sent by TSA  headquarters may include “read and sign” memos, in the case of  standard operating procedure changes, or may be presented online  for other important matters.
TSA training personnel stated that they maintain a flexible approach by  using different methods to prepare TSOs to implement PIL changes since  the changes have differed in their complexity, and therefore some PIL  changes require less training and preparation than others. TSA training  personnel stated they work closely with the Office of Security Operations  (OSO) to determine the proper approach to prepare TSOs to implement  each change. As an example of how the training approach can vary  based on the nature of the PIL change, the TSA training officials cited the  2005 change to prohibit all lighters from sterile areas or aircraft as one  that required less TSO preparation, in terms of training, compared with  the 2013 proposal to remove small knives. This was because the small  knives proposal encompassed more variables with regard to which knives  could be allowed (e.g., length of knife, type of knife, etc.) and therefore  required more evaluation and judgment on the part of the TSOs to  implement and operationalize the change correctly. By contrast, for the  lighters change, TSOs simply had to know they would not allow any  lighters past the checkpoint.
section_title: TSA Used Web-Based Training and Briefings to Prepare TSOs to Implement the PIL Change to Remove Small Knives
paragraphs: In developing training for the rollout of the small knives decision, TSA  required all TSOs to complete web-based training, individually or as a   TSA’s web-based training  group, covering the specifics of the change.was followed by a “training brief” that a TSA trainer would provide either  (1) immediately following a web-based training group session or (2) as  part of a “shift brief” at the beginning of TSOs’ work period (after  completion of the web-based session) in order to allow TSOs to ask  questions and gain clarity on the specifics of the PIL change. TSA  required TSOs to complete all training within a 20-day window prior to the  planned implementation of the approved knives proposal.
TSA’s web-based training sessions on the knives decision included  images that provided examples of knives and sporting equipment that  would not be allowed under the new guidelines. As shown in figure 2,  these examples included illustrations of knives that would not be allowed  into secure areas or on board aircraft because of their size (length greater  than 2.36 inches, width greater than 0.5 inch) and design features (e.g.,  locking blades, hand-molded grip, etc.) of knives that should be prevented  from being carried into sterile areas and on board aircraft. In addition, the  training included X-ray images to train TSOs on what an allowed and a  disallowed knife would look like on the screen. TSA’s web-based training  also covered the new procedures associated with knives that TSOs were  to follow at the checkpoint, such as requiring travelers to remove any  knives they may be carrying from their carry-on baggage or their person  so that these items may be screened separately. Last, the web training  tested TSOs in their knowledge of the new guidelines for the upcoming  PIL change.
Similar to the web-based training, TSA’s training brief included example  images of allowed/disallowed knives and sporting equipment. The training  brief also included coverage of the revised standard operating procedures  associated with this PIL change.
section_title: Conclusions
paragraphs: Proposals to add or remove items from TSA’s PIL can have critical  impacts, not just for the security of millions of air travelers each year, but  on the efficiency and effectiveness of passenger screening at airport  security checkpoints and perceptions of risk by external stakeholders.  Making determinations about potential PIL changes can take time and  extensive consideration on the part of TSA as the agency balances its  aviation security goals with efficient passenger throughput. While we  commend TSA’s efforts to consider the risk posed by each item on the  PIL, and potential screening efficiencies that may be created by allowing  small knives and other items to be carried onto aircraft, conducting the  analyses to demonstrate the potential efficiencies and to show that such  efficiencies would offset the added risk presented by allowing small  knives to be carried on board aircraft would help ensure that critical  changes to the PIL will have the intended impact on both security and  efficiency. These types of analyses would be consistent with the previous  recommendation we made that TSA should strengthen its evaluation of  proposed modifications to checkpoint screening procedures. Further, TSA  stated in its risk assessment and other planning documents that it would  be critical to involve stakeholders in its deliberations regarding the change  to the PIL. However, by not taking the necessary steps to sufficiently  consult with relevant external stakeholders who may be directly affected  by the proposal to allow small knives onto aircraft, TSA ultimately  reversed its decision to implement the small knives change to the PIL  after having already publicly announced its decision and invested  resources in training and implementation. Developing a formal process for  stakeholder coordination when making changes to the PIL would help to  ensure that TSA’s decisions to change the PIL are fully informed by  stakeholder perspectives, and help to ensure the efficient use of agency  resources when revising and implementing PIL policies.
section_title: Recommendation for Executive Action
paragraphs: To help ensure its proposed PIL modifications fully account for the views  of key external stakeholders in the aviation industry, we recommend that  the Transportation Security Administration’s Administrator establish a  formal process to ensure the solicitation of input from relevant external  stakeholders on proposed changes to the PIL, including when in the PIL  modification process TSA officials are to coordinate with such  stakeholders, before deciding to make a PIL change.
section_title: Agency Comments and Our Response
paragraphs: We provided a draft of this report to DHS for comment.  DHS provided  written comments, which are summarized below and reproduced in full in  appendix I. TSA concurred with our recommendation and described  actions planned to address it. In addition, DHS provided written technical  comments, which we incorporated into the report as appropriate.   In concurring with our recommendation, DHS agreed with the need for a  formal process to ensure the solicitation of input from relevant external  stakeholders on proposed changes to the PIL. DHS stated that TSA’s  senior leadership team works year-round to build and maintain strategic  partnerships with various stakeholders to develop policy, share best  practices, and participate in setting industry security standards, among  other things, and that a formal process for making changes to the PIL will  build upon these activities to ensure relevant stakeholders are offered the  opportunity to engage with TSA and inform its decisions. DHS stated that  a formal process should also make stakeholder engagement more  disciplined and concise and result in decisions that are viable and  acceptable. TSA has identified the Office of Security Policy and Industry  Engagement and the Office of Security Operations as the appropriate  offices to create such a process and plans for them to work closely with  the Office of Intelligence and Analysis, the Office of the Chief Risk Officer,  and the Office of Chief Counsel. TSA plans to create such a formal  process by November 30, 2015. This process, when fully implemented,  should address the intent of our recommendation.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the appropriate  committees and the Secretary of Homeland Security. In addition, the  report will be available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7141 or groverj@gao.gov. Contact points for our Office   of Congressional Relations and Public Affairs may be found on the last  page of this report. Key contributors to this report are listed in appendix  III.
section_title: Appendix I: Comments from the Department of Homeland Security
section_title: Appendix II: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Chris Ferencik (Assistant  Director), Dan Rodriguez (Analyst-in-Charge), Mike Harmond, Brendan  Kretzschmar, Thomas Lombardi, Stanley Kostyla, Susan Hsu, Kathryn  Godfrey, Linda Miller, and Eric Hauswirth made key contributions to this  report.