Aviation Security: Actions Needed to Systematically Evaluate Cost and Effectiveness Across Security Countermeasures
Released: 2017-09-11T00:00:00

Letter

Background

Enacted in November 2001, the Aviation and Transportation Security Act (ATSA) established TSA as the primary federal agency responsible for implementing and overseeing the security of the nation’s civil aviation system. In accordance with ATSA, TSA is to ensure that all passengers and property transported by commercial passenger aircraft to, from, or within the United States are adequately screened. Among other things, TSA is responsible for ensuring that for all flights and flight segments originating in the United States, such screening takes place before boarding and is carried out by a federal government employee except as otherwise permitted in statute. Pursuant to TSA-established policies and procedures in effect at about 440 airports at which TSA performs, or oversees the performance of screening operations (i.e., TSA-regulated airports), all passengers, their accessible property, and their checked baggage are to be screened prior to entering the sterile area of the airport or boarding the aircraft. Among other things, these procedures generally provide that passengers pass through security checkpoints where their person, identification documents, and accessible property are screened by Transportation Security Officers (TSO).

Overview of Selected Aviation Security Countermeasures

In this report, we examine six countermeasures specific to aviation security—passenger prescreening (Secure Flight), checkpoint screening, checked baggage screening, explosives detection canines, BDA, and FAMS. An overview of these countermeasures is provided below and figure 1 depicts an illustrative example of the process by which an aviation passenger may encounter these selected countermeasures.

Passenger Prescreening (Secure Flight): TSA uses its Secure Flight prescreening program to match passenger information against federal government watch lists and other information to assign each passenger to one of three risk categories—high risk, low risk, or unknown risk—that either corresponds to the level of screening they will experience at the checkpoint or may deny them an opportunity to board the aircraft. The program requires U.S.- and foreign-flagged commercial aircraft operators traveling to, from, within, or overflying the United States, as well as U.S. commercial aircraft operators with international point-to-point flights, to collect certain information from passengers—such as full name, gender, and date of birth—and transmit that information electronically to TSA. The Secure Flight program then identifies passengers’ risk levels by matching them against federal government watch lists—for example, the No Fly List, comprised of individuals who should be precluded from boarding an aircraft, and the Selectee List, comprised of individuals who should receive enhanced screening at the passenger security checkpoint. Passengers identified as matching the No Fly List, for example, are precluded from obtaining a boarding pass and proceeding through the screening checkpoint. For passengers matching the Selectee List, air carriers must mark their boarding passes accordingly so TSA can identify them for enhanced screening.

In 2010, TSA began using risk-based criteria to create additional lists for Secure Flight screening, which are composed of high-risk passengers who may not be in the Terrorist Screening Database but whom TSA has determined should be subject to enhanced screening procedures. TSA also began conducting watch list matching against an Expanded Selectee List in order to designate more passengers who are known or suspected terrorists as selectees for enhanced screening. In addition, as part of TSA Pre✓™—a 2011 initiative to preapprove passengers for expedited screening—TSA uses Secure Flight to screen passengers against several lists of preapproved low-risk travelers. Passengers determined to be eligible for TSA Pre✓™ are identified as such on their boarding passes.

Checkpoint Screening: TSA screens individuals and property at airport screening checkpoints to deter and prevent the carriage of any unauthorized or prohibited items on board an aircraft or into the airport sterile area. In general, passengers undergo one of three types of checkpoint screening, based on the Secure Flight determinations shown on boarding passes—standard screening, enhanced screening for selectees, and expedited screening for low-risk passengers. Standard screening typically includes passing through a walk-through metal detector or advanced imaging technology (AIT) machine, which identifies objects or anomalies on the outside of the body. Passengers may also be subject to a pat down if they are screened by the AIT or walk-through metal detector and the equipment alarms. Standard screening also typically includes X-ray screening for the passenger’s accessible property. During X-ray examination of the property, TSOs review the X- ray images, and if potential prohibited items are detected, the property will be manually inspected and screened with an explosives trace detection (ETD) machine to identify any traces of explosives material. Enhanced screening generally includes, in addition to the procedures applied during a typical standard screening experience, a pat-down and an explosives trace detection or physical search of the interior of the passenger’s accessible property, electronics, and footwear. Expedited screening typically includes walk-through metal detector screening and X-ray screening of the passenger’s accessible property, but unlike in standard screening, travelers do not have to, among other things, remove their belts, shoes, or light outerwear.

Checked Baggage Screening: TSA inspects passengers’ checked baggage to deter, detect, and prevent the transport of any unauthorized explosive, incendiary, or weapon onboard an aircraft. Checked baggage screening is accomplished through the use of explosives detection systems (EDS)—which use X-rays with computed tomography technology to automatically measure the physical characteristics of objects in baggage and trigger an alarm when objects that exhibit the physical characteristics of explosives are detected—and ETD machines, which use chemical analysis to manually detect traces of explosive materials’ vapors and residue. At airports with EDS, EDS machines are generally employed for primary screening of checked baggage while ETD machines are used for secondary screening to help resolve questions raised by EDS screening. At airports without EDS machines, ETDs are used as the primary method for screening checked baggage.

Explosives Detection Canines: TSA’s National Explosives Detection Canine Team Program trains, deploys, and certifies explosives detection canine teams in order to deter and detect the introduction of explosive devices into U.S. transportation systems. Each canine team consists of a handler paired with a canine trained in explosives detection. The canine handlers are generally either a state or local law enforcement officer (LEO) or a TSA employee. Two types of LEO teams and two types of TSA-based teams were trained to operate in the aviation environment during fiscal year 2015. First, TSA explosives detection canine teams patrol terminals, curbside areas, and other airport environments while TSA passenger screening canine teams primarily search for explosives odor on passengers in airport terminals. Second, LEO aviation teams patrol airport terminals, curbside areas, and sterile areas while LEO multimodal teams operate in the airport environment and screen air cargo but also operate in mass transit and maritime environments.

Behavior Detection and Analysis: TSA’s BDA program employs behavior detection officers (BDO) at passenger screening checkpoints to identify potential threats by observing individuals for certain behavioral indicators—behaviors indicative of stress, fear, or deception. These behavioral indicators include, for example, assessing the way an individual swallows or the degree to which an individual’s eyes are open. According to TSA, these verbal and nonverbal cues and behaviors may indicate mal-intent, such as the intent to carry out a terrorist attack, and provide a means for TSA to identify passengers who may pose a risk to aviation security and refer them for additional screening. During this referral screening, if passengers exhibit additional such behaviors, or if other events occur, such as the discovery of a suspected fraudulent document, BDOs are to refer these passengers to a LEO for further investigation. In fiscal year 2015, the program deployed BDOs primarily in teams of two at passenger screening checkpoints. However, TSA officials reported that in the summer of 2016, the agency began taking steps to integrate BDOs into the TSO workforce by assigning BDOs to the travel document checker position and other positions at passenger screening checkpoints where they are able to observe and interact with passengers in the performance of their screening duties.

U.S. Federal Air Marshal Service: FAMS deploys federal air marshals on passenger flights to detect, deter, and defeat hostile acts targeting U.S. air carriers, airports, passengers, and crews. In accordance with ATSA, as amended, TSA is authorized to deploy federal air marshals on every passenger flight of a U.S. air carrier and is required to deploy federal air marshals on every such flight determined by the Secretary of Homeland Security to present high security risks, with nonstop, long- distance flights, such as those targeted on September 11, 2001, considered a priority. One of FAMS’s top priorities is to deploy air marshals on flights that have a known or suspected terrorist on board. When FAMS assigns air marshals to cover such flights, it refers to these flights as special mission coverage assignments.

TSA’s System of Aviation Security Countermeasures

TSA uses a risk management strategy—referred to as “layers of security”—whereby TSA simultaneously deploys a mix of screening and other security countermeasures to deter and detect threats. TSA deploys countermeasures in varying combinations at each airport based on available resources, specific security concerns, and the airport’s risk category, among other things. Since the terrorist attacks of September 11, 2001, TSA has implemented and added countermeasures, and refined security procedures in response to specific attacks or threats— such as the liquid explosives plot in 2006. Figure 2 depicts examples of this progression, illustrating the addition or enhancement of certain TSA countermeasures over the years.

TSA Has Effectiveness Data on Some Countermeasures That Show Mixed Results, But Does Not Measure Deterrence

Data on the Effectiveness of Selected Countermeasures in Detecting and Disrupting Threats to Aviation Security Vary in Extent and Reliability

TSA collected fiscal year 2015 data on the effectiveness of four of the six countermeasures we selected—passenger prescreening, checkpoint screening, checked baggage screening, and explosives detection canines—in detecting or disrupting threats to passenger aviation security. TSA assesses this effectiveness differently for each of these four countermeasures. For example, TSA assessed the effectiveness of its passenger prescreening countermeasure in detecting passengers that may pose a threat to aviation security by measuring the percentage of airline passenger records vetted through its Secure Flight system and the number of high-risk passengers identified. In contrast, TSA assessed the effectiveness of its canine program in detecting and disrupting potential security threats by measuring canine-handler team performance during their annual certification tests as well as covert scenario-based tests called short notice assessments (SNA).

Some of the effectiveness data TSA has for fiscal year 2015 are of limited reliability and TSA is taking steps to improve this information. For instance, we reported in September 2016 that checkpoint and checked baggage screening effectiveness data from TSA’s Aviation Screening Assessment Program (ASAP) Advantage covert tests conducted in fiscal year 2015 were not reliable. Specifically, TSA found that TSOs performed more poorly in ASAP tests conducted by an independent contractor than in the same tests conducted by local TSA personnel at the same airports. This raised questions about the validity of ASAP tests conducted by local TSA personnel and indicated that TSA’s fiscal year 2015 ASAP pass rates likely showed a higher level of TSO performance in screening for prohibited items than was actually the case. In response to this issue, and to provide ongoing quality assurance for field-based covert testing results, in April 2016, TSA began deploying headquarters- based covert testing teams in both the checkpoint and checked baggage screening environments. TSA officials stated that comparing the results of field- and headquarters-based tests provides TSA with a useful indication of whether or not the field-based covert testing results are valid.

In another example, we determined that fiscal year 2015 SNA data were not reliable for the purpose of reporting explosives detection canine teams’ covert testing pass rates. Specifically, in the course of our review we found that these data included duplicate entries and errors, and TSA officials stated that the results of an unknown number of SNAs may not have been recorded. Further, we found that TSA’s data collection process for SNA results that were recorded lacked procedures to ensure that manually entered data were accurate and complete. To address these data limitations, canine program officials stated that a new process was implemented in October 2016 to incorporate SNA results directly into the Canine Website System—a central electronic management database for various canine program data. According to these officials, this new process will better ensure that SNA data are complete, accurate, and reliable for use by program officials and TSA leadership in evaluating the effectiveness of the program. Appendix II presents specific fiscal year 2015 effectiveness data for the four selected countermeasures for which TSA had effectiveness information.

During fiscal year 2015, TSA did not collect data on the effectiveness of two of the six countermeasures we selected—FAMS and the BDA program—in detecting and disrupting threats to aviation security. For FAMS, TSA officials explained that it is very difficult to empirically measure the effectiveness of federal air marshals and the program has no efforts underway to collect such data. We discuss this issue later in this report.

For the BDA Program, we reported in November 2013 that TSA had not demonstrated that BDOs could consistently identify the behavioral indicators and, further, that decades of peer-reviewed, published research on the complexities associated with detecting deception through human observation also called into question the scientific basis for TSA’s behavior detection activities. As a result, we recommended that TSA limit future funding for the agency’s behavior detection activities until TSA can provide scientifically validated evidence that demonstrates that behavioral indicators can be used to identify passengers who may pose a threat to aviation security. DHS did not concur with the recommendation but has since reduced funding for the BDA Program and taken steps to begin to assess program effectiveness. For example, in 2014 TSA revised its list of behavioral indicators and contracted for a literature review to identify additional sources of evidence supporting these indicators. However, in July 2017, we reported that in our review of all 178 sources TSA cited in support of its revised list, we found that 98 percent (175 of 178) did not provide valid evidence applicable to the specific indicators TSA identified them as supporting. Based on our findings, we continue to believe that TSA should limit future funding for the agency’s behavior detection activities until TSA can provide valid evidence that demonstrates that behavioral indicators can be used to identify passengers who may pose a threat to aviation security, as we recommended in our November 2013 report.

Table 1 identifies whether TSA has information on the effectiveness of the six selected countermeasures in detecting and disrupting threats to aviation security during fiscal year 2015, the data limitations we identified, and steps TSA officials have taken to improve this effectiveness information.

TSA Effectiveness Data on Selected Countermeasures Indicate Mixed Results

Some of TSA’s fiscal year 2015 data indicate countermeasure effectiveness while other data highlight vulnerabilities in the agency’s ability to detect and disrupt threats to aviation security. For example, for the passenger prescreening countermeasure, TSA officials reported that in fiscal year 2015, TSA’s Secure Flight program vetted 100 percent of the more than 816 million records of passengers who flew into, out of, over, or within the United States, and on U.S.-flagged aircraft operating internationally point-to-point. In addition, for the checkpoint and checked baggage countermeasures, TSA uses Annual Proficiency Reviews (APR) to evaluate TSOs’ skill in performing various checkpoint and checked baggage screening functions, such as pat downs of passengers, bag searches, and use of explosives detection equipment. In 2015, the average rate at which TSOs passed all APR component tests on the first try was nearly 95 percent.

On the other hand, some fiscal year 2015 effectiveness data indicate vulnerabilities. For example, results from covert testing conducted by TSA’s OOI during fiscal year 2015 indicate vulnerabilities in the checkpoint and checked baggage screening systems. Specific details about OOI’s test results are omitted because the information is classified.

TSA Does Not Measure Deterrence for Any of Its Aviation Security Countermeasures

While TSA has methods to measure its effectiveness in detecting and disrupting threats, the agency has no such methods to measure progress toward its goal of deterring attacks on the U.S. aviation system. TSA officials have cited the deterrent effect of various countermeasures— including FAMS, canine teams, BDOs, and AIT machines—but does not have information on the deterrent effect of any of these countermeasures. For example, TSA officials explained that canine teams that patrol airports—searching unattended bags and unattended vehicles, among other activities—provide a deterrent presence at airports, but officials noted that they do not have any data on these canines’ deterrent effect. Most notably, with regard to FAMS, TSA officials explained that one of the primary security contributions and a key aspect of the FAMS’s mission is to deter attacks. However, FAMS officials explained that they do not have information on FAMS’s deterrent effect because it is difficult to model, measure, and quantify. TSA officials in multiple offices explained that this difficulty applies not just to FAMS, but also to other TSA countermeasures with an intended deterrent effect.

OMB and GAO have acknowledged the difficulty in measuring the effect of deterrence programs, but have identified options to overcome these challenges. OMB guidance recognizes that programs with a deterrence or prevention focus can be difficult to measure and suggests that proxy measures that are closely tied to the outcome can be used to determine how well a deterrence process is functioning. We have similarly acknowledged such methodological challenges and identified alternate evaluation methods that could be helpful to agencies, such as using simulations. TSA could, for example, develop theoretical game scenarios and have testers simulate would-be attackers’ decisions when attempting to carry out an attack on the aviation system. Officials with CREATE—a DHS-funded research center—told us that they have conducted some conceptual research on the value of deterrence and believe it would be possible to assess TSA’s deterrent effect by, for example, allowing covert testers to choose their method of attack. Such an assessment could provide TSA with insights regarding which countermeasures a would-be attacker might choose to avoid in various scenarios.

In a March 2016 report prepared for TSA, CREATE analyzed a prospective risk-based security initiative TSA had begun developing and highlighted the need for further research into deterrence including the need to model the economic value of deterrence. CREATE officials explained that they highlighted this issue because in a resource constrained environment, optimizing TSA’s deterrent effect may be a more cost effective solution to aviation security threats than focusing solely on detection and interdiction. A senior official with CPER stated that the office believes there is value in pursuing further research regarding deterrence and noted that the office had included a request for funding to study deterrence in its fiscal year 2017 expenditure plan, but the request was on hold due to limited funding.

In accordance with GPRA, as updated by the GPRA Modernization Act, agencies are to establish performance measures to assess progress toward goals. Measuring performance allows organizations to track the progress they are making toward their goals and gives managers critical information on which to base decisions for improving their progress. For example, they can use performance information when developing strategies, allocating resources, identifying problems, and taking corrective action.

TSA officials told us that developing a means to assess TSA’s deterrent effect would be difficult and require a multi-year effort but having such a means would be helpful. For example, TSA’s prior Chief Risk Officer told us that TSA’s countermeasures deter nefarious actors from attempting an attack on an aircraft, but better understanding this concept will be critical to TSA in its transition into a more holistic, system-wide approach to aviation security. Additionally, a senior ORCA official explained that a better understanding of the deterrent effect of TSA countermeasures could help TSA optimize use of its resources. For example, this official noted that there may be a point at which adding additional federal air marshals has diminishing returns in terms of deterrence and better understanding FAMS’s deterrent effect could help TSA identify that point. This official further stated that developing a method to assess deterrence for this purpose would be challenging but feasible.

In the absence of any systematic or methodological approach to assessing TSA’s deterrent value, TSA officials have relied on theories of causality and limited evidence available from U.S. intelligence sources. For example, FAMS officials cited the fact that there has not been a hijacking on a U.S. carrier since 2002 as evidence of FAMS’s deterrent effect, but had no specific evidence to support FAMS’s contribution to this outcome. In another example, ORCA officials noted that a 2014 article in an online magazine published by al-Qaeda encouraging would-be- attackers to avoid airports with a certain countermeasure provided evidence of its deterrent value. These observations may provide limited insight into TSA’s deterrent effect, but developing a method to systematically assess the deterrent effect of TSA’s security efforts would better position TSA to improve progress toward its goal—deterring attacks on the U.S. aviation system.

TSA Can Compare the Effectiveness of Certain Combinations of Aviation Security Countermeasures, but Does Not Systematically Analyze Cost and Effectiveness Tradeoffs Across All Countermeasures TSA Has a Tool to Assess the Security Effectiveness of Alternate Combinations of Some Countermeasures

In 2014, TSA’s ORCA began using a Risk and Trade Space Portfolio Analysis Tool (RTSPA) to analyze the security effectiveness of alternate combinations of some aviation security countermeasures for the purpose of informing TSA acquisition and deployment decisions. RTSPA provides a means for TSA to model its security effectiveness in different scenarios. For example, the tool could be used to compare the security effectiveness of a theoretical airport screening checkpoint with canines to that of a checkpoint modeled without canines.

According to ORCA officials, they developed RTSPA to assess security effectiveness tradeoffs among countermeasures that they believed would most benefit from the detailed quantitative analyses that the tool provides, rather than across TSA’s entire system of aviation security countermeasures. Specifically, TSA officials explained that RTSPA is designed to analyze tradeoffs among checkpoint screening countermeasures—including canine teams and BDOs—and checked baggage screening, but was not developed to analyze tradeoffs among other countermeasures TSA deploys. For example, ORCA officials told us that the tool was not developed to analyze crew vetting or FAMS because understanding the security tradeoffs of these countermeasures, while important, does not require the use of such a resource intensive tool like RTSPA. In addition, RTSPA does not account for the full system of aviation security countermeasures, including countermeasures such as hardened cockpit doors and Federal Flight Deck Officers—flight crew members authorized and trained to use firearms. ORCA officials further explained that in 2014, when initially developing the tool, they also developed comparable countermeasure cost data to allow for cost- effectiveness comparisons among countermeasures. However, ORCA officials report that they subsequently stopped analyzing cost tradeoffs because they believed other TSA offices could conduct such analysis.

In the last two years, TSA officials have used the results of RTSPA analyses to inform some resource tradeoff decisions. For example, ORCA officials told us that in 2015, TSA leadership used the results of a RTSPA analysis when considering options for improving overall security effectiveness at airports that did not have AIT machines. Specifically, TSA used RTSPA to consider the level of risk and potential risk mitigation value of alternative security measures at these airports. TSA officials report that this RTSPA analysis contributed to TSA’s decision to deploy 146 additional AIT machines to such airports. In another example, ORCA officials noted that in early 2017, they used RTSPA to analyze options for resolving checked baggage alarms, taking into consideration the relative risks of military-grade explosive materials and homemade explosive devices.

TSA officials stated that their use of RTSPA has been limited to date because it is still a relatively new tool. However, ORCA officials told us that they expect use of the tool’s analysis to grow as the agency increasingly seeks to use analytic tools to inform acquisition and deployment decisions. As such, ORCA officials plan to update RTSPA and expand its analytical capabilities.

TSA Has Not Systematically Analyzed Potential Cost and Effectiveness Tradeoffs Across the Entire System of Aviation Security Countermeasures

TSA does not have any efforts underway to systematically evaluate the potential cost and effectiveness tradeoffs across the full aviation security system. Although TSA’s use of RTSPA to identify effectiveness tradeoffs among selected countermeasures provides some such information, the tool’s analyses are limited and the tool is not designed to offer a system- wide view of effectiveness. When we asked TSA’s prior Chief of Staff about any such efforts, he stated that TSA had not systematically evaluated cost and effectiveness tradeoffs because TSA’s aviation security system is constantly evolving to meet emerging threats, and assessing a system in flux is challenging. However, he told us that such an analysis would be helpful.

DHS policy and TSA’s strategic plan call for the systematic evaluation of the costs and effectiveness of TSA’s chosen mix of aviation security countermeasures. Specifically, DHS’s 2010 Policy for Integrated Risk Management calls on components, including TSA, to evaluate the performance of risk management strategies it decides to implement. In the case of TSA, TSA’s chosen mix of aviation security countermeasures represents TSA’s current risk management strategy. The policy further establishes that components should develop and analyze alternative strategies to manage risks by considering the projected costs, benefits, and ramifications of each alternative. In addition, TSA’s current Strategic Plan establishes the goal of increasing efficiency and operational effectiveness through disciplined processes and dynamic resource management. One of the stated outcomes associated with this goal is the ability to effectively optimize resource allocation to strike a balance of costs, benefits, and risk. In addition, it was the stated objective of ORCA’s predecessor—the Office of Security Capabilities (OSC)—to develop and implement a comprehensive tradeoff analysis across the security system to inform investment decisions. OSC’s strategic plan further states that such an analysis would include a full set of strategic choices TSA should consider when determining how to respond to a threat or making an investment decision, helping to determine which alternatives provide the greatest risk mitigation value for each dollar spent.

A senior ORCA official explained that while there is a need for a system- wide tradeoff analyses, RTSPA alone may not be the right tool for this. This official explained that TSA may not require detailed quantitative analyses from a resource-intensive tool such as RTSPA to understand the effectiveness tradeoffs among all aviation security countermeasures, and a portfolio of tools of varying precision and depth could be used to obtain a system-wide view. This official noted that developing TSA’s capability for system-wide tradeoff analysis would be challenging and require a multi-year effort. However, RTSPA could serve as a useful starting place for a more comprehensive system-wide analysis. For example, TSA could build upon ORCA’s past efforts to analyze the comparative cost effectiveness of countermeasures and its experience isolating the security effectiveness contributions of individual countermeasures.

Without a systematic analysis of the cost and effectiveness tradeoffs across aviation security countermeasures TSA is limited in its ability to achieve its stated goal of optimizing resource allocation and striking a balance of costs, effectiveness, and risk across the system. In an environment of constrained resources and continuing threats to aviation security, producing such analysis could assist TSA leadership in targeting its limited resources to achieve the greatest system-wide risk mitigation value for each dollar spent.

Conclusions

Since the terrorist attacks of September 11, 2001, TSA has spent billions of dollars on a range of aviation security programs with the goal of detecting, disrupting, and deterring threats. However, TSA does not have a complete understanding of the contributions these programs are making to this goal. Specifically, TSA has some information on how well it can detect and disrupt threats and is taking steps to improve this information, but does not have information on its ability to deter attacks—a key component of TSA’s goal. For example, in fiscal year 2015, TSA spent approximately $800 million on FAMS—a program with a focus on deterring attacks on aircraft—yet the agency has no information on its effectiveness in doing so. While we and OMB have acknowledged the difficulty in measuring deterrence, we have also suggested options to overcome these challenges. Further, in accordance with GPRA, as updated by the GPRA Modernization Act, agencies are to assess the effectiveness of their programs and leading practices established in GAO’s prior work stress the importance of agencies tracking progress toward goals. Developing a method to assess the deterrent effect of aviation security countermeasures would better position TSA to improve progress toward a key goal—deterring attacks on the U.S. aviation system.

Since September 11, 2001, TSA has added countermeasures and refined security procedures in response to specific attacks or threats, but has not systematically evaluated its chosen combination of aviation security countermeasures as called for in DHS policy and TSA’s strategic plan. Specifically, TSA does not have any efforts underway to evaluate the potential cost and effectiveness tradeoffs across the full aviation security system because, according to a senior TSA official, the aviation security system is constantly evolving in response to emerging threats, and assessing a system in flux is challenging. However, it is using a model— known as RTSPA—that could serve as a useful starting place for a more comprehensive system-wide analysis. Developing and implementing a means to systematically evaluate the potential cost and effectiveness tradeoffs across aviation security countermeasures would better position TSA to achieve its stated goal of optimizing resource allocation and striking a balance of costs, effectiveness, and risk. In an environment of constrained resources and continuing threats to aviation security, producing such an analysis could assist TSA leadership in targeting its limited resources to achieve the greatest system-wide risk mitigation value for each dollar spent.

We recognize that developing these analytical methods will be a difficult undertaking that may take years to achieve. Nonetheless, as TSA improves the reliability and extent of its countermeasure effectiveness data, the agency will also improve its ability to perform system-wide cost and effectiveness tradeoff analyses. In this high threat environment, it is essential that TSA determine how to allocate its finite resources to best position the agency to detect, disrupt and deter threats to aviation security.

Recommendations for Executive Action

We are making the following two recommendations to TSA: 1. The Administrator of TSA should explore and pursue methods to assess the deterrent effect of TSA’s passenger aviation security countermeasures; such an effort should identify FAMS—a countermeasure with a focus on deterring threats—as a top priority to address. (Recommendation 1) 2. The Administrator of TSA should systematically evaluate the potential cost and effectiveness tradeoffs across countermeasures, as TSA improves the reliability and extent of its information on the effectiveness of aviation security countermeasures. (Recommendation 2)

Agency Comments and Our Evaluation

We provided a draft of this report to DHS for review and comment. The department’s letter is included in appendix III. In its comments, DHS generally concurred. DHS also provided technical comments, which we incorporated as appropriate.

With regard to our first recommendation that TSA explore and pursue methods to assess the deterrent effect of its passenger aviation security countermeasures, DHS concurred, noting that this may require proxy or output measures and assumptions about potential adversary choices. DHS also concurred with our second recommendation that TSA systematically evaluate the potential cost and effectiveness tradeoffs across countermeasures. In its comments, DHS stated that TSA will continue efforts to improve both its analysis of information related to security effectiveness and its cost information, leading to better informed cost-benefit decisions for individual countermeasures. To address the intent of our recommendation, TSA will need to evaluate the costs and effectiveness of individual aviation security countermeasures and then use this information to systematically evaluate the potential cost and effectiveness tradeoffs across countermeasures. We will continue to monitor TSA’s efforts in addressing these recommendations.

We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7141 or GroverJ@gao.gov. GAO staff who made key contributions to this report are listed in appendix IV.

Appendix I: Transportation Security Administration (TSA) Information on the Direct Costs of Selected Countermeasures

As part of this review, we analyzed TSA’s fiscal year 2015 cost data for six selected aviation security countermeasures—passenger prescreening (Secure Flight), checkpoint screening, checked baggage screening, explosives detection canines, the Behavior Detection and Analysis (BDA) program, and the U.S. Federal Air Marshal Service (FAMS). We selected these six passenger aviation security countermeasures because they involved direct interaction with passengers, their belongings, or their personal information and are largely operated and funded by TSA. We determined that TSA can generally identify the fiscal year 2015 direct costs to TSA of the six passenger aviation security countermeasures that we reviewed, as shown in Table 2. TSA generally does not budget or track costs by countermeasure, but is able to identify most direct costs from their financial management system. For those passenger aviation security countermeasures that align with TSA’s budget categories, such as FAMS and passenger prescreening, TSA can run a single report to obtain the direct cost information. However, for those countermeasures that do not align with TSA’s budget categories, such as checkpoint screening and checked baggage screening, TSA is able to run multiple reports and use estimation based on their staffing model to estimate the direct costs.

Appendix II: Fiscal Year 2015 Effectiveness Data for Selected Passenger Aviation Security Countermeasures

The Transportation Security Administration (TSA) collected fiscal year 2015 data on the effectiveness of four of the six countermeasures we selected—passenger prescreening, checkpoint screening, checked baggage screening, and explosives detection canines. These data show mixed results with some data indicating TSA countermeasure effectiveness and other data highlighting vulnerabilities. Below, we describe what TSA knows about the fiscal year 2015 effectiveness of these four countermeasures in detecting or disrupting threats to passenger aviation security.

Overview of Passenger Prescreening

TSA uses its Secure Flight prescreening program to match passenger information against federal government watch lists and other information to assign each passenger to one of three risk categories—high risk, low risk, or unknown risk—that either corresponds to the level of screening they will experience at the checkpoint or may deny them an opportunity to board the aircraft. Since TSA began implementing Secure Flight in 2009, the passenger prescreening program has changed from a program that identifies passengers as high risk solely by matching them against federal government watch lists—for example, the No Fly List, comprised of individuals who should be precluded from boarding an aircraft, and the Selectee List, comprised of individuals who should receive enhanced screening at the passenger security checkpoint—to one that uses additional lists and risk-based criteria to assign passengers to a risk category. Specifically, Secure Flight now identifies passengers as high risk if they are matched to watch lists of known or suspected terrorists or other lists developed using certain high-risk criteria and as low risk if they are deemed eligible for expedited screening through TSA Pre✓™—a 2011 initiative to preapprove passengers for expedited screening—or through the application of low-risk rules. Secure Flight identifies passengers as unknown risk if they do not fall within the other two risk categories.

To separate passengers into these risk categories, TSA utilizes lists in addition to the No Fly and Selectee Lists, and TSA has adapted the Secure Flight system to perform risk assessments, a system functionality that is distinct from both watch list matching and matching against lists of known travelers. At airport checkpoints, those passengers identified as high risk receive enhanced screening, passengers identified as low risk are eligible for expedited screening, and passengers identified as unknown risk generally receive standard screening. Passengers matched to the No Fly List or the Centers for Disease Control and Prevention’s Do Not Board List—a list which includes individuals who pose a significant health risk to other travelers and are not allowed to fly—are considered highest risk, and thus are not to receive boarding passes, and should not be allowed entry into the sterile area. Figure 3 illustrates this passenger prescreening process.

The Effectiveness of Passenger Prescreening in Fiscal Year 2015

TSA officials reported that the percentage of passengers vetted and the number of high-risk passengers identified by Secure Flight demonstrate the effectiveness of this passenger prescreening program. Specifically, TSA data indicate that in fiscal year 2015, Secure Flight vetted 100 percent of the over 816 million records submitted for passengers who flew into, out of, over, or within the United States, and on U.S.-flagged aircraft operating internationally point-to-point. Of these, TSA identified 15,383 (0.002 percent of passenger records vetted) as confirmed matches to watch lists. Specifically, in fiscal year 2015, TSA identified 9,639 passengers as expanded selectees, 5,019 passengers on the Selectee List, and 725 passengers on the No Fly List.

In September 2014, we reported that TSA collects and regularly reviews data on the number of passengers identified by the Secure Flight system as potential matches to the No Fly, Selectee, and Expanded Selectee Lists. However, we found that TSA did not measure the extent to which Secure Flight was missing passengers who were actual matches to these lists—false negatives. We recommended that TSA establish such measures. In response, in August 2016, TSA contracted with a third party to conduct an independent assessment of the effectiveness of the Secure Flight automated vetting system including whether Secure Flight identifies the matches it should (i.e., how well the system minimizes false negatives). TSA officials expect this assessment to be complete at the end of calendar year 2017.

Overview of Checkpoint Screening

TSA ensures that all individuals and accessible property are screened as part of its checkpoint screening process to deter and prevent the carriage of any unauthorized explosive, incendiary, weapon, or other prohibited items on board an aircraft or into the airport sterile area—in general, an area of an airport that provides passengers access to boarding aircraft and to which access is controlled through the screening of persons and property. Ordinarily, screening of accessible property at the screening checkpoint begins when an individual places accessible property on the X-ray conveyor belt or hands accessible property to a Transportation Security Officer (TSO). As shown in figure 4, TSOs then review images of the property running through the X-ray machine and look for signs of prohibited items. If a TSO identifies a potential prohibited item, the accessible property will be manually inspected and screened with an explosives trace detection (ETD) machine to identify any traces of explosives material. The passengers themselves are typically screened via a walk-through metal detector or an advanced imaging technology (AIT) machine—often referred to as a full-body scanner—and passengers generally have the option to request screening by a pat down if they do not wish to be screened by these technologies. Passengers will also be subject to a pat down if they are screened by a walk through metal detector or the AIT and the equipment alarms (in order to resolve the alarm).

TSOs use several screening technologies in order to screen passengers and carry-on bags for prohibited items. For more information on the specific screening technologies deployed at the checkpoint in fiscal year 2015, see Table 3.

The Effectiveness of Checkpoint Screening in Fiscal Year 2015

In fiscal year 2015, TSA collected data on the effectiveness of checkpoint screening by testing TSOs, screening technology (e.g., the AIT and X- ray), and the checkpoint screening system as a whole (i.e., the combination of TSOs and technology).

Checkpoint Screening TSOs

TSA collected fiscal year 2015 data on the effectiveness of its TSO workforce in detecting or disrupting threats to aviation security at the checkpoint in three ways: (1) annual proficiency review (APR) of TSOs, (2) threat-image projection (TIP) testing, and (3) Aviation Screening Assessment Program (ASAP) Advantage covert tests.

Annual Proficiency Reviews. APRs evaluate TSOs’ skill in performing the various checkpoint and checked baggage screening functions and all TSOs must successfully complete the required APR component tests related to their job function on an annual basis as a condition of employment with TSA in their capacity as a screener. Components of the APR focused on checkpoint screening specifically included tests that evaluate TSOs’ ability to identify prohibited items on an X-ray machine and tests that evaluate whether TSOs can perform various practical skills such as pat downs, bag searches, and use of explosive trace detection technology.

In calendar year 2015, TSA conducted roughly 150,000 APR component tests focused on checkpoint screening. Table 4 provides descriptions of these component tests.

Threat Image Projection (TIP) Testing. TSA’s TIP testing system displays fictional threat items, such as guns or explosives, onto X-ray images of actual passengers’ carry-on bags to test TSOs’ ability to identify prohibited items in a live operational environment. TSOs operating the X-ray machine at the checkpoint are monitored to see if they positively identify the threat image and call for the bag to be searched. TSA officials report that they use TIP images on a daily basis to monitor TSOs’ ability to identify prohibited items, aid in keeping them focused and attentive, and keep their skills sharp in identifying items they do not routinely see. TSA requires airport personnel to conduct TIP tests and upload monthly results data into TSA’s national database.

In September 2016, we reported that TSA’s TIP data from fiscal year 2009 through 2014 was incomplete as TSA could not provide TIP scores for every airport during this period. Specifically, during fiscal year 2013, nearly 14 percent of airports failed to report any TIP data. TSA officials also acknowledged that, in addition to the airports that did not report any TIP data for a year or more at a time, other airports may have reported only partial TIP results data during this same time frame. We recommended that TSA officials at individual airports submit complete TIP results to the TSA national database as required and, further, that TSA analyze national TIP data for trends that could inform training needs and improve future training and TSO performance assessments. TSA concurred with our recommendations and is taking steps to address them. Specifically, a new TIP Operations Directive was implemented in October 2016 to disseminate procedures for performance data collection and submission to improve TIP data. According to agency officials, the number of non-compliant airports decreased during fiscal year 2016. However, since these improvements occurred during fiscal years 2016 and 2017, fiscal year 2015 TIP data remained incomplete and unreliable for the purposes of assessing TSO’s effectiveness at identifying TIP images. Therefore, we do not present fiscal year 2015 TIP test results in this report.

Aviation Screening Assessment Program (ASAP) Advantage Testing. To measure TSO performance nationwide in fiscal year 2015, TSA used standardized ASAP covert tests conducted by local TSA testers at each airport. ASAP tests focused on checkpoint screening were designed to assess the operational effectiveness of TSOs in identifying and preventing prohibited items, such as knives, guns, or simulated improvised explosive devices, from being taken through the checkpoint by testers. In fiscal year 2015, TSA conducted 5,213 ASAP covert tests on checkpoint screening at 170 airports.

TSA hired a contractor in fiscal year 2015 to independently conduct ASAP standard scenario tests at 40 airports to assess the validity of TSA testing results at those airports. When comparing the contractor’s results to the local TSA testers’ results, TSA found moderate to significant differences in the two sets of test results for most of the 40 airports. According to TSA officials, TSOs generally performed more poorly in the ASAP tests conducted by the independent contractor personnel when compared to the ASAP testing conducted by the local TSA personnel—indicating that pass rates for tests conducted by local TSA personnel were likely showing a higher level of TSO performance than was actually the case. TSA officials reported that the differences in test results have led them to question the extent to which the ASAP tests accurately measure TSO performance. As a result, we do not present the fiscal year 2015 ASAP test results in this report.

To address this validity issue, in April 2016, TSA officials reported that they began using both headquarters-based covert testing teams composed of headquarters-based TSA employees and field-based covert testing teams composed of local testers in both the checkpoint and checked baggage screening environments at all airports. Both headquarters-based and field-based teams conduct the same scenario- based covert tests that were previously conducted as part of ASAP testing. TSA officials stated that comparing the results of these separate tests has provided TSA with a way to gauge the validity of its test results.

Checkpoint Screening Technology

TSA officials reported that the effectiveness of checkpoint screening technology in fiscal year 2015 is best described by each type of machine’s detection standard—the specified rate of detection each technology is required to achieve in identifying explosives or prohibited items. Specific details about TSA’s detection standards are omitted because the information is classified. Prior to acquiring and deploying a potential new screening technology, TSA conducts testing to evaluate whether potential technologies can effectively achieve the detection standards required by TSA, among other things.

Once technology is deployed in the airport environment, TSA policy requires at least daily calibration testing of each individual piece of technology deployed at the checkpoint—AIT machines, walk through metal detectors, ETDs, and X-ray machines, among others—to ensure the technology is functioning properly and able to achieve the required detection standards. For example, each day when the screening checkpoint opens, TSOs must ensure that AIT machines successfully complete an image quality verification, a calibration test, and an operational test process before they are cleared for screening operations. TSA policy requires that TSOs record the results of these tests in logbooks and, further, that any screening equipment that does not pass daily testing be immediately taken out of service.

TSA’s Checkpoint Screening System as a Whole

In fiscal year 2015, TSA collected data on the effectiveness of its checkpoint screening system as a whole—including both screening technology and TSO performance—through Red Team covert testing conducted by TSA’s Office of Inspection (OOI). In fiscal year 2015, TSA conducted numerous Red Team covert tests on checkpoint screening at a random sample of U.S. airports. During passenger checkpoint testing, each team of inspectors carries threat items, such as simulated explosive devices, through the passenger checkpoint. If the TSO identifies the threat item during screening, the inspector identifies him or herself to the TSO and the test is considered a pass. If the TSO does not identify the threat item, the inspector proceeds to the sterile area of the airport and the test is considered a failure. According to TSA, these tests are designed to approximate techniques that terrorists may use in order to identify vulnerabilities in the people, processes, and technologies that comprise the aviation security system. In addition to OOI’s Red Team testing, in fiscal year 2015 the Department of Homeland Security (DHS) Office of Inspector General (OIG) also conducted covert tests of certain TSA checkpoint operations at 8 U.S. airports that use AIT machines to screen passengers. According to the DHS OIG, the objective of the tests was to determine the effectiveness of TSA’s AIT, automated target recognition software (which displays a box around anomalies on a generic outline of a body), and checkpoint screener performance in identifying and resolving anomalies and potential security threats at airport checkpoints. The results of both the OOI Red Team and the DHS OIG’s covert tests are omitted because the information is classified.

Overview of Checked Baggage Screening

TSA inspects passengers’ checked baggage to deter, detect, and prevent the transport of any unauthorized explosive, incendiary, or weapon onboard an aircraft. Checked baggage screening is accomplished through the use of explosives detection systems (EDS)—which use X- rays with computed tomography technology to automatically measure the physical characteristics of objects in baggage and automatically trigger an alarm when objects that exhibit the physical characteristics of explosives are detected—and explosives trace detection (ETD) machines, in which TSOs swab baggage and use chemical analysis to manually detect traces of explosive materials’ vapors and residue.

Generally, a checked baggage screening system at airports with EDS includes a three-level screening process. First, EDS machines perform automated screening. If the EDS machine determines that a checked bag requires additional screening, it sends an alarm to a TSO who performs a secondary inspection known as On-Screen Resolution by reviewing an image of the contents of the bag on a computer monitor. If the TSO cannot resolve the alarm using on-screen resolution tools and determines a physical bag search is necessary, the bag goes to the Checked Baggage Resolution Area where a TSO performs a manual inspection of the bag assisted by an ETD machine.

At the end of fiscal year 2015, TSA had 1,717 EDS machines deployed at 263 airports. At airports without EDS, which are typically smaller airports, ETD machines are the primary method for manually screening checked baggage. At the end of fiscal year 2015, TSA had 2,291 ETD machines deployed at all 437 commercial (i.e., TSA-regulated) airports for primary or secondary screening of checked baggage.

TSA officials estimate that 25 percent of total TSO time is spent on checked baggage screening, and in fiscal year 2015, this would be the full-time equivalent of approximately 11,000 of TSA’s roughly 45,000 TSOs conducting checked baggage screening.

The Effectiveness of Checked Baggage Screening in Fiscal Year 2015

In fiscal year 2015, TSA collected data on the effectiveness of its checked baggage screening by testing screening personnel (i.e., TSOs), screening technology (EDS and ETD machines), and the checked baggage screening system as a whole (i.e., the combination of TSOs and technology).

Checked Baggage Screening TSOs

In fiscal year 2015, TSA collected data on the effectiveness of its TSO workforce in detecting or disrupting threats to aviation security in the checked baggage environment through its APR evaluations and ASAP Advantage covert tests.

Annual Proficiency Reviews (APR). As discussed above, APRs evaluate TSOs’ skill in performing the various checkpoint and checked baggage screening functions. Components of the APR focused on checked baggage screening include tests that evaluate TSOs’ ability to resolve EDS machine alarms using the appropriate tools and practical skills such as bag searches and the use of ETD technology.

In calendar year 2015, TSA conducted nearly 35,000 APR component tests specific to the checked baggage screening environment. Table 5 provides descriptions of these component tests.

Aviation Screening Assessment Program (ASAP) Advantage. In fiscal year 2015, TSA used standardized ASAP covert tests conducted by local TSA testers at each airport to measure TSO performance in both the checkpoint and checked baggage environments. Tests focused on checked baggage screening were designed to assess the operational effectiveness of TSOs in identifying and preventing a threat object concealed in a checked bag from being cleared for loading onto a passenger aircraft. In fiscal year 2015, TSA conducted 1,859 ASAP covert tests on checked baggage screening at 225 airports.

TSA began deploying headquarters-based covert testing teams in fiscal year 2016 to provide a means to validate the results of covert tests conducted by local TSA testers for both checkpoint and checked baggage screening. However, unlike in the checkpoint environment, the contractor did not perform ASAP covert testing on checked baggage screening during fiscal year 2015. When we compared fiscal year 2016 headquarters-based and field-based pass rates for covert testing of checked baggage screening, we found discrepancies that indicate covert tests conducted by local field-based TSA testers on checked baggage may not be reliable in accurately portraying TSO performance. Additionally, TSA officials stated that they cannot be certain these data are reliable. As a result, we do not present ASAP Advantage data in this report.

Checked Baggage Screening Technology and TSA’s Checked Baggage Screening System as a Whole

As with checkpoint screening technology discussed above, TSA officials reported that in fiscal year 2015, technology deployed at airports for checked baggage screening was calibrated and tested daily to ensure that it was operating as intended. According to TSA officials, these daily tests help to ensure that its screening technologies are meeting the detection standards they were designed to achieve. TSA officials reported that any equipment found not to meet required detection standards was immediately taken out of service. As described above, OOI also conducted Red Team covert testing on checked baggage screening at airports with EDS machines in fiscal year 2015. Specific details about TSA’s detection standards and the results of OOI’s covert tests are omitted because the information is classified.

Explosives Detection Canines

Through its National Explosives Detection Canine Team Program, TSA trains, deploys, and certifies explosives detection canine teams in order to deter and detect the introduction of explosive devices into U.S. transportation systems. Each canine team consists of a handler— generally either a state or local law enforcement officer (LEO) or TSA employee—paired with a canine trained in explosives detection.

As of September 2015, TSA had 692 canine teams deployed to 88 airports across the United States. These teams were composed of four types of canine teams trained to operate in the airport environment: TSA explosives detection canine (EDC) and Passenger Screening Canine (PSC) teams as well as LEO aviation and multimodal teams. Table 6 shows the number of canine teams by type deployed in the airport environment as of September 2015 and describes their roles and responsibilities.

The Effectiveness of Explosives Detection Canines in Fiscal Year 2015

In fiscal year 2015, TSA collected data on the effectiveness of its canine teams in detecting or disrupting threats to aviation security through its annual certification evaluation process and short notice assessments (SNA)—covert tests conducted to assess canine teams’ operational effectiveness in detecting and responding to possible explosives.

Annual Certification Evaluations. TSA’s annual evaluations assess whether canine teams meet the explosives detection certification standards established by the program. Following initial training, new canine teams must demonstrate certain critical skills in order to be certified to work in their home operating environment. After initial certification, all TSA canine teams are evaluated on an annual basis to maintain certification. Canine teams that fail their annual evaluation are decertified and limited to training and operating as a visible deterrent until they successfully complete the annual evaluation and are recertified to conduct screening.

To achieve EDC certification, canine teams must demonstrate their ability to detect hidden explosive training aids across a specified number of areas, a certain percent of the time. After passing this conventional evaluation, PSC teams undergo further testing in different locations within the sterile area of an airport. To achieve PSC certification, canine teams must successfully identify an explosives-carrying target/decoy in a specified number of search areas.

In fiscal year 2015, TSA conducted 673 EDC annual certification evaluations and 116 PSC evaluations. The fiscal year 2015 first-time pass rates for EDC and PSC canine teams has been designated as sensitive security information and thus cannot be included in a public report.

Short Notice Assessments. TSA conducts covert testing of canine teams to measure their effectiveness in detecting and responding to explosives odor during normal operations. These covert tests, known as SNAs, are conducted using one of four scenarios chosen to match a canine team’s primary area of operations—an unattended bag, unattended vehicle, cargo screening, and passenger screening. Field Canine Coordinators—TSA officials that administer SNAs—are responsible for debriefing participants after the assessment, determining if corrective actions are necessary, and officially documenting outcomes.

We assessed the reliability of SNA results in fiscal year 2015 and determined that the data were not reliable for the purpose of reporting overall pass rates. Specifically, we found duplicate entries and errors in the data. In addition, we found that fiscal year 2015 data on pass rates may be incomplete since the results of some SNAs may not have been subsequently recorded in TSA’s system. Further, TSA’s process of manually recording SNA results in fiscal year 2015 lacked procedures to ensure that data entered into TSA’s system were accurate and complete.

To address these data limitations, canine program officials stated that a new process was implemented in October 2016 to incorporate SNA results directly into the Canine Website System—a central electronic management database for various canine program data. According to these officials, this new process will better ensure that SNA data are complete, accurate, and reliable for use by program officials and TSA leadership in evaluating the effectiveness of the program.

Appendix III: Comments from the Department of Homeland Security

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Maria Strudwick (Assistant Director), Chuck Bausell, Claudia Becker, Bryan Bourgault, Bruce Crise, Dominick Dale, Brianna Dieter, Michele Fejfar, Eric Hauswirth, Susan Hsu, James Kernen, and Tom Lombardi made key contributions to this report.