Aviation Security: TSA Improved Covert Testing but Needs to Conduct More Risk-Informed Tests and Address Vulnerabilities
Released: 2019-04-04T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: TSA uses covert testing to identify potential vulnerabilities in checkpoint and checked baggage screening systems at U.S. airports. In 2015, TSA identified deficiencies in its covert testing process, and in 2017, the Department of Homeland Security Office of Inspector General's covert testing identified deficiencies in screener performance. Since these findings, TSA has taken steps intended to improve its covert test processes and to use test results to better address vulnerabilities.

GAO was asked to review TSA's covert test programs, including how the results are used to address vulnerabilities. This report analyzes the extent to which (1) TSA covert tests are risk-informed, (2) TSA covert tests for fiscal years 2016 through March 2018 produced quality information, and (3) TSA uses covert test results to address any identified security vulnerabilities.

GAO observed 26 TSA covert tests, reviewed TSA guidance, analyzed test data for fiscal years 2016, 2017, and through March 2018, and interviewed TSA officials.

section_title: What GAO Found
paragraphs: Two offices within the Transportation Security Administration (TSA) conduct covert tests at U.S. airports—Inspection and Security Operations. The Department of Homeland Security requires that agencies use risk information to make decisions, and TSA issues annual risk assessments of threats that its program offices should consult when making risk-based decisions, such as what covert tests to conduct. Of the two TSA offices that conduct covert tests, Inspection officials used TSA's risk assessment to guide their efforts. However, Security Operations officials relied largely on their professional judgment in making decisions about what scenarios to consider for covert testing. By not using a risk-informed approach, TSA has limited assurance that Security Operations is targeting the most likely threats.

Both Inspection and Security Operations have implemented processes to ensure that their covert tests produce quality results. However, GAO found that only Inspection has established a new process that has resulted in quality test results. Specifically, for the two reports Inspection completed for testing conducted in fiscal years 2016 and 2017 using its new process, GAO found that the results were generally consistent with quality analysis and reporting practices. On the other hand, Security Operations has not been able to ensure the quality of its covert test results, and GAO identified a number of factors that could be compromising the quality of these results. Unless TSA assesses the current practices used at airports to conduct tests, and identifies the factors that may be impacting the quality of covert testing conducted by TSA officials at airports, it will have limited assurance about the reliability of the test results it is using to address vulnerabilities.

In 2015, TSA established the Security Vulnerability Management Process to leverage agency-wide resources to address systemic vulnerabilities; however, this process has not yet resolved any identified security vulnerabilities. Since 2015, Inspection officials submitted nine security vulnerabilities identified through covert tests for mitigation, and as of September 2018, none had been formally resolved through this process. GAO found that in some cases, it took TSA officials overseeing the process up to 7 months to assign an office responsible to begin mitigation efforts. In part, this is because TSA has not established time frames and milestones for this process or established procedures to ensure milestones are met, in accordance with best practices for program management. Without doing so, TSA cannot ensure efficient and effective progress in addressing security vulnerabilities.

This is a public version of a classified report that GAO issued in January 2019. Information that TSA deemed classified or sensitive security information, such as the results of TSA's covert testing and details about TSA's screening procedures, have been omitted.

section_title: What GAO Recommends
paragraphs: GAO is making nine recommendations, including that TSA use a risk-informed approach for selecting covert test scenarios, take steps to improve the quality of airport covert test results, and establish time frames and milestones for the key steps in its vulnerability management process. TSA concurred with all nine GAO recommendations.

section_title: Letter
paragraphs: Threats to commercial aviation persist and continue to evolve. In March  2017, more than 15 years after the terrorist attacks of September 11,  2001, the Transportation Security Administration (TSA) imposed new  screening measures to enhance security after intelligence agencies  confirmed that terrorist organizations had the capability to plant  explosives in personal electronic devices, such as laptops. Further, in  November 2017, the Acting Secretary of Homeland Security reported that  the aviation sector remains a primary target for terrorist activity. To help  thwart possible attacks, TSA uses covert testing as a key method to  identify possible vulnerabilities in the checkpoint and checked baggage  screening systems at TSA-regulated (i.e., commercial) airports across the  United States. During covert tests, undercover personnel (testers) attempt  to pass threat items (i.e., guns, simulated improvised explosive devices,  etc.) through checkpoint and checked baggage screening equipment  undetected. TSA’s covert tests are intended to help officials identify  vulnerabilities and then address or mitigate them through various means,  such as conducting additional training, changing existing screening  procedures, or adopting new ones.

Recent investigations identified vulnerabilities both in TSA’s checkpoint  and checked baggage screening and with its covert testing of these  processes. For example, in 2017, the Department of Homeland Security    (DHS) Inspector General identified deficiencies in TSA screener  performance. In addition, in 2016, we reported that TSA’s detection rates  for the Aviation Screening Assessment Program (its prior covert testing  program) were unreliable. In 2016 TSA redesigned its covert test  processes to strengthen test procedures and enhance the quality of  covert test data and analysis, as well as improve its use of test results to  address vulnerabilities.

Within TSA, two offices carry out covert tests of checkpoint and checked  baggage screening operations at airports: Inspection and Security  Operations. Inspection’s tests identify vulnerabilities related to any  aspect of TSA’s checkpoint and checked baggage screening systems, to  include the procedures for screening and whether the system is  vulnerable to threats identified in intelligence reporting. Security  Operations’ tests focus entirely on Transportation Security Officers’ (TSO)  performance against standard operating procedures for checkpoint and   checked baggage screening. In July 2018, TSA began a transfer of  existing covert test programs managed by Security Operations to  Inspection for the purposes of improving covert testing and increasing the  validity of data collection and reporting. Until this transfer is complete,  both Inspection and Security Operations continue to perform covert tests  at the nation’s commercial airports using distinct processes.

Given that TSA continues to refine its processes for conducting covert  tests and using the results, you asked us to review TSA’s current covert  test program, including how the results are used to address identified  vulnerabilities. This report (1) describes how TSA has changed its covert  test processes since 2016 and analyzes the extent to which these  processes are risk-informed; (2) analyzes the extent to which TSA covert  tests for fiscal years 2016 through March 2018 produced quality  information; and (3) analyzes the extent to which TSA has used the  results of covert tests to address any identified security vulnerabilities.

To understand how both Security Operations and Inspection changed  their respective covert test processes since 2016, we reviewed agency  documentation, interviewed agency officials, and observed 22 Security  Operations and four Inspection covert tests at five airports. See appendix  I for more information on how we selected airports for observations. For  all these observations, we were able to observe TSOs performing  checkpoint or checked baggage screening activities during tests. To  determine the extent to which Security Operations and Inspection testing  is risk-informed, we reviewed program documentation and spoke with  agency officials. We compared the results of TSA risk assessments to the  threat items and locations that Inspection and Security Operations  selected for tests in fiscal years 2016 and 2017. We evaluated each  office’s process for making risk-informed decisions against DHS risk  management policies, which require that agencies use risk information  and analysis to inform decision making and document risk management  methodologies.

To assess the quality of Security Operations’ test information, we  observed Security Operations tests and reviewed its efforts to assess the  quality of airport-run testing by comparing results for the same covert  tests carried out by two different groups—TSA airport staff and TSA  headquarters staff. Specifically, we calculated detection rates for 12,000  covert tests conducted in fiscal year 2017 and about 3,600 covert tests  conducted during the first half of 2018, and compared the results against  Security Operations’ internal criterion for determining quality test  information. We assessed Security Operations’ quality assurance  methods for covert testing against program criteria and federal internal  control criteria for documenting processes. To assess the quality of  Inspection’s test information, we observed Inspection’s tests, reviewed  completed reports based on fiscal year 2016 and 2017 testing, and  conducted interviews with program managers and technical experts to  identify the extent to which Inspection followed its documented  requirements for quality assurance.

To assess the extent to which Inspection and Security Operations  address security vulnerabilities, we reviewed their efforts separately  because each office used a different approach. To assess Inspection’s  efforts, we focused on its use of a new, agency-wide vulnerability  management process that Inspection designated in 2016 as the principal  means by which it addresses its identified vulnerabilities. To obtain a  more complete understanding of the extent to which TSA’s vulnerability  management process has addressed vulnerabilities identified by  Inspection, we reviewed documentation related to the process and other  information pertaining to all vulnerabilities Inspection submitted to the  process, including those that were unrelated to checkpoint and checked  baggage screening. We assessed the new vulnerability management  process against standards for program management issued by the  Project Management Institute, a not-for-profit association that provides  global standards for, among other things, project and program  management. To determine how Security Operations headquarters  officials address vulnerabilities involving screener performance, we  reviewed program documentation and interviewed program managers. To  understand how the results of covert testing are used at the airport level  to improve TSO performance, we conducted semi-structured interviews  with 10 Federal Security Directors (FSD) at airports across the United  States, and with three TSA Regional Directors. We selected FSDs for  interviews to reflect a range of airport performance on fiscal year 2017  covert tests, among other factors (see appendix I). We assessed Security  Operations’ and TSA officials at airports’ efforts against federal internal  control standards and criteria in the National Infrastructure Protection  Plan for improving program outcomes through information sharing.

This is the public version of a classified report that we issued on January  10, 2019. The classified report included an objective related to  identifying the results of covert testing for fiscal years 2016 and 2017 and  assessing the quality of this test information. DHS deemed covert testing  results (including detection rates and identified vulnerabilities) to be  classified information, which must be protected from loss, compromise, or  inadvertent disclosure. Consequently, this report omits part of an  objective identifying the results of covert testing. DHS also deemed some  of the information in our January report to be sensitive security  information, which must be protected from unauthorized release.  Therefore, this report omits information describing TSA screening  procedures, specific information related to agency risk assessments, and  airport-level covert test results.

The performance audit upon which this report is based was conducted  from September 2017 to January 2019 in accordance with generally  accepted government auditing standards. Those standards require that  we plan and perform the audit to obtain sufficient and appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  from this work provides a reasonable basis for our findings and  conclusions based on our audit objectives. We worked with DHS from  February 2019 through April 2019 to prepare this unclassified, non- sensitive version of the original classified report for public release. This  public version was also prepared in accordance with these standards.
subsections: section_title: Background
subsections: section_title: TSA’s Aviation Security Responsibilities
paragraphs: TSA is the primary federal agency responsible for implementing and  overseeing the security of the nation’s civil aviation system and is  responsible for ensuring that all passengers and property transported by  commercial passenger aircraft to, from, within, or overflying the United   States are adequately screened. Specifically, TSA performs, or  oversees the performance of, screening operations at about 440 TSA- regulated (i.e., commercial) airports nationwide. These airports range in  size from smaller airports (category III and IV airports) to larger airports  (categories X, I, and II airports). According to TSA policies and  procedures in effect at these airports, all passengers, their accessible  property, and their checked baggage are to be screened prior to entering  the airport sterile area—the portion of an airport beyond the security  screening checkpoint that provides passengers access to boarding         aircraft. Among other things, these policies and procedures generally  provide that passengers must pass through security checkpoints where  their person, identification documents, and accessible property are to be  screened by TSOs, and that all checked baggage must be screened by  TSOs.

section_title: TSA Checkpoint and Checked Baggage Screening
paragraphs: Checkpoint Screening. The checkpoint screening process, as set forth  in TSA’s procedures, is intended to deter and prevent passengers from  carrying any unauthorized or prohibited items into the airport’s sterile area  and onboard an aircraft. Upon entering the airport terminal security  checkpoint, passengers provide travel document checkers their boarding  passes for review. Based on the printed boarding pass result, travel  document checkers are to direct passengers to designated areas for  standard, enhanced, or expedited screening.

Standard screening is generally applied to all passengers with  boarding passes that are not marked for enhanced or expedited  screening. This screening typically includes passing through either a  walk-through metal detector or advanced imaging technology (the  latter of which identifies objects or anomalies concealed on the          person) and using X-ray equipment to screen the passenger’s  accessible property. In the event that any of these screening devices  identify a potential item of concern, additional security measures are  to result as part of the alarm resolution process. These measures may  include pat downs, explosives trace detection searches (which involve  a device to detect explosive particles), and colorimetric testing to  identify the concentration of certain chemical elements.

Enhanced screening is generally required for passengers TSA  identifies as high risk, such as passengers that have been matched to  federal government lists of known or suspected terrorists. Enhanced  screening involves the same procedures applied during a typical  standard screening experience, as well as a pat down and an  explosives trace detection search or physical search of the interior of  the passenger’s accessible property, electronics, and footwear.

Expedited screening is allowed for passengers TSA believes to be low  risk. One group of passengers who routinely receive expedited  screening are those enrolled in TSA’s Pre✓®—a program through  which individuals vetted and approved by TSA are eligible for this  level of screening. At airports with dedicated TSA Pre✓® lanes,  expedited screening includes walk-through metal detector screening  and X-ray screening of the passenger’s accessible property, and  travelers do not have to remove their belts, shoes, or light outerwear,  or remove items such as laptops from carry-on baggage.

Checked Baggage Screening. TSA procedures for checked baggage  screening establish a process intended to deter, detect, and prevent the  transport of any unauthorized explosive, incendiary, or weapon aboard an  aircraft. Checked baggage screening generally entails the use of  explosives detection systems—which use X-rays and other technology to  automatically measure the physical characteristics of objects in baggage  and trigger an alarm when objects that exhibit the physical characteristics  of explosives are detected.

section_title: Overview of Inspection and Security Operations Testing Processes
paragraphs: Inspection’s tests are intended to identify vulnerabilities related to any  aspect of TSA’s checkpoint and checked baggage screening systems, to  include the procedures for screening, the TSOs who implement these  procedures, and the technology for screening (e.g., X-ray machines and  advanced imaging technology). Security Operations’ testing focuses  entirely on TSO performance of existing standard operating procedures  for checkpoint and checked baggage screening, and unlike Inspection’s  testing, does not test other aspects of screening, such as the  performance of screening equipment.

To carry out covert testing, both Inspection and Security Operations  create test scenarios that describe the overall intent of the test, the threat  item, the method of execution (e.g., an explosive device concealed in a  shoe carried through the checkpoint), and other pertinent details.  Generally, Security Operations’ scenarios have tested TSOs’  performance of procedures pertaining to one of three different paths       travelers must follow to have either their persons or property screened  (i.e., screening paths):  checkpoint on-person—the tester travels through the checkpoint with  the threat item concealed on his or her person;  checkpoint in-property—the tester travels through the checkpoint with  the threat item concealed in a carry-on bag; and  checked baggage—the threat item is concealed in checked baggage.

For both offices, covert tests begin when program managers notify an  airport’s FSD and local law enforcement agency that testing is scheduled  to begin. Testers typically pose as passengers and attempt to smuggle a  threat object, concealed either on their person or in their property, through  one or more layers of the checkpoint or checked baggage screening  process (see fig. 1). These layers of screening include the travel  document checker and the walk-through metal detector or the advanced  imaging technology machine, among others.

In general, TSA’s covert tests conclude with a meeting between either  Inspection or Security Operations staff and the TSOs and their  supervisors who were tested to discuss the results. These meetings,  known as post-test reviews, allow officials to reinforce actions resulting in  test successes, review the correct procedures for any failures, and collect  additional data relating to factors contributing to success and failure. In  addition, documented test results are reported to local TSA airport  officials, so that they may schedule and track TSO participation in the  remedial training that is required by law when screeners fail a test. More  broadly, Inspection and Security Operations report test results to certain  internal and external stakeholders. Historically, Inspection has reported its  test results directly to TSA management to inform executive leadership  about the aviation screening system’s potential vulnerabilities to new and  evolving threats. In addition, Security Operations has reported test results  for its prior testing program to the Office of Management and Budget  quarterly and has also briefed TSA senior leadership on results  periodically.

section_title: Using a Risk-Informed Approach for Covert Testing
paragraphs: DHS policy requires that its components, including TSA, use risk  information and analysis to inform decision making. A risk-informed  approach helps decision makers identify and evaluate potential risks so  that actions can be taken to mitigate those risks. DHS defines risk as a  calculation of threat, vulnerability, and consequence. These elements are  defined as follows:

Threat likelihood is estimated based on intent and capability of an  adversary.

Vulnerability is a physical feature or operational attribute that renders  an entity open to exploitation or susceptible to a given hazard. In  calculating risk, vulnerability is based on the likelihood that an attack  is successful, given that it is attempted.

Consequence refers to the negative effect of an event, incident, or  occurrence.

According to the 2010 DHS Risk Lexicon, which defines key risk- management terms for DHS agencies and components, risk-based  decision making uses the assessment of risk as the primary decision  driver, while risk-informed decision making may consider other relevant  factors in addition to risk assessment information, for decision making.

To guide agency efforts to make risk-based decisions, TSA issues  annually its Transportation Sector Security Risk Assessment—a report on  transportation security that assesses risk by establishing risk scores for  various attack scenarios within different transportation sectors, including  domestic aviation. These scenarios are continuously refined to reflect  evolving threats to the various transportation modes and feedback from  subject matter experts. In scoring risk scenarios for the Transportation  Sector Security Risk Assessment, TSA considers the three elements of  risk (threat likelihood, vulnerability, and consequence).

section_title: TSA Revised Its Covert Test Processes since 2016 but Is Not Fully Using and Documenting a Risk-Informed Approach for Selecting Test Scenarios
subsections: section_title: Inspection Redesigned Its Covert Test Process to Be More Risk-Informed and Quantitative but Has Not Fully Documented Its Rationales for Selecting Test Scenarios
subsections: section_title: Inspection’s Redesigned Covert Test Process
paragraphs: In 2016, Inspection redesigned its process to conduct covert tests more  consistently across airports, and began using quantitative methods to  design tests and analyze results so that its findings might be applied more  broadly across airports nationwide. Inspection officials explained that,  prior to redesigning their process, Inspection’s findings could not be  applied more broadly because of how tests were designed and executed.  In addition, officials noted that some prior test practices risked diminishing  the quality of testing. For example, some testers consistently ran tests at  the same airports, increasing the likelihood that they might be recognized  by TSOs and compromise the covertness of tests.

As part of its new testing effort, Inspection recruited a technical team of  employees with expertise in statistics and engineering to enhance the  design, execution, analysis, and reporting of its covert tests. Inspection  also documented its new covert test process and rationales for key  program decisions, including its approach to performing quantitative  analysis of test results, in overarching guidance issued in October 2016.  These documents set forth a framework for conducting tests that includes  the creation of detailed scenarios that specify Inspection’s covert test  objectives and scope of testing. For example, for one Inspection test  scenario conducted in fiscal year 2016, Inspection conducted 280 tests at  larger airports to assess whether certain types of assembled explosive  devices contained in carry-on luggage could evade detection at the  checkpoint. Under new guidance, Inspection’s testers may not conduct  tests at the same airport within a predetermined period, to limit the  potential of being recognized by airport staff. In addition, under its new  process, Inspection selects airports for testing so that it may apply its  findings more broadly across airports nationwide. Once Inspection  testers complete all tests for a given scenario, Inspection develops  classified reports containing results of its quantitative analysis (including  detection rates for specific threat items) and suggested actions aimed at  addressing any identified vulnerabilities.

section_title: Inspection Has Not Fully Documented a Risk-Informed Approach for Testing
paragraphs: Inspection uses a risk-informed approach to select locations and  scenarios for covert tests, but has not fully documented this approach.  According to Inspection officials, to select airport locations for tests, they  use a tool to randomly select airports from various regions and of various  sizes to ensure appropriate representation. According to our review of the  locations Inspection tested in fiscal years 2016 and 2017, Inspection  predominantly conducted testing at the larger airports. As previously  discussed, this is consistent with a risk-informed approach, as TSA’s  analysis has shown that larger airports face an increased threat of a  terrorist attack.

In addition, Inspection officials said that they use a risk-informed  approach to select scenarios for their covert tests that takes into  consideration all three aspects of a comprehensive risk assessment— threat, vulnerability, and consequence. According to officials, Inspection’s  approach to each of the three components of risk is described below.

Efforts to Consider Threats. According to Inspection leadership  officials, Inspection has developed close working relationships with  key intelligence community agencies to obtain current and specific  intelligence information about threats to commercial aviation.  Inspection uses this information to create test scenarios involving  threat items and attack methods that correspond with the most current  threat intelligence. Inspection officials explained that they also consult  risk assessments such as the Transportation Sector Security Risk  Assessment to help determine which scenarios to test, but do not rely  solely on this information. Officials said this is because such  assessments can lack specificity about the type and placement of  threat items along different screening paths. For example, the  Transportation Sector Security Risk Assessment may not convey the  specific type of device or the mechanism by which an explosive  device will be presented at the checkpoint (e.g., in a laptop).  Inspection’s approach, which uses both current intelligence and risk  assessments, is consistent with a risk-informed approach, which  allows agencies to utilize resources beyond risk assessments to  inform decision making.

Efforts to Consider Vulnerability. Inspection officials told us they  have considered vulnerability as a factor for making risk-informed  decisions, and have found that it is not useful when deciding which  scenarios to test for two reasons. First, their covert testing is intended  to identify the existence of vulnerabilities in the aviation security  system. Second, officials explained that vulnerabilities at some  airports are well-documented and understood; therefore, they would  generally not use their limited resources to test a vulnerability that is  well-known.

Efforts to Consider Consequence. Inspection officials explained  that when selecting among possible scenarios to test, considering the  consequences that might result from a scenario is less important than  the likelihood of a given threat. However, Inspection officials explained  that they require that any scenario tested is one that would result in  the loss of life if the attack were actually to occur.

Although Inspection program officials could articulate the risk-informed  approach used to select scenarios for testing, they had not sufficiently  documented this approach. Specifically, we found that Inspection  documents its process for making risk-informed selections of scenarios in  formal work plans. This documentation includes general criteria that  Inspection leadership is to consider when developing threat scenarios,  one of which is threat likelihood. However, the work plans we reviewed  did not identify selection criteria that address the vulnerability or  consequence components of risk.

DHS’s Risk Management Fundamentals (2011) requires that agency  documentation include transparent assumptions about the rationale        behind risk management decisions. In addition, according to Standards  for Internal Control in the Federal Government, agencies should  document key decisions in a way that is complete and accurate.  According to Inspection officials, they have not fully documented their  risk-based process for selecting scenarios because their decision making  is often informed by unforeseen events associated with the most exigent  threats. Nevertheless, without documenting in its work plans how  consequence and vulnerability are considered when determining which  scenarios to test, current Inspection program managers may not be able  to ensure that their scenario selection decisions are appropriately  accounting for risk as called for by DHS and TSA guidance. Furthermore,  although vulnerability and consequence are less important criteria for     Inspection’s current risk-informed selections, documentation of its  approach toward each would serve as a baseline for how Inspection  makes risk-informed decisions for selecting scenarios to test. This  baseline could inform future program managers and agency leadership  seeking to make changes.

section_title: Security Operations Redesigned Its Covert Tests to Address Prior Deficiencies but Has Not Fully Incorporated Known Risks or Documented How It Selects Scenarios to Test
subsections: section_title: Security Operations Redesigned Its Covert Test Process
paragraphs: In 2016, Security Operations replaced its Aviation Screening Assessment  Program with a new covert test program. Security Operations issued  guidance for this new program that, among other things, established a  parallel test process carried out by headquarters staff to validate (i.e.,  determine the quality of) local covert test results from airports. In  conjunction with this process, Security Operations also developed and  launched a new web-based tool to collect more detailed information on  covert tests. According to Security Operations officials, the new program  is intended to address problems with its covert testing process identified  by an independent contractor in 2015. Specifically, the contractor  performed the same covert tests that TSA personnel at local airports  conducted, and the contractor’s test results showed that screeners  performed more poorly on its tests. In September 2016, we reported that,  based on the results of the contractor’s study, TSA had determined that  prior-year tests conducted by TSA officials at airports likely showed a  higher level of performance than was actually the case. Further, TSA  attributed these higher detection rates, in part, to local airport difficulties in  successfully maintaining the covert nature of their tests.

To address deficiencies identified by the TSA-contracted study, Security  Operations issued test guidance in December 2016 and January 2017  that provides more structure to the planning and execution of tests and is  intended to help ensure the quality of test results, among other things.  For example, the guidance directs local test coordinators to schedule  covert tests at varying times of day and varying days of the month, to  prevent TSOs from becoming accustomed to testing at particular times.  Also, to help ensure that testers are not recognizable by TSOs, the  guidance states that airports must not recruit testers from the airport in  which the test is to be conducted. Additionally, Security Operations’  guidance expands opportunities for recruiting testers at airports.

Security Operations’ new covert test program also features a  headquarters-based covert test effort, known as Headquarters Evaluation  Team (HET) testing, to help validate the results of covert tests conducted  by TSA officials at airports, known as Field Evaluation Team (FET)  testing. Under the new process, FET teams, which are composed of  TSA staff at airports and locally recruited testers, oversee testing at  airports where FSDs are located and at any smaller airports under the  FSD’s authority. FET teams perform tests of three different screening  paths—checkpoint in-property, checkpoint on-person, and checked  baggage—using a variety of scenarios assigned by Security Operations  program managers every 6 months. FET teams test scenarios for a  designated number of times over the 6-month period, after which,  program managers are to select and assign a new set of scenarios for  testing for the next 6-month period. For its HET tests, Security  Operations is to select, on a quarterly basis, three scenarios to test from  among the current set of scenarios assigned for FET testing. HET teams  are to travel to airports quarterly to conduct these tests and help validate  the FET testing results. Security Operations’ validation process involves  comparing detection rates—the percentage of tests in which TSA  screening recognized and prohibited a threat item from entering the  sterile area of an airport—for similar scenarios from both groups of  testers.

To assist HET and FET teams in collecting more detailed information  from its new test program, in April 2016, Security Operations developed a  web-based data collection instrument called the Task Process Factor  (TPF) tool that TSA officials use to record more detailed information on  covert tests. According to program officials, collecting more detailed  information about test failures was part of the agency’s effort to improve  screener performance following the DHS Inspector General’s 2015 covert  test findings that identified vulnerabilities in TSA’s checkpoint screening.  The tool defines the key TSO activities for conducting checkpoint and  checked baggage screening as tasks (e.g., interpret the X-ray image).  The tool also identifies the various processes associated with a given task  (e.g., move property into the X-ray scanner and stop when a full image  appears). For any task in which a TSO fails, testers are to use the TPF  tool to record the task and process associated with the failure—so that  Security Operations may identify points of failure for tests with greater  specificity. Furthermore, for all test failures, the tool requires HET and  FET testers to identify the factor, or root cause, for failure.

section_title: Security Operations Has Not Fully Incorporated or Documented a Risk-Informed Approach for Selecting Test Scenarios
paragraphs: Although Security Operations considers some TSA risk information when  selecting airport locations to test, we found that Security Operations does  not fully consider this information when determining which scenarios to  use for its covert tests, and also does not document its rationale for  choosing the scenarios it selects. According to its planning documents for  conducting HET and FET tests, Security Operations conducts more tests  at larger airports than smaller airports. According to TSA officials, this is  because larger airports generally have more TSOs who are subject to  covert testing. TSA’s decision to allocate more testing resources to larger  airports is based on its own risk analysis and, therefore, is consistent with  a risk-informed approach. However, Security Operations has not taken  steps to incorporate known risks—such as those documented in TSA’s  annual Transportation Sector Security Risk Assessment, TSA’s primary  risk assessment of threats for all transportation modes—into its process  for selecting covert test scenarios. As our prior work has shown,  implementing a risk-informed approach involves using risk assessments  or other risk information to determine the most pressing security needs  and developing strategies to address them.

In reviewing TSA’s 2016 Transportation Sector Security Risk  Assessment—the version that would have informed Security Operations’  selection of tests for fiscal year 2017—we identified numerous attack  scenarios that could have been incorporated into Security Operations’  selection of scenarios to test. Specifically, the 2016 risk assessment  included 20 scenarios that involved attacks that could be carried out  through expedited screening conducted in dedicated TSA Pre✓®  screening lanes. We reviewed all scenarios Security Operations  selected to test in fiscal year 2017, but found that only one involved a test  of the TSA Pre✓® lane. More generally, we also found that TSA’s  selection of threat items to test at the checkpoint in fiscal year 2017 did  not reflect threats identified in TSA’s 2016 Transportation Sector Security  Risk Assessment.

Security Operations officials acknowledged that they do not use formal  TSA risk assessments to determine what threat scenarios or items to test.  They also do not work with intelligence agencies or review classified  information when developing covert test scenarios. Instead, Security  Operations officials said they rely mainly on professional judgment  regarding which areas of checkpoint and checked baggage procedures  TSOs frequently overlook or may not perform correctly (e.g., pat downs).  Officials explained that their judgment is informed by monitoring covert  test results; unclassified media reports on threats; and requests from  agency leadership, such as from TSA’s Administrator. Security  Operations’ program managers further explained that because their tests  are intended to assess TSO performance of screening procedures and  identify any gaps, their selection of scenarios for testing is intended to  cover the breadth of checkpoint and checked baggage screening  procedures. However, as previously discussed, using a risk-informed  approach would allow program managers to balance other goals of  testing, such as the need to test a variety of screening procedures, with  risk information, when making decisions on what to test.

DHS’s Policy for Integrated Risk Management (2010) states that DHS  components should use risk information and analysis to inform decision  making. Additionally, the TSA Strategy 2018–2026 prioritizes structuring  programs to manage risk and optimize resource allocation. Formal risk  assessments such as the Transportation Sector Security Risk  Assessment identify the most significant risks to checkpoint and checked  baggage screening, and accordingly identify some of the most critical  skills TSOs need to detect or prevent possible attack scenarios. Using a  risk-informed approach to select scenarios that more fully account for  known risks—such as those identified in the Transportation Sector  Security Risk Assessment or a similar risk assessment—could better  ensure that TSA is using its finite testing resources to target screening  activities that will counter the most likely threats.

Additionally, DHS’s Risk Management Fundamentals (2011) requires that  agency documentation include transparent assumptions about the  rationale behind risk management decisions. However, Security  Operations has not documented its rationales for selecting covert test  scenarios in any of its overarching guidance or planning documentation.  Such rationales would delineate Security Operations’ framework for  determining what screening activities to test, and specify how Security  Operations officials balance a risk-informed selection of scenarios with  their need to test scenarios that cover the breadth of requirements within  existing screening procedures. Security Operations officials said they do  not document their scenario selection process because they review  covert test data on a frequent enough basis to identify which processes  have low detection rates and, thus, are in need of testing. However,  documenting a risk-informed rationale for its selection of scenarios would  better enable Security Operations or an external party to assess TSA’s  covert test programs and ensure that decisions are appropriately  accounting for risk as called for by DHS and TSA guidance. It would also  allow Security Operations to demonstrate how it balances its goal of  promoting a risk-informed culture, as required by DHS, with program  goals to ensure that TSOs are following all required screening procedures  correctly.

section_title: Inspection’s Updated Process Is Designed to Produce Quality Information, but Security Operations Faces Challenges with the Quality of Its Test Results
subsections: section_title: Inspection’s New Process is Designed to Produce Quality Test Results and Analysis
paragraphs: Inspection has established a new process and principles for conducting  covert tests, as well as collecting and analyzing test data, intended to  result in quality information on screening vulnerabilities. We reviewed two  reports on results of Inspection’s covert testing that were completed using  its new processes, and found they resulted in quality information on  screening vulnerabilities. With respect to its new processes  Inspection has implemented guidance to ensure a standardized  process for developing and executing tests. Specifically, Inspection  guidance requires that headquarters staff with expertise in relevant  fields (including physical security, explosives, and intelligence  analysis) develop all threat items used for testing and conceal these  items within test bags or on testers in the same manner across tests.  In addition, Inspection program managers require that testers have  detailed background stories to explain the purpose(s) of their travel.

Inspection now employs multiple standard practices to ensure test  covertness. We observed several of these practices during four  Inspection tests conducted at one airport. These four tests consisted  of two scenarios that were each tested at two different checkpoints  within the airport. First, we observed that Inspection teams notified the  FSD of their presence only immediately prior to beginning tests, to  limit the potential for local airport staff to be forewarned. We also  observed that Inspection conducted tests simultaneously across  checkpoints, and concluded testing at the airport after an initial round  of testing. According to Inspection program managers, conducting  tests simultaneously and leaving after the initial round of testing are  necessary because once TSOs at a tested checkpoint become aware  of testing, there is no reliable way to prevent this knowledge from  spreading to other checkpoints.

Inspection now integrates its technical operations team (technical  team) into all aspects of test design and data collection and analysis.  Inspection officials recruited staff with expertise in research and test  design, statistics, and systems engineering, among other relevant  fields, to analyze this information. Inspection has integrated these  staff into all aspects of its test process to ensure the quality of test  information collected and analyses performed. For example,  according to TSA documentation, Inspection technical team members  are to oversee the selection of airports for testing by first conducting  an analysis to determine the number of airports to be tested, and then  ensuring the selection of airports for testing is made using a random  process—a requirement, given that Inspection intends to use test      results to understand and describe screening activities at airports  nationwide.

Inspection now identifies data to be collected for each scenario and  monitors this data as it is being collected for quality assurance.  According to TSA documentation, Inspection’s technical team  develops the data collection forms used to record test information for  every scenario. Such data elements are specific to each scenario and  can include, for example, the time when the tester entered the  checkpoint, whether the TSO running the X-ray machine stopped the  belt to review the tester’s bag, and the brand of X-ray machine.  According to TSA documentation, the technical team is also to  monitor incoming data from scenarios on a regular basis to address  any problems as they arise.

Inspection now uses guidance to ensure consistency in analysis and  reporting. This includes requirements for reviewing all test data and  applying rules about which data should be excluded. Inspection also  developed guidance to specify the types of statistical analyses that  may be used to draw conclusions about test results and how to report  on the results to ensure that its analysis of test results is appropriate  and transparent. For example, Inspection guidance identifies what  technical information should be included in the report to help readers  interpret Inspection’s conclusions that are based on statistical analysis  of results. We reviewed the two full reports that Inspection issued  using this new guidance and found that Inspection generally followed  the guidance for using statistical analysis and reporting final results in  these reports.

section_title: Security Operations Faces Challenges with the Quality of Its Covert Test Information and Its Quality Assurance Process
subsections: section_title: Security Operations Faces Challenges with the Quality of Airport Test Results
paragraphs: As previously discussed, the primary method by which Security  Operations tries to ensure that quality covert test results are generated at  airports is by having HET and FET testers conduct the same test  scenarios at airports, and then comparing detection rates identified by the  two teams. Security Operations program managers explained that this  method presupposes that test results collected by HET and FET  (following Security Operations’ overarching guidance for conducting tests  and using the same test scenarios) should produce similar detection rates  at the national level. Security Operations program managers further  explained that, because HET testers are unaffiliated with the airports they  test, they can more easily maintain test covertness. According to program  managers, this aspect of HET testing, along with additional training HET  testers receive in conducting covert tests, gives them greater assurance  that HET tests accurately reflect screener performance at airports.  Therefore, program managers generally consider large disparities  between HET and FET detection rates to indicate problems with the  quality of local airport covert test results.

According to our analysis of Security Operations national covert test data  for fiscal years 2017 and 2018, checked baggage tests consistently met  the Security Operations criterion for quality test results, but checkpoint  tests did not. In fiscal year 2018, TSA included a new criterion for quality  test results for Regional Director and FSD annual performance  evaluations. The criterion requires that HET and FET covert test detection  rates at airports under their supervision be within a designated  percentage point difference for the three types of tests (checkpoint in- property, checkpoint on-person, and checked baggage).

According to our analysis of Security Operations national covert test data  for fiscal year 2017 and the first half of fiscal year 2018, checked baggage  tests consistently met the criterion for quality test results, however,  checkpoint on-person and in-property tests did not. Specifically, we  calculated HET and FET detection rates for the three kinds of Security  Operations tests (checkpoint on-person, checkpoint in-property, and  checked baggage tests) for three 6-month periods from fiscal year 2017  through the first half of fiscal year 2018. We found that, for each 6-month  period, HET detection rates for checkpoint tests were lower than FET  detection rates, and the differences exceeded TSA’s established criterion  for quality test information. Security Operations officials acknowledged  the differences between HET and FET rates, but noted that the  differences generally decreased from the last 6-month cycle of testing for  fiscal year 2017 through the first 6-month cycle of 2018, and program  managers are working to address them further. Nevertheless, our  analysis showed that for the first half of fiscal year 2018 (the most recent  cycle’s data available for our analysis) differences between HET and FET  test detection rates for checkpoint on-person and checkpoint in-property  remained greater than Security Operations’ criterion for quality test  information.

In our observations of FET tests, we identified practices in local airport  testing that impact the covertness of tests, and thus may contribute to  differences between HET and FET detection rates. First, in our  observations of local airport FET tests in which TSOs correctly identified  the threat items, at one airport the TSA airport official in charge of FET  testing was present at the checkpoint, and his presence may have  provided advance notice to the TSOs that testing was in progress.  Further, we learned from airport testing officials that having the FET test  coordinator present at the checkpoint was a routine practice when testing  was in progress. At another airport visit, one TSO told us that TSOs often  know a FET test is in progress because TSA airport officials use the  same test bag to conceal threat items across all tests performed at the  airport. According to TSA documentation, potential lapses in the  covertness of covert tests, similar to those we observed and were told  about, can make TSOs aware that they are being tested and lead to  results on tests that overstate actual TSO performance.

In addition, we found that the level of potential variability in how TSA  airport officials build threat items and test bags for FET tests may affect  the quality of the test results used for comparison purposes. Security  Operations requires that FET personnel build the threat items, such as  explosive devices, that are used for scenarios according to specifications  included within TSA headquarters-disseminated scenarios. These  scenarios provide a description of the test scenario, a list of materials  needed for the threat item, assembly instructions, and directions on how  to conceal the threat item within checked or carry-on baggage. TSA  provides standard kits to local airports that contain some of the materials  FET teams need to build threat items (e.g., an explosive simulant), but  TSA staff at the airport must independently procure a number of items    needed for each scenario. Given that approximately 80 different teams  of FET testers use non-standardized items to build and conceal threat  items for tests, the test bags used by teams of FET testers vary to a  certain extent across test programs nationwide. According to TSA  officials, variations in the construction of test bags (including the  simulated explosive devices and test bag assembly) can affect how easy  or difficult it is to detect a threat item.

The program manager for the HET-FET testing program agreed there is a  need for greater assurance of the quality of covert test results, but stated  that Security Operations has not taken action on this issue due to  resource constraints. However, quality assurance is critical to ensure that  the resources TSA has invested in covert testing will yield valid and  usable information. Moreover, given its resource constraints, Security  Operations’ actions to improve local airport test results could encompass  less resource-intensive undertakings, such as providing more  standardized items for FET tests or improving guidance to address issues  that impact the covertness and consistency of tests.

Standards for Internal Control in the Federal Government states that  management should use quality information to achieve an entity’s  objectives, and that reliable internal sources should provide data that are  reasonably free from error and bias and faithfully represent what they     purport to represent. By assessing its current FET testing processes— including factors that may compromise the covertness and consistency of  tests—Security Operations could identify opportunities to improve the  quality of its testing. Further, making changes to its testing process based  on its assessment of the current FET testing process could help improve  the quality of test results. This, in turn, would better position those who  use these results (including agency leadership and TSA airport officials)  to reliably identify and address vulnerabilities based on TSO  performance.

In addition, we found that issues we identified with the quality of FET test  results also affect Security Operations’ reporting to external stakeholders.  As previously discussed, officials internal and external to TSA use  Security Operations test results to assess the effectiveness of TSO  performance. Currently, Security Operations reports quarterly FET  detection rates as a performance measure to the Office of Management  and Budget. The measure identifies the percent of time that TSOs  correctly detect threat items at the checkpoint (concealed in carry-on  baggage and on the passenger’s body) and within checked baggage.  However, as previously discussed, we found that airport testers were not  generating quality covert test information on checkpoint screening  because their FET detection rates were higher than the HET rates used  for comparison, and the difference between the rates exceeded the  criterion TSA established for quality covert test information. TSA  management officials acknowledged that the agency needs to use more  reliable covert test results for measures reported to the Office of  Management and Budget. In October 2018, TSA notified the Office of  Management Budget that it is in the process of assessing the quality of  covert test results it uses to report on TSO performance, and expects to  develop new measures by fiscal year 2020.

section_title: Security Operations’ Testers Face Challenges Identifying the Root Cause of Some Test Failures
paragraphs: In addition to issues with the overall quality of airport test results, we  found that Security Operations faced challenges with the quality of  information it collected on the root cause of tests failures. For each test  failure, HET and FET testers are to use the TPF tool to identify and  record the factor, or root cause, leading to a covert test failure. The TPF  tool groups test failure factors into three main categories—(1) failures  characterized by the screener’s lack of knowing what is required to  effectively accomplish a task or job (a knowledge deficiency); (2) failures  caused by incorrectly performing a procedure (a skill deficiency); or (3)  failures due to the TSO not assigning the correct level of importance to  performing a specific screening procedure (a value deficiency).

Although Security Operations has provided some guidance on when to  apply a particular factor as a root cause for a covert test failure, this  guidance may not be adequate and some testers may not be selecting  factors appropriately as a root cause. In our analysis of the factors  assigned by both Security Operations HET and FET testers for all covert  test failures in fiscal year 2017, we found that testers assigned one factor  more than the other two. To assist HET and FET testers in conducting  root cause analyses for test failures, Security Operations provides  definitions of the three root causes (knowledge, skills, and value). It also  requires that all testers (HET or FET) complete three online exercises for  using the TPF tool to record results, but the exercises do not provide  additional guidance on how to appropriately select root causes. In  addition, Security Operations provides in-person training to all HET  testers that includes a practice case on selecting from among the factors,  and the training course material indicates that the process can be  subjective.

In our observation of HET tests, we observed numerous failures in which  HET testers had to assign a root cause. In a majority of these failures, the  tester attributed the same factor as the root cause. HET testers who  completed the root cause analyses for these failures all told us they  assigned this particular factor by default, once they ruled out the other  two causes. Our observations were consistent with a 2017 independent  evaluation of the TPF tool performed by the DHS Science and  Technology Directorate. Among other things, subject matter experts  conducting the 2017 evaluation found that testers they spoke with were  not clear on the meaning of the three root causes, and the evaluation  recommended that Security Operations provide better guidance to testers  on how to select the root cause of a test failure.

Security Operations’ program managers concurred with the DHS Science  and Technology Directorate’s recommendation that testers need better  guidance on how to select among the factors as the root cause for test  failures. They also stated they are working on guidance to assist testers  in selecting the appropriate root cause for failures. However, in  September 2018, program managers told us they had suspended these  efforts to address the recommendation as a result of TSA efforts to  transfer program operations to Inspection and in anticipation of broader  changes to the Security Operations testing program. Inspection officials,  who will assume responsibility for HET and FET testing once the transfer  of the program to Inspection is complete, stated that they were unsure  what changes they would make to Security Operations’ legacy testing  process with respect to HET and FET tests at local airports, but stated  both types of testing will continue to use their respective legacy testing  processes in fiscal year 2019 until final decisions are made.

Standards for Internal Control in the Federal Government states that  management should use quality information to achieve an entity’s  objectives, and that reliable internal sources should provide data that are  reasonably free from error and bias and faithfully represent what they  purport to represent. As long as Security Operations’ legacy testing  process is in use, testers will continue to inconsistently and potentially  incorrectly identify the root cause for test failures, and in doing so, will  diminish the usefulness of root cause information for addressing TSO  performance problems. Reviewing existing guidance and training and  providing, where appropriate, additional clarification on applying the  factors as a root cause would allow TSA to collect more reliable  information on the factors leading to test failures. This, in turn, would  better position those who use this information (including agency  leadership and TSA airport officials) to address root causes of screener  failures at individual airports and across the entire system.

section_title: Security Operations Has Not Documented Its Methodology for HET Testing
paragraphs: Security Operations has not fully documented its methodology for using  HET testing as a quality assurance process for FET test results. While  Security Operations has documented some aspects of the HET test  process, such as training for HET testers on how to conduct tests and  post-test reviews with TSOs, we found that Security Operations has not  documented its methodology for using HET tests to ensure the quality of  FET test results in either its program guidance or other internal  documentation. For example, Security Operations has no documentation  on how program managers should select airports (e.g., by airport  category) and scenarios for HET testing, as well as how they should  analyze, compare, and report on HET test results against FET test  results.

Security Operations officials described some aspects of how they  calculate HET and FET test detection rates for comparison purposes, but  they did not have a documented methodology for this quality assurance  process. For example, Security Operations officials said that they only  use data from the largest airports that receive both HET and FET tests  (approximately 120 of the about 440 commercial airports) for comparison  purposes. Security Operations officials also explained they exclude all  HET and FET tests involving enhanced screening from the rates used for  comparison purposes because enhanced screening involves a more  detailed inspection of the subject that tends to result in the screeners  identifying threat items at a higher rate. In addition to these explanations,  program managers provided a document explaining Security Operations’  rationale for selecting each of the HET test scenarios used for the last  half of fiscal year 2017. While these explanations and the accompanying  documentation helped clarify aspects of Security Operations’ process,  Security Operations has not developed a policy that provides a  comprehensive description (and therefore understanding) of the quality  assurance process that its program managers are to use for program  planning purposes. Such a policy would describe Security Operations’  approach to selecting HET test scenarios used for ongoing covert testing,  how it calculates and compares test results, and how it reports and uses  the results. Security Operations program managers agreed that more  transparent information regarding the use of HET test results to assess  FET test results would be beneficial, but, given that the program was  established in late 2016, they acknowledged that they have not had time  to document this process.

Standards for Internal Control in the Federal Government states that all  transactions and other significant events need to be clearly documented,  and this documentation should be readily available for examination. The  documentation should appear in management directives, administrative  policies, or operating manuals. By fully describing its methodology for  comparing the results of HET testing with FET test results as a quality  assurance process within its program guidance, Security Operations can  better ensure that all aspects of this process are clear and available for  assessment and validation by third party users of HET and FET test  information, such as TSA senior leadership officials. Doing so can also  ensure that future program managers for the HET-FET test program can  continue to use this quality assurance method appropriately by following  the guidance.

section_title: TSA Uses Covert Test Results to Help Address Vulnerabilities, but Has Made Limited Efforts to Implement Mitigation Activities, Analyze Test Results, and Disseminate Beneficial Practices
subsections: section_title: Inspection’s Test Results Inform an Agency-Wide Process Intended to Mitigate Vulnerabilities, but This Process Has Not Yet Resolved Any Identified Vulnerabilities
paragraphs: Inspection submits its covert test findings that it determines to be security  vulnerabilities to TSA’s Security Vulnerability Management Process. TSA  established this agency-wide process in 2015 to review and address any  systemic vulnerability facing TSA (including those related to checkpoint  and checked baggage screening). However, it is unclear if  vulnerabilities reviewed through this process are being addressed in a  timely manner because the process lacks clear timeframes and  milestones for mitigation steps, as well as an established method for  monitoring the achievement of such timeframes and milestones.

In 2015, before establishing the Security Vulnerability Management  Process, TSA conducted a review of then-existing processes for  evaluating and managing identified vulnerabilities, and found that they  were not centralized and did not ensure the level of visibility and  accountability needed to adequately mitigate and resolve (or close) the  vulnerabilities. Consequently, TSA determined that its processes for  tracking and managing the closure of identified security vulnerabilities  represented an organizational deficiency that should be addressed. In  addition, Inspection officials stated that, under the prior processes, they  lacked complete knowledge of all agency resources that could be  leveraged to develop mitigation strategies, as well as the necessary  authority to compel offices to share these resources, which made it  difficult to ensure identified vulnerabilities were addressed. As a result,  TSA created the Security Vulnerability Management Process to better  ensure the cooperation of various program offices within TSA that had the  expertise needed to address vulnerabilities identified by Inspection or  other offices within TSA. This process is intended to centralize agency  efforts to mitigate vulnerabilities by ensuring that they receive agency- wide visibility and are evaluated, resourced, and managed by appropriate  TSA program offices until fully addressed.

TSA’s Strategy, Policy Coordination, and Innovation office is responsible  for managing and overseeing the Security Vulnerability Management  Process, as well as enforcing deadlines for vulnerability mitigation. The  Strategy, Policy Coordination, and Innovation office submits  vulnerabilities for review by one of two groups of TSA stakeholders—the  Executive Risk Steering Committee or the Risk Assessment Integrated  Project Team. These two groups are responsible for identifying all TSA  program offices affected by the vulnerability in question and working with  those program offices to determine whether and how vulnerabilities can  be mitigated and formally closed (see fig. 2). According to TSA Strategy,  Policy Coordination, and Innovation office officials, to close a given  vulnerability, one of the two groups will assess whether the risk posed by  the vulnerability aligns to the identified amount of risk that TSA is willing  to accept. TSA officials told us that the agency is risk averse to any  vulnerability that could cause catastrophic consequences, such as the  loss of an airplane.

The Strategy, Policy Coordination, and Innovation office has responsibility  for enforcing deadlines for mitigating identified vulnerabilities, but our  review of TSA documentation found that the office does not establish  timeframes and milestones to ensure measured progress toward  mitigation of those vulnerabilities. Moreover, we found that although the  Security Vulnerability Management Process charter establishes a broad  framework for developing and implementing mitigation strategies, it does  not establish a method for how the Strategy, Policy Coordination, and  Innovation office is to monitor mitigation activities to ensure that TSA  program offices are meeting identified timeframes and milestones, such  as by identifying a person or entity responsible for escalating cases when  these requirements are not being met.

Specifically, we found that Inspection has submitted nine vulnerabilities  for consideration. With one exception, as of September 2018, none of  the vulnerabilities have been formally closed as a result of mitigation  steps taken via the vulnerability management process. Under the  process, a vulnerability owner has responsibility for developing and  leading mitigation efforts for a specific vulnerability. TSA closed one of  the nine vulnerabilities 2 years after submission to this process because  the relevant program office made policy changes that addressed  Inspection’s interim findings. The remaining vulnerabilities have been in  progress from 4 months to 2.5 years. Of these eight vulnerabilities, five  have had TSA offices assigned as vulnerability owners, and three of  these five have mitigation efforts in progress. The three remaining open  vulnerabilities that did not yet have vulnerability owners assigned at the  time of our review had been waiting for vulnerability owners for a period of  4, 5, and 7 months, respectively; however, TSA officials told us that these  three open vulnerabilities had owners assigned in September 2018.

TSA officials told us that timeframes for vulnerability mitigation can vary  due to the number of stakeholders required to address the situation. They  also explained that the complexity of certain threats affect the timeliness  of final mitigation solutions (e.g., those requiring technology solutions can  involve multiple TSA offices); and before such solutions are developed,  Inspection works with program offices to help them develop interim  mitigation procedures. Additionally, they cited factors beyond TSA’s  control that can delay mitigation efforts, such as changes to agency  leadership or in staff within a particular office. For example, mitigation has  been delayed for one of the vulnerabilities under review for over 2 years,  due to changes in agency leadership in 2016, among other things. In  another example, TSA officials told us that mitigation for a vulnerability  under review had been delayed for over two years due to personnel  changes within the office tasked with developing and leading mitigation  efforts. Inspection officials told us that while officials are working on  mitigation solutions for identified vulnerabilities, Inspection will assist TSA  program offices with implementing interim mitigation procedures before  formal mitigation plans are developed. For example, Inspection officials  stated that they worked with Security Operations to provide interim  guidance to TSA airport officials to address an identified vulnerability that  involved Transportation Security Specialists for Explosives using  screening equipment incorrectly to clear passengers through the  checkpoint.

Although TSA has implemented interim mitigation steps for some  vulnerabilities while its program offices develop long-term solutions, in  some cases Inspection’s findings represent system-wide vulnerabilities to  commercial aviation that could result in potentially serious consequences  for TSA and the traveling public. For this reason, it is important that TSA  make timely progress on formal mitigation solutions. Moreover, tracking  progress for a given vulnerability against timeframes and milestones  would not necessarily preclude TSA program managers from accounting  for complex mitigation efforts. Program managers could, for example,  establish longer timeframes at a mitigation effort’s onset and adjust these  as needed, should challenges arise.

The Standard for Program Management states that the governance of  programs includes establishing minimum acceptable criteria for success  and the standards by which they are measured and communicated to  achieve desired outcomes. Additionally, programs should include the  concept of time and incorporate schedules through which specific  milestone achievements are measured to ensure that appropriate  progress is made toward achieving a defined set of outcomes. In TSA’s  case, this would mean the mitigation of identified vulnerabilities. The  Standard for Program Management further states that program  governance plans are to describe the systems and methods to be used to  monitor a given program, and the responsibilities of specific roles for  ensuring the timely and effective use of those systems and methods.

TSA officials agreed that their vulnerability management process lacks a  clear set of deadlines for the timely completion of mitigation steps, as well  as a method for monitoring completion of these steps to ensure  vulnerabilities are closed. By establishing timeframes and milestones for  vulnerability mitigation, TSA would better ensure that progress toward  addressing vulnerabilities continues, despite internal challenges, such as  personnel changes, or external factors. In addition, by establishing the  methods by which TSA’s Strategy, Policy Coordination, and Innovation  office will monitor milestones for completion, and the steps it will take  when mitigation is not progressing as planned, TSA will be better  positioned to ensure that the agency is making measured progress  toward addressing the vulnerabilities managed through this process.

section_title: Security Operations Uses Test Data for Feedback and Reporting to Airports and Others, but Does Not Analyze National Data to Identify Potential Vulnerabilities in Screener Performance
subsections: section_title: Security Operations Monitors Covert Test Data to Identify Potential Vulnerabilities
paragraphs: Security Operations program managers said that they continuously  monitor covert test results to identify potential vulnerabilities and to  assess progress at airports in addressing vulnerabilities identified through  covert tests. Security Operations primarily monitors TSO performance by  reviewing information within its TPF tool. Specifically, program officials  said that they monitor the database each month to identify gaps between  HET and FET detection rates at an individual airport and regional level.  Security Operations officials said that they will alert TSA officials at  airports if they detect anomalies or large disparities between their HET  and FET test rates, and suggest strategies for conducting tests. While  reviewing the data, Security Operations officials told us they may also  identify specific test scenarios that TSOs are experiencing difficulties with,  and sometimes develop strategies to improve performance. For example,  officials said that when TSOs demonstrated difficulty with a scenario  involving colorimetric testing, Security Operations developed a pamphlet  for TSOs to clarify those procedures.

Security Operations’ monitoring has also resulted in changes to  processes and procedures. For example, according to TSA  documentation, in early 2016 Security Operations officials conducted an  ad hoc analysis of relevant covert test data. This analysis led to the  implementation of Enhanced Accessible Property Screening procedures  for personal property screened at airport checkpoints. According to TSA  documentation, these new procedures are intended to help TSA officers  obtain a clearer X-ray image to enhance screening effectiveness. Among  other things, they involve advising passengers to remove organic  materials from carry-on bags for X-ray screening, requiring that  electronics larger than a cell phone be removed from carry-on bags and  placed in bins for X-ray screening, and more targeted property search  protocols.

In addition to periodic monitoring of test data within the TPF tool’s  database, Security Operations officials also told us they monitor Threat  Detection Improvement Plans, which are based on recommended actions  stemming from each airport’s covert testing results. TSA officials told us  that these plans can include test-specific action plans and high-level  improvement strategies. Security Operations now monitors airport  progress against these plans in order to ensure that airports are taking  the necessary actions to improve TSO performance deficiencies identified  in covert testing.

section_title: Security Operations Uses Test Data to Provide Feedback and Reporting to Airports and Other Stakeholders
paragraphs: Security Operations officials told us they use covert test results as the  basis for feedback and periodic reporting on TSO performance and the  quality of covert test programs or results to headquarters, regional, and  local TSA officials and other stakeholders. According to Security  Operations officials, this feedback and reporting includes the following.

HET reports and feedback: Security Operations directly  communicates with TSA officials at airports on HET test performance.   For example, in our observations of HET tests at airports, testers  conducted an equal number of post-test reviews, during which they  reviewed with TSOs and their supervisors the intent and results of the  HET tests, reinforced actions resulting in test successes, and  reviewed the correct procedures for any failures. In addition to post- test reviews, at the conclusion of each HET test at an airport, Security  Operations program managers provide TSA management at the  airport a report compiling the results of the recent HET test and  statistics on the quality of the covert test program at the airport.  According to TSA documentation, these reports include a comparison  of local FET test results against the results of HET tests that were  conducted during that visit.

TPF Report: On a monthly basis, according to TSA documentation,  Security Operations also provides a classified spreadsheet report to  FSDs that contains a high-level analysis of HET and FET covert test  data collected for the fiscal year to date, as well as a copy of the most  current test results in the TPF tool’s database. Security Operations  program managers stated that allowing airports access to the entire  database allows FSDs to compare their airport’s performance against  counterparts in other regions and address any areas in which they are  lagging. In our interviews with FSDs, we found that officials from all of  the airports we spoke with used the TPF data to help manage TSOs.  For example five FSDs told us they download the raw test data into  local systems for use in their local processes for monitoring TSO  performance.

Classified monthly conference calls: According to TSA officials,  Security Operations hosts monthly classified conference calls with  local and regional TSA officials to discuss issues related to covert  testing. Security Operations officials told us these discussions  typically include the results of specific covert test rounds, methods for  using covert tests results, and FSDs’ beneficial practices for carrying  out covert testing at their airports.

Reporting to senior leadership and other stakeholders: Security  Operations officials said they continue to use covert test results for  monthly briefings to FSDs and TSA senior leadership. According to  TSA documentation, these briefings include high-level analysis of  regional covert test performance, as well as overall comparisons of  detection rates for on-person, in-property, and checked baggage tests  against the national averages. As previously discussed, TSA also  uses FET test results as the basis of a performance measure reported  quarterly to the Office of Management and Budget.

FSDs we spoke with told us they find the feedback and reporting they  receive from Security Operations program managers to be helpful. In  particular, all 10 FSDs we spoke with told us they find both the HET test  reports and accessibility to TPF data in the monthly spreadsheet report to  be beneficial and useful. FSDs also noted that the HET reports help  inform their assessments on individual and airport workforce performance  and efforts to improve their airport’s screening operations overall.

section_title: Security Operations Does Not Conduct and Share a Comprehensive Analysis of National Covert Test Data to Identify Potential Vulnerabilities
paragraphs: While Security Operations program officials perform some high-level  analysis of TPF data for periodic reporting, they do not analyze all  Security Operations-collected covert test data to identify potential national  trends in screener performance that could constitute system-wide  vulnerabilities. For example, according to officials and TSA  documentation, Security Operations officials use FET and HET covert test  data to describe broad trends in screening performance in monthly  briefings to TSA management. However, the briefings do not include a  breakdown of the different screening tasks and processes that may be  most often associated with TSO failures nationally. In addition, although  the TPF tool’s database contains information on the task, process, and  factors associated with each TSO test failure, Security Operations does  not typically include a comprehensive analysis of this information within  the monthly covert test reports it provides to TSA leadership at airports.  For example, based on our review of Security Operations’ monthly TPF  reports, they identify which processes have resulted in the most failures,  but do not identify which factors—knowledge, skill, or value—were the  root cause of these failures. Moreover, none of this reporting reflects a  broader analysis to identify whether failures or causes were associated  with a certain size of airport or reflected across one or more regions.

Standards for Internal Control in the Federal Government states that an  agency should design its information systems to respond to the entity’s  objectives and risks. Furthermore, agencies may use information from  these systems to evaluate the agency’s performance in achieving key  objectives. As discussed previously, Security Operations officials have  performed similar types of analysis in the past with positive results. For  example, when TSA developed the Enhanced Accessible Property  Screening procedures in 2017, these actions were based (in part) on ad  hoc analysis Security Operations conducted with national covert test data.  At the time, Security Operations’ analysis showed that X-ray operators at  checkpoints had problems determining the threat nature of certain  categories of objects. This led to repeated failures in detection given the  time and cognitive load requirements for interpreting those types of X-ray  images. In response, TSA created or adjusted specific procedures based  on the analysis of root causes of testing failures and the results of piloting  new screening procedures at multiple sites to ensure effectiveness and  efficiency could be sustained.

Security Operations officials agreed that conducting a more  comprehensive, national-level analysis, and utilizing more of the covert  test data currently within the TPF tool’s database, would be useful in  identifying system-wide vulnerabilities that could inform efforts to improve  TSO performance. Security Operations officials told us that at present,  they do not have a standard process to comprehensively analyze and  report trends in TPF data across all airports. This is because the intent of  the current program has been to make test data available to TSA airport  and regional officials so they can identify factors affecting screener  performance and take actions to remediate and improve any deficiencies.  In addition, Security Operations officials cited a lack of resources  available to dedicate to this activity, given that headquarters officials have  been more focused on revising and improving their current covert test  program. However, Security Operations’ TPF tool and database has  enabled it to document and communicate detailed information on TSO  performance, such as the different screening tasks (e.g., advanced  imaging technology operation) and processes (e.g., resolving advanced   imaging technology anomalies) where screeners encounter difficulties.  Given the breadth of testing conducted and information collected, more  comprehensive analysis of TPF data could help TSA identify and  communicate important potential trends in the vulnerabilities that TSOs  face across all airports.

A comprehensive analysis of TSO performance at the national level  beyond calculation of overall detection rates would provide Security  Operations greater knowledge about the reasons for, and factors  associated with, system-wide vulnerabilities due to TSO performance of  checkpoint and checked baggage screening, which would better position  TSA to address these security gaps. For example, having this information  could allow Security Operations to provide more focused training and  testing for these functions at the airport level. The information could also  position TSA to allocate resources for high-priority issues across all  airports.

section_title: TSA Airport Officials Have Developed Beneficial Practices for Conducting Covert Tests and Using Test Data, but Security Operations Does Not Systematically Document and Disseminate This Information
paragraphs: TSA officials at individual airports reported using different tools,  techniques, and processes for conducting covert tests and using test  data, but Security Operations does not document and disseminate this  information. In our discussions with 10 FSDs and their management  teams, officials identified a variety of tools, processes, and methods that  were developed based on their experiences with covert tests and the  resulting actions they took to utilize test data to improve TSO  performance. Specifically, 5 of the 10 FSDs we spoke with said their  teams developed some type of customized internal databases to  aggregate all of their airports’ covert test results, other performance- related data, and any additional Inspection information. FSDs and their  staff said such a tool helped present a holistic picture of TSO  performance for training and development purposes. Likewise, 5 of the 10  FSDs we spoke with said that they use test results to develop TSO  performance baselines and training plans with requirements that exceed  TSA’s minimum standards for remediation. Additionally, 5 of 10 FSDs  stated that they now include supervisory TSOs and/or TSA leadership  officials at airports in remediation discussions with individual TSOs after  covert tests take place to provide leadership officials with experience on  how best to coach and develop staff.

TSA officials we spoke with at airports and at the regional level said that  individual airports are often a source for innovation with respect to  executing covert tests and using test results, which has at times led to  pilot efforts that were adopted at other airports either regionally or  nationally. For example, officials from one TSA region told us that they  were the first to develop and use performance scorecards (which  incorporate covert test results) as an additional tool for improving  screener performance. These scorecards were eventually adopted  nationwide. Most of the FSDs we spoke with said they communicate  with their counterparts at other airports to discuss covert test practices  and beneficial methods for using test results at their respective airports.  For example, officials from one airport we spoke with reported traveling to  an airport in a different region to learn more about the team’s TSO  remediation process, which involved using the results of covert testing,  Threat Image Projections, and other assessments to create tailored  corrective action plans for TSOs. The officials said that this process was  an improvement from the one they used previously because it  incorporated a greater variety of remediation actions, such as training  courses or shadowing opportunities.

As discussed previously, Security Operations officials communicate with  TSA officials at airports on their covert test programs during a monthly  classified call with all FSDs and their teams. This allows Security  Operations program managers to provide FSDs with an update on results  from recent HET and FET tests, among other things. Security Operations  program managers stated that during these calls, they encourage TSA  officials not only to discuss particular issues or challenges they have  faced with respect to covert testing at their airports, but also to highlight  beneficial practices for conducting tests and using test results to improve  TSO performance that they and their teams have self-identified and  implemented. Therefore, these calls also serve as a forum for FSDs to  discuss successful techniques for running covert tests and using test  results. In our discussions with 10 FSDs, 8 out of 10 told us they have  independently adopted beneficial practices used by other airports.

Security Operations program managers are privy to beneficial practices  discussed during their teleconferences with local and regional TSA  officials, but they told us that they do not regularly document or  disseminate this information to TSA officials at airports. Security  Operations program managers explained that the call itself is adequate  for TSA airport officials to share information, and that local or regional  officials can follow up with one another if they want to discuss them  further. However, while a monthly conference call may be helpful for  informal sharing of practices, it does not capture the breadth of methods  or practices used by some TSA airport officials. Moreover, according to  headquarters officials, while conference calls provide an opportunity for  FSDs to discuss beneficial practices, sharing is ad hoc and the level of  detail provided about methods and practices can vary. Systematically  documenting and disseminating these practices would provide TSA  officials at airports more accurate and complete information about  beneficial practices in use at airports nationwide, so that they could be  more readily implemented at other airports.

The National Infrastructure Protection Plan states that in order to ensure  that situational awareness capabilities keep pace with a dynamic and  evolving risk environment, officials should improve practices for sharing  information and applying the knowledge gained through changes in  policy, process, and culture based on shared understanding of efforts to  improve security and resilience. This plan also states that documenting  and building upon beneficial practices is a key part of information sharing  within a critical infrastructure risk management framework. Our interviews  with FSDs revealed an array of tools, techniques, and processes for  covert testing that TSA officials at airports developed to address local and  regional needs. A process to systematically document and disseminate  more accurate and complete information on these tools, techniques, and  processes that captures the breadth of methods or practices used by  some TSA airport officials could help TSA conduct better covert tests and  more successfully use test results to improve TSO performance, as well  as inform revisions to TSA’s national covert test program.

section_title: Conclusions
paragraphs: Given the persistent threats to the aviation system, TSA must ensure that  its covert testing program operates as effectively as possible to identify  and address potential vulnerabilities in the checkpoint and checked  baggage screening systems across the nation’s airports. TSA has  strengthened the quality and rigor of its covert test programs since 2016,  but additional steps are needed to better ensure that TSA targets the  areas of highest risk in selecting attack scenarios for testing. Without  using a risk-informed approach to selecting screening activities to test,  TSA cannot ensure that it is targeting those aspects of TSA screening  that pose the greatest known risks. In addition, without documenting its  rationales behind how and why certain scenarios are selected for covert  testing, TSA cannot demonstrate how its selections reflect identified risks  in the aviation environment.

New processes for covert testing implemented by Security Operations  and Inspection have identified important vulnerabilities in checkpoint and  checked baggage screening for fiscal years 2016 and 2017. However,  these results can only be useful if they meet internal standards for quality  test results. While Inspection’s new process generally produced quality  test results on screening vulnerabilities, Security Operations continues to  face challenges with the quality of test results collected by TSA staff at  local airports. Without taking steps to ensure that Security Operations  collects more valid and usable information on vulnerabilities, including the  root cause of test failures, TSA will not be positioned to reliably identify  and address important security vulnerabilities. In addition, without  documenting its methodology for comparing the results of covert tests,  TSA cannot ensure that its quality assurance process is consistently  applied and transparent.

Once vulnerabilities have been identified through covert testing, it is  paramount that they are effectively and efficiently mitigated or addressed.  Establishing the Security Vulnerability Management Process was a good  step toward better tracking the vulnerabilities identified through covert  tests and deploying resources to mitigate them, but key identified  vulnerabilities have been stalled in the process and none have been  closed using this process. This has largely been caused by the absence  of timeframes and milestones for achieving mitigation and monitoring key  activities in the process. Unless TSA incorporates these aspects into its  vulnerability management guidance, it cannot ensure that it is effectively  addressing security vulnerabilities that could result in potentially serious  consequences for the traveling public. Additionally, while TSA shares  some covert test information with TSA officials at airports, more  comprehensive analysis of covert test information is needed to enhance  TSA’s knowledge about the reasons for, and the factors associated with,  TSO performance vulnerabilities that exist system-wide. Furthermore,  although TSA officials at individual airports informally share information  about beneficial practices they use to conduct covert tests and how they  use test information, without systematically documenting and  disseminating these practices, TSA cannot ensure that airport officials are  fully informed about the different tools, techniques, and processes used  by their colleagues.

section_title: Recommendations for Executive Action
paragraphs: We are making the following nine recommendations to TSA:  The Administrator of TSA should document its rationale for key decisions  related to its risk-informed approach for selecting covert test scenarios,  for both the Security Operations’ and the Inspection’s testing process.  (Recommendation 1)

The Administrator of TSA should incorporate a more risk-informed  approach into Security Operations’ process for selecting the covert test  scenarios that are used for tests conducted by TSA officials at airports.  (Recommendation 2)

The Administrator of TSA should assess the current covert testing  process used by TSA officials at airports—including factors that may  affect the covertness and consistency of the tests—to identify  opportunities to improve the quality of test data, and make changes as  appropriate.  (Recommendation 3)

The Administrator of TSA should assess Security Operations guidance for  applying root causes for test failures, and identify opportunities to clarify  how they should be applied. (Recommendation 4)

The Administrator of TSA should document the methodology for using the  results of covert testing conducted by headquarters staff as a quality  assurance process for covert testing conducted by TSA officials at  airports. (Recommendation 5)

The Administrator of TSA should establish timeframes and milestones for  key steps in its Security Vulnerability Management Process that are  appropriate for the level of effort required to mitigate identified  vulnerabilities. (Recommendation 6)

The Administrator of TSA should revise existing guidance for the Security  Vulnerability Management Process to establish procedures for monitoring  vulnerability owners’ progress against timeframes and milestones for  vulnerability mitigation, including a defined process for escalating cases  when milestones are not met. (Recommendation 7)

The Administrator of TSA should develop processes for conducting and  reporting to relevant stakeholders a comprehensive analysis of covert test  results collected by TSA headquarters officials and TSA officials at  airports to identify vulnerabilities in screener performance and common  root causes contributing to screener test passes and failures.  (Recommendation 8)

The Administrator of TSA should develop a standard process for  systematically documenting and disseminating to airport Federal Security  Directors beneficial practices for conducting covert tests and using test  results. (Recommendation 9)

section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DHS and TSA for review and  comment. DHS provided written comments which are reprinted in  appendix II. In its comments, DHS concurred with all 9 recommendations  and described actions planned to address them. TSA also provided  technical comments, which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of the Department of Homeland Security, and  other interested parties. In addition, the report is available at no charge on  the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-8777 or russellw@gao.gov. Contact points for our Office  of Congressional Relations and Public Affairs may be found on the last  page of this report. Key contributors to this report are listed in appendix  IV.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report addresses the Transportation Security Administration’s (TSA)  covert testing for checkpoint and checked baggage screening. More  specifically, the report (1) describes how TSA has changed its covert test  processes since 2016 and analyzes the extent to which these processes  are risk-informed; (2) analyzes the extent to which TSA covert tests for  fiscal years 2016 through March 2018 produced quality information; and  (3) analyzes the extent to which TSA has used the results of covert tests  to address any identified security vulnerabilities.

To understand how both the Security Operations and Inspection offices  changed their respective covert test processes since 2016, we reviewed  agency documentation, interviewed agency officials, and observed 22  Security Operations and 4 Inspection covert tests at 5 different airports. In  addition to Inspection testing, our observations included two types of  testing overseen by Security Operations—Headquarters Evaluation Team  (HET) testing and Field Evaluation Team (FET) testing. To gather  information on how covert tests are carried out in different airport  environments, we observed tests at four category X and one category I  airports. We selected airports for observations on the basis of airport  category and screener workforce (private vs. TSA-employed screeners).  For all observations, we were able to observe TSOs performing  checkpoint or checked baggage screening activities during tests.  Following all observations, we observed post-test reviews and, when  appropriate, interviewed TSA airport officials, including the Transportation  Security Officers (TSO) and private sector screeners (collectively referred  to as TSOs in this report) who were tested, about their experience with  these tests.

To determine the extent to which Security Operations and Inspection  testing is risk-informed, we reviewed program documentation and spoke  with agency officials. Specifically, we reviewed operational guidance and  test scenarios, which describe the overall intent of the test, the threat  item, and method of execution (e.g., an explosive device concealed in a  shoe carried through the checkpoint) to identify how program officials  incorporated the components of risk—threat, vulnerability, and  consequence—in their selection of threats and airports to test. We also  reviewed the TSA risk assessments that would have been available to  Inspection and Security Operations when planning which threats and  airports to test for fiscal year 2017, namely TSA’s 2016 Transportation  Sector Security Risk Assessment and TSA’s 2012 Current Airports Threat  Assessment. The 2016 Transportation Security Sector Risk Assessment  contained attack scenarios for the five transportation modes for which  TSA is responsible, including domestic and international commercial  aviation, as well as other mass transit systems, such highway and mass  transit. For our analysis, we used those scenarios relevant to our scope— domestic commercial checkpoint and checked baggage screening. We  compared the results of these assessments to the threat items and  locations that Security Operations selected for tests in fiscal year 2017  and Inspection selected for tests in fiscal years 2016 and 2017. We  evaluated each office’s process for making risk-informed decisions with  Department of Homeland Security (DHS) risk management policies,  which require that agencies use risk information and analysis to inform  decision making, and that risk management methodologies should be  transparent and properly documented.

To assess the quality of Security Operations data, we reviewed program  guidance and interviewed program officials to understand how Security  Operations uses HET test results to validate the quality of FET testing at  local airports. We also reviewed a 2016 validation study of Security  Operations’ test process conducted by the DHS Office of Science and  Technology, and spoke with subject matter experts who conducted the  study about their findings and recommendations related to improving the  quality of test information. We concluded the study’s findings were  reasonably sufficient to use as additional support for patterns we also  observed during site visits. We were also informed by our HET and FET  test observations, which included observations of 19 HET tests at 3  different airports, and 3 FET tests at 1 airport. We supplemented our  understanding of how airports conduct FET tests through semi-structured  telephone interviews with 10 different Federal Security Directors (FSD)  and their staff. To select FSDs for interviews, we identified the airports at  which TSA conducted more than the average number of HET covert tests  in fiscal year 2017. We focused on the number of HET (as opposed to  FET) tests because they are Security Operations’ quality assurance  method for airport covert test programs, and we wanted to ensure FSDs  had sufficient experience with these tests to provide us perspectives.  From this group, we identified the airports with the highest and lowest  pass rates for HET tests, and selected among these to reflect variation in  several factors, including airport category, difference between HET and  FET detection rates, and whether the airport had been tested by  Inspection in fiscal years 2016 and 2017.

Finally, to assess the quality of Security Operations’ testing, we  calculated detection rates for its two types of testing—Headquarters  Evaluations Team (HET) tests, in which Security Operations  headquarters staff travel to airports to conduct tests, and Field  Evaluations Team (FET) tests, which are conducted by staff at local  airports. We assessed FET test results against Security Operations’  criterion stating that differences in HET and FET detection rates must be  within a designated number of percentage points. We made these  comparisons analyzing complete test results for fiscal year 2017 and the  first 6 months of fiscal year 2018, over three 6-month periods in order to  identify trends. We used for our analysis the12,000 fiscal year 2017  Security Operations TPF records documenting the results of individual  covert tests, and an additional 3,600 records from fiscal year 2018. For  our analysis, we calculated HET and FET detection rates (i.e., number of  items successfully detected) for three screening paths: a checkpoint test  with the item concealed on the tester, a checkpoint test with the item  concealed in a carry-on bag, and a checked baggage test with the item  concealed in the checked bag. In calculating these detection rates, we  included only results for scenarios tested within the 18-month period that  had both HET and FET tests, and we excluded any test results for  scenarios involving enhanced screening. Also, in our calculation of the  FET detection rate, we included FET test results for all airports, including  those from smaller (category III and IV) airports, which HET teams  generally do not visit. We chose to include FET results from all airports  in our analysis because it better reflected the overall performance of  airports on covert tests. In addition to comparing Security Operations’  quality assurance process against the program’s criteria, we assessed it  against federal internal control criteria for documenting processes.

To assess the quality of Inspection testing, we reviewed program  guidance to identify testing requirements, methods, and limitations. We  also observed four different tests conducted at a Category X airport. In  addition, we reviewed Inspection guidance to identify and assess  requirements for analyzing and reporting covert test results, and reviewed  completed reports to identify the extent to which Inspection followed these  requirements.  We met with Inspection technical experts to discuss  Inspection processes for selecting a sample of airports for tests and for  analyzing and compiling covert test findings.

To assess the extent to which Inspection and Security Operations  address security vulnerabilities, we reviewed their efforts separately  because each office utilized a different approach. To assess Inspection’s  efforts, we focused on its use of the Security Vulnerability Management  Process, an agency-wide process that Inspection designated in 2016 as  the principal means by which it addresses its identified vulnerabilities.  To obtain a more complete understanding of the extent to which this  process has addressed Inspection vulnerabilities, we reviewed  documentation related to the process (such as its charter) and other  information pertaining to all vulnerabilities Inspection has submitted to the  process, including those that were unrelated to checkpoint and checked  baggage screening (e.g., cargo screening). We analyzed timeframes  associated with the vulnerabilities reviewed under the process and the  progress made toward closing nine Inspection-identified vulnerabilities.  We assessed the vulnerability management process against standards  for program management issued by the Project Management Institute, a  not-for-profit association that provides global standards for, among other  things, project and program management.

Given the focus of Security Operations’ testing on screener performance,  the vulnerabilities it identified involved TSO failures on tests of specific  procedures. To determine how Security Operations headquarters officials  address vulnerabilities involving screener performance, we reviewed  program documentation, including program guidance and periodic  reporting of results, and interviewed program managers. To understand  how the results of covert testing are used at the airport level to improve  TSO performance and address other identified vulnerabilities, we  conducted semi-structured interviews with 10 TSA FSDs stationed at  airports across the United States, and with three TSA Regional  Directors. We selected the latter based on whether the Regional  Director had under his or her direction at least 1 of 10 FSDs we selected  for interviews, and to reflect variety in geographic location. We assessed  Security Operations’ and TSA officials at airports’ efforts to use covert test  results to address vulnerabilities against federal internal control standards  and criteria within the National Infrastructure Protection Plan.

This is the public version of a classified report that we issued on January  10, 2019. The classified report included an objective related to  identifying the results of covert testing for fiscal years 2016 and 2017 and  assessing the quality of this test information. DHS deemed covert testing  results (including detection rates and identified vulnerabilities) to be  classified information, which must be protected from loss, compromise, or  inadvertent disclosure. Consequently, this report omits part of an  objective identifying the results of covert testing. DHS also deemed some  of information in our January report to be sensitive security information.  Therefore, this report omits information describing TSA screening  procedures, the results of agency risk assessments, and airport-level  covert test results.

The performance audit upon which this report is based was conducted  from September 2017 to January 2019 in accordance with generally  accepted government auditing standards. Those standards require that  we plan and perform the audit to obtain sufficient and appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  from this work provides a reasonable basis for our findings and  conclusions based on our audit objectives. We worked with DHS from  February 2019 through April 2019 to prepare this unclassified, non- sensitive version of the original classified report for public release. This  public version was also prepared in accordance with these standards.

section_title: Appendix III: GAO Contact and Staff Acknowledgments
paragraphs: William Russell (202) 512-8777 or RussellW@gao.gov.
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Ellen Wolfe (Assistant Director),  Mona Nichols Blake (Analyst in Charge), James Ashley, Chuck Bausell,  Jason Blake, Michele Fejfar, Eric Hauswirth, Susan Hsu, Tom Lombardi,  Minette Richardson, and Nina Thomas-Diggs made significant  contributions to this report.

Fast facts: To test security screening at U.S. airports, TSA regularly tries to sneak guns and simulated bombs through checkpoints or in checked baggage. TSA changed its testing practices to better identify and address screening vulnerabilities.

We observed 26 covert tests and reviewed the test program and how results are used. We found that TSA’s ability to run covert tests has improved, but a new process intended to address vulnerabilities found in testing hasn't fully worked.

We made 9 recommendations, including that TSA establish timeframes for addressing the vulnerabilities it discovers.

Question-summary annotations: questions: question: What is GAO's finding about the two offices' covert tests?
answer: Both Inspection and Security Operations have implemented processes to ensure that their covert tests produce quality results. However, GAO found that only Inspection has established a new process that has resulted in quality test results.
child_questions: question: What is specifically different between the two offices' test results?
answer: Specifically, for the two reports Inspection completed for testing conducted in fiscal years 2016 and 2017 using its new process, GAO found that the results were generally consistent with quality analysis and reporting practices. On the other hand, Security Operations has not been able to ensure the quality of its covert test results, and GAO identified a number of factors that could be compromising the quality of these results.

question: How can TSA improve the reliability of its test results in addressing vulnerabilities?
answer: Unless TSA assesses the current practices used at airports to conduct tests, and identifies the factors that may be impacting the quality of covert testing conducted by TSA officials at airports, it will have limited assurance about the reliability of the test results it is using to address vulnerabilities.

questions: question: How is TSA's Security Vulnerability Management Process working?
answer: In 2015, TSA established the Security Vulnerability Management Process to leverage agency-wide resources to address systemic vulnerabilities; however, this process has not yet resolved any identified security vulnerabilities.
child_questions: question: How were the identified security vulnerabilities addressed through this process from 2015 to 2018?
answer: Since 2015, Inspection officials submitted nine security vulnerabilities identified through covert tests for mitigation, and as of September 2018, none had been formally resolved through this process.

question: What did GAO find about TSA officials in resolving the vulnerabilities?
answer: GAO found that in some cases, it took TSA officials overseeing the process up to 7 months to assign an office responsible to begin mitigation efforts.
child_questions: question: Why is this problem happening?
answer: In part, this is because TSA has not established time frames and milestones for this process or established procedures to ensure milestones are met, in accordance with best practices for program management.
child_questions: question: What could be the negative result of this loophole?
answer: Without doing so, TSA cannot ensure efficient and effective progress in addressing security vulnerabilities.

questions: question: How does TSA identify potential vulnerabilities at U.S. airports?
answer: TSA uses covert testing to identify potential vulnerabilities in checkpoint and checked baggage screening systems at U.S. airports.

question: What were the findings on deficiencies in TSA's covert testing process?
answer: In 2015, TSA identified deficiencies in its covert testing process, and in 2017, the Department of Homeland Security Office of Inspector General's covert testing identified deficiencies in screener performance.
child_questions: question: What has TSA done in response?
answer: Since these findings, TSA has taken steps intended to improve its covert test processes and to use test results to better address vulnerabilities.