Transportation Security: TSA Has Developed a Risk-Based Covert Testing Program, but Could Better Mitigate Aviation Security Vulnerabilities Identified Through Covert Tests
Released: 2008-08-14T00:00:00

Letter

Background

Congress and the Administration have advocated the use of covert or red team testing in all modes of transportation. Following the terrorist attacks on September 11, 2001, on November 19, 2001, the President signed ATSA into law, with the primary goal of strengthening the security of the nation’s commercial aviation system. ATSA created TSA within the Department of Transportation (DOT) as the agency responsible for securing all modes of transportation. Among other things, ATSA mandated that TSA assume responsibility for screening passengers and their property, which includes the hiring, training, and testing of the screening workforce. ATSA also mandated that TSA conduct annual proficiency reviews and provide for the operational testing of screening personnel, and that TSA provide remedial training to any screener who fails such tests. In 2002, the President issued The National Strategy for Homeland Security that supports developing red team tactics in order to identify vulnerabilities in security measures at our Nation’s critical infrastructure sectors, including the transportation sector. In 2007, TSA issued the TS-SSP that outlines its strategy and associated security programs to secure the transportation sector. While the TS-SSP does not address covert testing in aviation, it does identify that mass transit and passenger rail operators should develop covert testing exercises. Moreover, the Implementing Recommendations of the 9/11 Commission Act of 2007 requires DHS to develop and implement the National Strategy for Railroad Transportation Security, which is to include prioritized goals, actions, objectives, policies, mechanisms, and schedules for assessing the usefulness of covert testing of railroad security systems. Furthermore, the explanatory statement accompanying Division E of the Consolidated Appropriations Act, 2008 (the DHS Appropriations Act, 2008), directs TSA to be more proactive in red teaming for all modes of transportation. Specifically, the statement directs approximately $6 million of TSA’s appropriated funds for red team activities to identify potential vulnerabilities and weaknesses in airports and air cargo facilities, as well as in transit, rail, and ferry systems.

Prior to the creation of TSA, the Department of Transportation’s Federal Aviation Administration (FAA) monitored the performance of airport screeners. FAA created the “red team,” as it came to be known, to assess the commercial aviation industry’s compliance with FAA security requirements and to test whether U.S. aviation passenger and checked baggage screening systems were able to detect explosives and other threat items. TSA began its covert testing program in September 2002. TSA’s covert testing program consists of a nationwide commercial aviation testing program conducted by OI, and a local commercial airport testing program implemented by OSO and FSDs at each airport.

TSA’s National Covert Testing Program for Commercial Aviation

OI conducts national covert tests of three aspects of aviation security at a commercial airport: (1) passenger checkpoint; (2) checked baggage; and (3) access controls to secure areas and airport perimeters. OI conducts covert tests by having undercover inspectors attempt to pass threat objects, such as guns, knives, and simulated improvised explosive devices (IED), through passenger screening checkpoints and in checked baggage, and to attempt to access secure areas of the airport undetected. OI officials stated that they derived their covert testing protocols and test scenarios from prior FAA red team protocols, but updated the threat items used and increased the difficulty of the tests. According to OI officials, they also began conducting tests at airports on a more frequent basis than FAA. Initially, OI conducted tests at all of the estimated 450 commercial airports nationwide on a 3-year schedule, with the largest and busiest airports being tested each year. TSA also began using threat information to make tests more closely replicate tactics that may be used by terrorists.

The number of covert tests that OI conducts during testing at a specific airport varies by the size of the airport. The size of the OI testing teams also varies depending upon the size of the airport being tested, the number of tests that OI plans to conduct, and the number of passenger checkpoints and access points to secure areas at a particular airport. OI testing teams consist of a team leader who observes the tests and leads post-test reviews with TSOs, and inspectors who transport threat items through passenger checkpoints and secure airport areas and record test results. Team leaders usually have previous federal law enforcement experience, while inspectors often include program analysts, administrative personnel, and other TSA personnel. Prior to testing, each team leader briefs their team to ensure that everyone understands their role, the type of test to be conducted, and the threat item they will be using. For tests at passenger checkpoints and in checked baggage, OI uses different IED configurations and places these IEDs in various areas of each inspector’s body and checked baggage to create different test scenarios. Figure 1 provides an overview of TSA’s passenger checkpoint and checked baggage screening operations and equipment.

According to OI officials, on the day of testing, OI typically notifies the airport police about one half hour, and the local FSD 5 minutes, before testing begins and instructs them not to notify the TSOs that testing is being conducted. OI officials stated that they provide this notification for security and safety reasons.

Covert Testing Procedures at Passenger Checkpoints

During passenger checkpoint testing, each team of inspectors carries threat items through the passenger checkpoint. If the TSO identifies the threat item during screening, the inspector identifies him or herself to the TSO and the test is considered a pass. If the TSO does not identify the threat item, the inspector proceeds to the sterile area of the airport and the test is considered a failure. For each test, inspectors record the steps taken by the TSO during the screening process and test results, and the team leader assigns any requirements for remedial training as a consequence of a failed test. The specific types of covert tests conducted by TSA at the passenger checkpoint is sensitive security information and cannot be described in this report.

Covert Testing Procedures for Checked Baggage

Covert tests of checked baggage are designed to measure the effectiveness of the TSOs’ ability to utilize existing checked baggage screening equipment, not to test the effectiveness of the screening equipment. In covert tests of checked baggage screening, an inspector poses as a passenger and checks their baggage containing a simulated threat item at the airline ticket counter. The bag is then screened by TSOs using one of two checked baggage screening methods. At airports that have explosive detection systems (EDS), the TSO uses these machines to screen each bag. At airports that do not have EDS and at airports where certain screening stations do not have EDS, such as curbside check-in stations, the TSOs use an Explosive Trace Detection (ETD) machine to screen checked baggage. During the ETD screening process of both carry-on and checked baggage, TSOs attempt to detect explosives on passengers’ baggage by swabbing the target area and submitting the swab into the ETD machine for chemical analysis. If the machine detects an explosive substance, it alarms, and produces a readout indicating the specific type of explosive detected. The TSO is then required to resolve the alarm by performing additional screening steps such as conducting a physical search of the bag or conducting further ETD testing on and X-raying of footwear. When testing EDS and ETD screening procedures, OI uses fully assembled objects such as laptop computers, books, or packages.

Whether using EDS or ETD, if the TSO fails to identify the threat item, the inspectors immediately identify themselves to stop the checked baggage from being sent for loading onto the aircraft, and the test is considered a failure. If the TSO identifies the threat item, the inspectors also identify themselves and the test is considered a pass. If the OI inspector determines that the test failure was due to the screening equipment not working correctly, the test is considered invalid. OI conducts two types of checked baggage covert tests: Opaque object: This test is designed to determine if a TSO will identify opaque objects on the X-ray screen and conduct a physical search of the checked bag. During these tests, OI inspectors conceal a threat item that cannot be penetrated by the X-ray and appears on the EDS screen as an opaque object among normal travel objects within checked baggage.

IED in bag: This test is designed to determine if a TSO will identify an IED during a search of the bag and use proper ETD procedures to identify it as a threat. During these tests, OI inspectors conceal a simulated IED within checked baggage. In addition, the IED may be contained within other objects inside of the bag.

Covert Testing Procedures for Access Controls

OI inspectors conduct covert tests to determine if they can infiltrate secure areas of the airport, such as jet ways or boarding doors to aircraft. Each U.S. commercial airport is divided into different areas with varying levels of security. Secure areas, security identification display areas (SIDA), and air operations areas (AOA) are not to be accessed by passengers, and typically encompass areas near terminal buildings, baggage loading areas, and other areas that are close to parked aircraft and airport facilities, including air traffic control towers and runways used for landing, taking off, or surface maneuvering. Figure 2 is a diagram of the security areas at a typical commercial airport.

If inspectors are able to access secure areas of the airport or are not challenged by airport or airline employees, then the test is considered a failure. OI conducts four types of covert tests for airport access controls.

Access to SIDA: During these tests, OI inspectors who are not wearing appropriate identification attempt to penetrate the SIDA through access points, such as boarding gates, employee doors, and other entrances leading to secure areas to determine if they are challenged by airport or airline personnel.

Access to AOA: During these tests, OI inspectors who are not wearing appropriate identification attempt to penetrate access points leading from public areas to secured areas of the AOA, including vehicle and pedestrian gates through the perimeter fence, cargo areas, and general aviation facilities that provide a direct path to passenger aircraft in secure areas to determine if they are challenged by airport or airline personnel.

Access to Aircraft: During these tests, OI inspectors who are not wearing appropriate identification or who do not have a valid boarding pass attempt to penetrate access points past the passenger screening checkpoint which lead directly to aircraft, including boarding gates, employee doors, and jet ways to determine if they are challenged by airport or airline personnel.

SIDA Challenges: During these tests, OI inspectors attempt to walk through secure areas of the airport, such as the tarmac and baggage loading areas, without appropriate identification to determine if they are challenged by airport personnel. If not challenged, then the test is considered a failure.

Post-Test Reviews and Analysis

After testing at the airport is complete, team leaders conduct post-test reviews with the TSOs, supervisors, and screening managers involved in the testing. These post-test reviews include a hands-on demonstration of the threat items used during each test and provide an opportunity for TSOs to ask questions about the test. According to OI officials, the purpose of these post-test reviews is to serve as a training tool for TSOs. Following the post-test review, OI officials meet with the airport FSD to discuss the test results and any vulnerabilities identified at the airport. OI also provides the FSD with the names of each TSO required to undergo remedial training. OI usually completes all aspects of its covert tests at an airport within several days. After completing tests at each airport, OI staff document test results on standardized data collection instruments and meet to discuss the results and identify the actions that they will recommend to TSA management to address the vulnerabilities identified by the tests. The airport testing data collected are then inputted into a database by OI headquarters staff, who develop reports that summarize the tests results and the vulnerabilities identified. These reports are then presented to TSA management, such as the Administrator. OI staff also regularly brief TSA’s Administrator and management, such as the Assistant Administrator of OSO, on the results of covert tests. Since 2003, when OI completed its first covert testing report, most of OI’s reports contained specific recommendations aimed at addressing the vulnerabilities identified during covert testing.

TSA’s Local Covert Testing Program for Commercial Aviation

In February 2004, OSO authorized FSDs to conduct their own testing of local passenger and checked baggage screening operations at their airports to serve as a training tool for the TSOs and to measure their performance. Referred to as Screener Training Exercises and Assessments (STEA), FSDs conducted these local covert tests using federal employees, such as TSOs from other local airports and other federal law enforcement officers, and were given discretion to determine the number of tests conducted at their airports, the manner with which the tests were conducted, and the type of tests conducted. OSO considered STEA a tool for training TSOs in detecting threat items, and issued modular bomb kits (MBS II kits) containing simulated IEDs to be used during local testing. During STEA tests, staff placed simulated IEDs in passenger and checked baggage to determine if they would be detected by TSOs. Unlike OI’s national covert tests, STEA tests did not include tests of airport access controls. TSOs that failed STEA tests were required to undergo remedial training. In May 2005, we reported that TSA officials stated that they had not yet begun to use data from STEA testing to identify training and performance needs for TSOs because of difficulties in ensuring that local covert testing was implemented consistently nationwide. For example, because FSDs had discretion regarding the number of tests conducted, some airports conducted STEA tests regularly, while others rarely conducted tests. In addition, we previously reported that FSDs had difficulty in finding enough staff to help conduct STEA tests on a consistent basis. OSO officials recognized the limitations of the STEA program and, as a result, began to re-structure the program in September 2006. This local covert testing program was renamed the Aviation Screening Assessment Program (ASAP). ASAP is designed to test the performance of passenger and checked baggage screening systems and identify security vulnerabilities at each airport.

In April 2007, OSO began its initial 6-month cycle of ASAP, in which 1,600 tests were conducted in each grouping of airports—Category X (27 airports), category I (55 airports), and Category II through IV (369 airports). OSO compliance inspectors at each airport conduct the tests. Specific test requirements are distributed to FSDs before the start of each 6-month cycle. These test requirements stipulate the percentage of tests to conduct during peak and non-peak passenger screening periods; the percentage of basic, intermediate, or advanced tests to be conducted; and specific types of threat items that should be used during each type of test, such as IEDs or weapons. Following each test, inspectors are to brief the TSOs, supervisors, and screening managers involved in the tests on the results and notify the FSD of the results. With the first cycle of tests initiated in April 2007, TSA officials plan that any recommendations resulting from ASAP tests will be submitted to OSO management and other offices within TSA that need to know the test results. Although the testing requirements, including the level of frequency and types of tests, will not change during the initial 6-month cycle to preserve the validity of the test results, TSA officials plan to analyze the results of the tests and evaluate the need to revise the structure of the tests or the type of threat items used after testing is complete. According to OSO officials, the first cycle of ASAP tests are complete, but the results are still being analyzed by TSA to determine the overall findings from the tests.

Covert Testing as a Key Component of TSA’s Broader Risk Management Approach

TSA’s national and local aviation covert testing programs contribute to TSA’s broader risk management approach for securing the transportation sector by applying principles of risk assessment to identify vulnerabilities in commercial aviation. Risk management is a systematic and analytical process to consider the likelihood that a threat will endanger an asset, individual, or function, and to identify actions to reduce the risk and mitigate the consequences of an attack. Risk management, as applied in the homeland security context, can help federal decision-makers determine where and how to invest limited resources within and among the various modes of transportation. In recent years, the President, through Homeland Security Presidential Directives (HSPD), and laws such as the Intelligence Reform and Terrorism Prevention Act of 2004, have provided that federal agencies with homeland security responsibilities should apply risk-based principles to inform their decision making regarding allocating limited resources and prioritizing security activities. The 9/11 Commission recommended that the U.S. government should identify and evaluate the transportation assets that need to be protected, set risk-based priorities for defending them, select the most practical and cost-effective ways of doing so, and then develop a plan, budget, and funding to implement the effort. In 2002, the President issued The National Strategy for Homeland Security that instructs the federal government to allocate resources in a balanced way to manage risk in our border and transportation security systems while ensuring the expedient flow of goods, services, and people. Further, the Secretary of DHS has made risk-based decision-making a cornerstone of departmental policy. In May 2007, TSA issued the TS-SSP and supporting plans for each mode of transportation that establish a system based risk management approach for securing the transportation sector.

We have previously reported that a risk management approach can help to prioritize and focus the programs designed to combat terrorism. A risk assessment, one component of a risk management approach, consists of three primary elements: a vulnerability assessment, a threat assessment, and a criticality assessment. A vulnerability assessment is a process that identifies weaknesses in physical structures, personnel protection systems, processes, or other areas that may be exploited by terrorists, and may suggest options to eliminate or mitigate those weaknesses. TSA uses both national and local aviation covert testing as a method to identify and mitigate security vulnerabilities in the aviation sector. A threat assessment identifies and evaluates threats based on various factors, including capability and intentions as well as the lethality of an attack. Criticality assessment evaluates and prioritizes assets and functions in terms of specific criteria, such as their importance to public safety and the economy, as a basis for identifying which structures or processes require higher or special protection from attack.

TSA Has a Risk-Based Covert Testing Strategy to Identify Vulnerabilities and Measure the Performance of Selected Aviation Security Systems, but Could Strengthen Its Testing Efforts

TSA has designed and implemented risk-based national and local covert testing programs to achieve its goals of identifying vulnerabilities in and measuring the performance of passenger checkpoint and checked baggage screening systems and airport perimeters and access controls, and has begun to determine the extent to which covert testing will be used to identify vulnerabilities and measure the effectiveness of security practices related to non-aviation modes of transportation. OI used information on terrorist threats to design and implement its national covert tests and determine at which airports to conduct tests based on analyses of risks. However, OI inspectors did not systematically record specific causes for test failures related to TSOs, procedures, or screening equipment that did work properly. OI also did not systematically collect and analyze information on effective screening practices that may contribute to TSOs ability to detect threat items. Without systematically recording reasons for test failures, such as failures caused by screening equipment not working properly, as well as reasons for test passes, TSA is limited in its ability to mitigate identified vulnerabilities. TSA recently redesigned its local covert testing program to address limitations in its previous program. The new program, ASAP, should provide TSA with a measure of the performance of passenger and checked baggage screening systems and help to identify security vulnerabilities. Furthermore, TSA has begun to determine the extent to which covert testing will be used to identify vulnerabilities and measure the effectiveness of security practices in non-aviation modes of transportation. While TSA coordinates with domestic and foreign organizations regarding transportation security efforts, they do not have a systematic process in place to coordinate with these organizations regarding covert testing in non-aviation settings, and opportunities for TSA to learn from these organizations’ covert testing efforts exist.

TSA Uses a Risk-Based Covert Testing Strategy

OI uses threat assessments and intelligence information to design and implement national covert tests that meet its goals of identifying vulnerabilities in passenger checkpoint and checked baggage screening systems, and airport perimeters and access controls. While OI currently focuses it covert tests on these three areas of aviation security, it has recently begun to establish procedures for the testing of air cargo facilities. According to OI officials, as of March 2008, OI has not yet conducted any tests of air cargo. In designing its covert tests, OI works with DHS’s Transportation Security Laboratory to create threat items to be used during covert tests. OI also uses threat information to replicate tactics that may be used by terrorists. The tactics that OI uses are all designed to test the capabilities of passenger checkpoint and checked baggage screening systems to identify where vulnerabilities exist. The process OI uses to select which airports to test has evolved since covert testing began in September 2002 to focus more on those airports determined to be at greater risk of a terrorist attack. Initially, OI’s goals were to conduct covert tests at all commercial airports, with tests being conducted more frequently at those airports with the largest number of passenger boardings than smaller airports with fewer flights. In August 2005, when TSA began focusing on the most catastrophic threats, OI changed its testing strategy to utilize a risk-based approach to mitigate those threats.

OI Could Better Identify Vulnerabilities by Recording and Analyzing Specific Causes of Covert Testing Failures and Passes of National Covert Tests in Its Testing Database

OI inspectors record information on the results of national covert tests on data collection instruments after each test is conducted, including the extent to which TSOs properly followed TSA screening procedures and whether the test was passed or failed. After airport testing is complete, OI headquarters analysts input the covert test results into a centralized database. While analysts input whether the test was a pass or a fail and inspectors observations regarding some tests, they do not systematically capture OI’s assessment of the cause of the test failure and include that information in the database. Test failures could be caused by (1) TSOs not properly following existing TSA screening procedures, (2) screening procedures that are not clear to TSOs, (3) screening procedures that lack sufficient guidance to enable TSOs to identify threat items, and (4) screening equipment that does not work properly. Moreover, when inspectors determine the cause of a covert test failure to be due to screening equipment, such as the walk through metal detector, the hand- held metal detector, or ETD not alarming in response to a threat item, OI considers these tests to be invalid. While OI officials stated that they report instances when equipment may not be working properly to the airport FSD and officials from the Transportation Security Laboratory, they do not input that equipment caused the failure in the covert testing database. TSA management may find this information useful in identifying vulnerabilities in the aviation system that relate to screening equipment not working properly. OI officials stated that they do not record information on equipment failures because there is always a possibility that the simulated threat item was not designed properly and therefore should not have set off the alarm. Further, they stated that DHS’s Transportation Security Laboratory is responsible for ensuring that screening equipment is working properly. However, the Laboratory does not test screening equipment at airports in an operational environment. Furthermore, according to OI officials, identifying a single cause for a test failure may be difficult since covert testing failures can be caused by multiple factors. However, in discussions with OI officials about selected individual test results, inspectors were able in their view, in most of these cases, to identify the cause they believed contributed most to the test failure. According to the Standards for Internal Control in the Federal Government, information should be recorded and communicated to management and others in a form and within a time frame that enables them to carry out their internal control and other responsibilities. The Standards further call for pertinent information to be identified, captured, and distributed in a form and time frame that permits people to perform their duties efficiently. By not systematically inputting the specific causes for test failures in its database, including failures due to equipment, OI may be limiting its ability to identify trends that impact screening performance across the aviation security systems tested.

In addition to not identifying reasons the inspectors believed caused the test failures, OI officials do not systematically record information on screening practices that may contribute to covert test passes. However, OI inspectors occasionally captured information of effective practices used by TSOs to detect threat items during covert tests in the data collection instruments used during these tests. Further, during covert tests that we observed, OI inspectors routinely discussed with us those practices used during certain tests that they viewed as effective, such as effective communication between TSOs and supervisors in identifying threat items. In 2006, OSO officials requested a TSA internal review of differences in checkpoint screening operations at three airports to identify whether the airports employed certain practices that contributed to their ability to detect threat items during covert tests, among other things. Between June and October 2006, OI’s Internal Reviews Division (IRD) reviewed passenger checkpoint covert test results for each airport, observed airport operations, interviewed TSA personnel, and reviewed documents and information relevant to checkpoint operations. IRD’s review identified a number of key factors that may contribute to an airport’s ability to detect threat items. While IRD conducted this one time review of effective screening practices that may have led to higher test pass rates, OI does not systematically collect information on those practices that may lead to test passes. As discussed earlier in this report, Standards for Internal Control in the Federal Government stated the need for pertinent information to be identified and captured to permit managers to perform their duties efficiently. Without collecting information on effective screening practices that, based on the inspectors’ views, may lead to test passes, TSA managers are limited in their ability to identify measures that could help to improve screening performance across the aviation security system.

TSA Redesigned Its Local Covert Testing Program to Address Limitations of Its Previous Program and to Measure the Performance of Passenger Checkpoint and Checked Baggage Screening

In April 2007, TSA initiated its local covert testing program, the Aviation Screening Assessment Program (ASAP). TSA is planning to use the results of ASAP as a statistical measure of the performance of passenger checkpoint and checked baggage screening systems, in addition as a tool to identify security vulnerabilities. TSA ASAP guidance applies a standardized methodology for the types and frequency of covert tests to be conducted in order to provide a national statistical sample. If implemented as planned, ASAP should provide TSA with a measure of the performance of passenger and checked baggage screening systems and help identify security vulnerabilities. According to OSO officials, the first cycle of ASAP tests were completed, but the results are still being internally analyzed by TSA to determine the overall findings from the tests. As a result, it is too soon to determine whether ASAP will meet its goals of measuring the performance of passenger and checked baggage screening systems and identifying vulnerabilities.

Similar to OI’s national covert testing program, OSO applies elements of risk in designing and implementing ASAP tests. Unlike national covert tests, the ASAP program does not use elements of a risk-based approach to determine the location and frequency of the tests because, according to TSA officials, in order to establish a national baseline against which TSA can measure performance, all airports must be tested consistently and with the same types of tests. OSO officials plan to analyze the results of the tests and evaluate the need to revise the tests or the type of threat items used after the first and second testing cycle and annually thereafter. Furthermore, OSO officials stated that they plan to assess the data, including the types of vulnerabilities identified and the performance of the TSOs in detecting threat items, and develop recommendations for mitigating vulnerabilities and improving screening performance. Officials stated that OSO also plans to conduct follow-up testing to determine whether vulnerabilities that were previously identified have been addressed or if recommendations made were effective.

According to TSA’s ASAP guidance, individuals conducting the ASAP tests will be required to identify specific causes for all test failures. In addition to identifying test failures attributed to TSOs, such as the TSO not being attentive to their duties or not following TSA screening procedures, individuals conducting ASAP tests are also required to identify and record causes for failures related to TSOs, screening procedures that TSOs said were not clear or lack sufficient detail to enable them to detect threat items, and screening equipment.

OSO officials further stated that they plan to develop performance measures for the ASAP tests after the results of the first 6 month cycle of tests are evaluated. However, officials stated that performance measures for the more difficult category of tests will not be developed because these tests are designed to challenge the aviation security system and the pass rates are expected to be low. Furthermore, TSA officials stated that the results of ASAP tests will not be used to measure the performance of individual TSOs, FSDs, or airports, but rather to measure the performance of the passenger checkpoint and checked baggage screening system. TSA officials stated that there will not be a sufficient number of ASAP tests to measure individual TSO, FSD, or airport performance. We previously reported that TSA had not established performance measures for its national covert testing program and that doing so would enable TSA to focus its improvement efforts on areas determined to be most critical, as 100 percent detection during tests may not be attainable. While TSA has chosen not to establish performance measures for the national covert testing program, as stated above, they plan to develop such measures for only the less difficult ASAP tests.

Covert Testing in Non- Aviation Modes of Transportation

Since the initiation of TSA’s covert testing program in 2002, the agency has focused on testing commercial aviation passenger checkpoints, checked baggage, and airport perimeters and access controls. However, TSA in is the early stages of determining the extent to which covert testing will be used to identify vulnerabilities and measure the effectiveness of security practices in non-aviation modes of transportation. In addition, TSA officials stated that it would be difficult to conduct covert tests in non- aviation modes because these modes typically do not have established security screening procedures to test, such as those in place at airports.

Specifically, passengers and their baggage are not generally physically screened through metal detectors and X-rays prior to boarding trains or ferries as they are prior to boarding a commercial aircraft, making it difficult to conduct tests. OI officials also stated that they do not currently have the resources necessary to conduct covert tests in both aviation and non-aviation modes of transportation.

Although OI does not regularly conduct covert tests in non-aviation modes of transportation, it has conducted tests during three TSA pilot programs designed to test the feasibility of implementing airport style screening in non-aviation modes of transportation to include mass transit, passenger rail, and maritime ferry facilities. In 2004, TSA conducted a Transit and Rail Inspection pilot program in which passenger and baggage screening procedures were tested on select railways. TSA also tested similar screening procedures at several bus stations during the Bus Explosives Screening Technology pilot in 2005. In addition, TSA has also been testing screening equipment on ferries in the maritime mode through the Secure Automated Inspection Lanes program. According to OI officials, during these three pilot programs, OI conducted covert testing to determine if they could pass threat objects through the piloted passenger screening procedures and equipment. However, these tests were only conducted on a trial basis during these pilot programs. While OI has not developed plans or procedures for testing in non-aviation modes of transportation, the office has begun to explore the types of covert tests that it might conduct if it receives additional resources to test in these modes.

In addition to OI, TSA’s Office of Transportation Sector Network Management (TSNM) may have a role in any covert tests that are conducted in non-aviation modes of transportation. TSNM is responsible for securing the nation’s intermodal transportation system and has specific divisions responsible for each mode of transportation—mass transit, maritime, highway and motor carriers, freight rail, pipelines, commercial airports, and commercial airlines. TSNM is also responsible for TSA’s efforts to coordinate with operators in all modes of transportation. A TSNM official stated that TSNM has only begun to consider using covert testing in mass transit. In April 2007, TSA coordinated with the Los Angeles County Metropolitan Transportation Authority, Amtrak, and Los Angeles Sheriff’s Department during a covert test of the effectiveness of security measures at Los Angeles’ Union Station. During the test, several individuals carried threat items, such as simulated IEDs, into the rail system to determine if K-9 patrols, random bag checks, and other random procedures could detect these items. The official from TSNM’s mass transit office stated that the agency is incorporating the use of covert testing as a component of the mass transit and passenger rail national exercise program being developed pursuant to the Implementing Recommendations of the 9/11 Commission Act of 2007. However, TSNM has not developed a strategy or plan for how covert testing will be incorporated into these various programs. The TSNM official further stated that he was not aware of other mass transit or passenger rail operators that are currently conducting or planning covert testing of their systems. Furthermore, TSNM does not have a systematic process in place to coordinate with domestic or foreign transportation organizations to learn from their covert testing experiences.

The use of covert or red team testing in non-aviation modes of transportation has been supported in law. The Implementing Recommendations of the 9/11 Commission Act of 2007 directs DHS to develop and implement the National Strategy for Railroad Transportation Security, which is to include prioritized goals, actions, objectives, policies, mechanisms, and schedules for assessing, among other things, the usefulness of covert testing of railroad security systems. Furthermore, the explanatory statement accompanying the Homeland Security Appropriations Act, 2008, directed TSA to be more proactive in red teaming for airports and air cargo facilities, as well as in transit, rail, and ferry systems. Specifically, the statement directed approximately $6 million of TSA’s appropriated amount for red team activities to identify vulnerabilities in airports and air cargo facilities, as well as in transit, rail, and ferry systems. Regarding covert testing of non-aviation modes of transportation, the report of the House of Representatives Appropriations Committee, which accompanies its fiscal year 2008 proposal for DHS appropriations, directed TSA to randomly conduct red team operations at rail, transit, bus, and ferry facilities that receive federal grant funds to ensure that vulnerabilities are identified and corrected.

DHS has also identified covert, or red team, testing as a priority for the Department. The President’s July 2002 National Strategy for Homeland Security identified that DHS, working with the intelligence community, should use red team or covert tactics to help identify security vulnerabilities in the nation’s critical infrastructure, which includes the transportation sector. The strategy further identifies that red team techniques will help decision makers view vulnerabilities from the terrorists’ perspective and help to develop security measures to address these security gaps. In addition, TSA’s May 2007 TS-SSP identified that transit agencies should develop meaningful exercises, including covert testing, that test the effectiveness of their response capabilities and coordination with first responders. However, the TS-SSP does not provide any details on the type of covert testing that transit agencies should conduct and does not identify that TSA itself should conduct covert testing in non-aviation modes of transportation.

Select Domestic and Foreign Transportation Organizations and DHS Component Agencies Use Covert Testing to Identify Vulnerabilities and Measure System Effectiveness

Domestic and foreign transportation organizations and DHS component agencies that we interviewed conduct covert testing to identify and mitigate vulnerabilities in non-aviation settings that lack the standardized passenger screening procedures found in the commercial aviation sector and measure the effectiveness of security measures. Our previous work on passenger rail security identified foreign rail systems that use such covert testing to keep employees alert about their security responsibilities. One of these foreign organizations—the United Kingdom Department for Transport’s Transport Security and Contingencies Directorate (TRANSEC)—conducts covert testing of passenger rail and seaports in addition to aviation facilities to identify vulnerabilities related to people, security processes, and technologies. According to a TRANSEC official, TRANSEC’s non-aviation covert testing includes testing of the nation’s passenger rail system and the United Kingdom’s side of the channel tunnel between the United Kingdom and France. TRANSEC conducts a number of covert tests to determine whether employees are following security procedures established by TRANSEC or the rail operator, whether processes in place assist employees in identifying threat items, and whether screening equipment works properly. A TRANSEC official responsible for the agency’s covert testing program stated that these tests are carried out on a regular basis and are beneficial because, as well as providing objective data on the effectiveness of people and processes, they encourage staff to be vigilant with respect to security.

In our September 2005 report on passenger rail security, we recommended that TSA evaluate the potential benefits and applicability—as risk analyses warrant and as opportunities permit—of implementing covert testing processes to evaluate the effectiveness of rail system security personnel. Like TRANSEC in the United Kingdom, TSA has existing security directives that must be followed by passenger rail operators that could be tested. TSA generally agreed with this recommendation. In responding to the recommendation, TSA officials stated that the agency regularly interacts and communicates with its security counterparts in foreign countries to share best practices regarding passenger rail and transit security and will continue to do so in the future. TSA officials further stated that the agency has representatives stationed overseas at U.S. embassies that are knowledgeable about security issues across all modes of transportation. While TSA coordinates with domestic and foreign organizations regarding transportation security efforts, they do not have a systematic process in place to coordinate with these organizations regarding covert testing in non-aviation modes of transportation, and opportunities for TSA to learn from these organizations’ covert testing efforts exist.

In the United States, Amtrak has conducted covert tests to identify and mitigate vulnerabilities in their passenger rail system. Amtrak’s Office of Inspector General has conducted covert tests of intercity passenger rail systems to identify vulnerabilities in the system related to security personnel and Amtrak infrastructure. The results from these tests were used to develop security priorities that are currently being implemented by Amtrak. According to an Amtrak official, as the security posture of the organization matures, the covert testing program will shift from identifying vulnerabilities to assessing the performance of existing rail security measures.

Transportation industry associations with whom we spoke, who represented various non-aviation modes of transportation, supported the use of covert testing as a means to identify security vulnerabilities and to test existing security measures. Officials from the American Association of Railroads (AAR), which represents U.S. passenger and freight railroads, and the American Public Transportation Association (APTA), which represents the U.S. transit industry, stated that covert testing in the passenger rail and transit industries would help to identify and mitigate security vulnerabilities and increase employee awareness of established security procedures. AAR and APTA officials stated that covert testing might include placing bags and unattended items throughout a rail station or system to see if employees or law enforcement personnel respond appropriately and in accordance with security procedures. AAR and APTA officials further stated that any testing conducted by TSA would require close coordination with rail operators to determine what should be tested, the testing procedures to be used, and the practicality of such testing.

Within DHS, the U.S. Customs and Border Protection (CBP) also conducts covert testing at land, sea, and air ports of entry in the United States to test and evaluate CBP’s capabilities to detect and prevent terrorists and illicit radioactive material from entering the United States. According to CBP officials, the purpose of CBP’s covert testing program is to identify potential technological vulnerabilities and procedural weaknesses related to the screening and detection of passengers and containers entering the United States with illicit radioactive material, and to assess CBP officers’ ability to identify potential threats. As of June 2008, CBP tested and evaluated two land border crossings on their capabilities to detect and prevent terrorists and illicit radioactive material from entering the United States. In addition, CBP covertly and overtly evaluated the nation’s 22 busiest seaports for radiation detection and the effectiveness of the non- intrusive imaging radiation equipment deployed at the seaports. CBP officials also stated that the agency is planning to expand testing to address overseas ports that process cargo bound for the United States.

In addition to CBP, the DHS Domestic Nuclear Detection Office (DNDO) conducts red team testing to measure the performance of and identify vulnerabilities in equipment and procedures used to detect nuclear and radiological threats in the United States and around the world. According to DNDO officials, the agency uses the results of red team tests to help mitigate security vulnerabilities, such as identifying nuclear detection equipment that is not working correctly. DNDO also uses red team testing to determine if unclassified information exists in open sources, such as on the internet, which could potentially be used by terrorists to exploit vulnerabilities in nuclear detections systems. DNDO’s program, according to its officials, provides a means to assess vulnerabilities that an adversary is likely to exploit, and to make recommendations to either implement or improve security procedures.

TSA Could More Fully Use the Results of Covert Tests to Mitigate Security Vulnerabilities Identified in the Commercial Aviation System

TSA’s national aviation covert testing program has identified vulnerabilities in select aspects of the commercial aviation security system at airports of all sizes; however, the agency is not fully using the results of these tests to mitigate identified vulnerabilities. The specific results of these tests are classified and are presented in our classified May 2008 report. Covert test failures can be caused by various factors, including TSOs not properly following TSA procedures when screening passengers, screening equipment that does not detect a threat item, or TSA screening procedures that do not provide sufficient detail to enable TSOs to identify the threat item. Senior TSA officials, including TSA’s Administrator, are routinely briefed on the results of covert tests and provided with OI reports that describe the vulnerabilities identified by these tests and recommendations to correct identified vulnerabilities. However, OSO lacks a systematic process to ensure that OI’s recommendations are considered, and does not systematically document its rationale for why it did or did not implement OI’s recommendations. OSO and OI also do not have a process in place to assess whether the corrective action implemented mitigated the identified vulnerabilities through follow-up national or local covert tests, and if covert test results improved. According to OSO officials, TSA has other methods in place to identify whether corrective actions or other changes to the system are effective; however, officials did not provide specific information regarding these methods. Moreover, in those cases where OSO took no action to address OI’s recommendation, they did not systematically document their rationale for why they took no action. In the absence of a systematic process for considering OI’s recommendations, documenting their decision-making process, and evaluating whether corrective actions mitigated identified vulnerabilities, TSA is limited in its ability to use covert testing results to improve the security of the commercial aviation system. OSO senior leadership stated that opportunities exist to improve the agency’s processes in this area.

TSA Covert Test Results Identified Vulnerabilities in the Aviation Security System

Between September 2002 and June 2007, OI conducted more than 20,000 covert tests of passenger checkpoints, checked baggage screening systems, and airport perimeters and access control points collectively at every commercial airport in the United States regulated by TSA. The results of these tests identified vulnerabilities in select aspects of the commercial aviation security system at airports of all sizes. While the specific results of these tests and the vulnerabilities they identified are classified, covert test failures can be caused by multiple factors, including TSOs not properly following TSA procedures when screening passengers, screening equipment that does not detect a threat item, or TSA screening procedures that do not provide sufficient detail to enable TSOs to identify the threat item. TSA cannot generalize covert test results either to the airports where the tests were conducted or to airports nationwide because the tests were not conducted using the principles of probability sampling. For example, TSA did not randomly select times at which tests were conducted, nor did they randomly select passenger screening checkpoints within the airports. Therefore, each airport’s test results represent a snapshot of the effectiveness of passenger checkpoint screening, checked baggage screening, and airport access control systems, and should not be considered a measurement of any one airport’s performance or any individual TSO’s performance in detecting threat objects. Although the results of the covert tests cannot be generalized to all airports, they can be used to identify vulnerabilities in the aviation security system. TSA officials stated that they do not want airports to achieve a 100 percent pass rate during covert tests because they believe that high pass rates would indicate that covert tests were too easy and therefore were not an effective tool to identify vulnerabilities in the system.

TSA Lacks a Systematic Process to Ensure that Covert Testing Recommendations Are Considered and Actions Are Taken to Address Them If Determined Necessary

After completing its covert tests, OI provides written reports and briefings on the test results to senior TSA management, including TSA’s Administrator, Assistant Administrator of OSO, and area FSDs. In these reports and briefings, OI officials provide TSA management with the results of covert tests, describe the security vulnerabilities identified during the tests, and present recommendations to OSO that OI believes will mitigate the identified vulnerabilities. TSA’s Administrator and senior OSO officials stated that they consider the aviation security system vulnerabilities that OI presents in its reports and briefings as well as the recommendations made. However, OSO officials we spoke with stated that they do not have a systematic process in place to ensure that all of OI’s recommendations are considered or to document their rationale for implementing or not implementing these recommendations. Furthermore, TSA does not have a process in place to assess whether corrective actions taken in response to OI’s recommendations have mitigated identified vulnerabilities. Specifically, in those cases where corrective actions were taken to address OI’s recommendation, neither OSO nor OI conducted follow-up national or local covert tests to determine if the actions taken were effective. For example, in cases where OI determined that additional TSO training was needed and OSO implemented such training, OSO or OI did not conduct follow-up national or local covert testing to determine if the additional training that was implemented to address the recommendation helped to mitigate the identified vulnerability. According to OSO officials, TSA has other methods in place to identify whether corrective actions or other changes are effective; however, officials did not provide specific information regarding these methods.

Standards for Internal Control in the Federal Government require that internal controls be designed to ensure that ongoing monitoring occurs during the course of normal operations. Specifically, internal controls direct managers to (1) promptly evaluate and resolve findings from audits and other reviews, including those showing deficiencies and recommendations reported by auditors and others who evaluate agencies’ operations, (2) determine proper actions in response to findings and recommendations from audits and reviews, and (3) complete, within established time frames, all actions that correct or otherwise resolve the matters brought to management’s attention. The standards further identify that the resolution process begins when audit or other review results are reported to management, and is completed only after action has been taken that (1) corrects identified deficiencies, (2) produces improvements, or (3) demonstrates the findings and recommendations do not warrant management action. In the absence of a systematic process for considering and resolving the findings and recommendations from OI’s covert tests and ensuring that the effectiveness of actions taken to address these recommendations are evaluated, TSA management is limited in its ability to mitigate identified vulnerabilities to strengthen the aviation security system.

OI Made 43 Recommendations to OSO to Mitigate Vulnerabilities Identified by Covert Tests from March 2003 to June 2007

While neither OSO nor OI have a systematic process for tracking the status of OI covert testing recommendations, at our request, OSO officials provided information indicating what actions, if any, were taken to address OI’s recommendations. From March 2003 to June 2007, OI made 43 recommendations to OSO designed to mitigate vulnerabilities identified by national covert tests. To date, OSO has taken actions to implement 25 of these recommendations. For the remaining 18 of OI’s 43 recommendations, OSO either took no action to address the recommendation, or it is unclear how the action they took addressed the recommendation. OI did not make any recommendations to OSO related to screening equipment. The specific vulnerabilities identified by OI during covert tests and the specific recommendations made, as well as corrective actions taken by OSO, are classified.

Conclusions

TSA has developed a risk-based covert testing strategy to identify vulnerabilities and measure the performance of select aspects of the aviation security system. OI’s national covert testing program is designed and implemented using elements of a risk-based approach, including using information on terrorist threats to design simulated threat items and tactics. However, this program could be strengthened by ensuring that all of the information from the tests conducted is used to help identify and mitigate security vulnerabilities. For example, without a process for recording and analyzing the specific causes of all national covert test failures, including TSOs not properly following TSA’s existing screening procedures, procedures that are unclear to TSOs, or screening equipment that is not working properly, TSA is limited in its ability to identify specific areas for improvement, such as screening equipment that may be in need of repair or is not working correctly. Moreover, without collecting and analyzing information on effective practices used at airports that performed particularly well on national covert tests, TSA may be missing opportunities to improve TSO performance across the commercial aviation security system. TSA has only recently begun to determine the extent to which covert testing may be used to identify vulnerabilities and measure the effectiveness of security practices in non-aviation modes of transportation if it receives additional resources to test in these modes. Nevertheless, several transportation industry stakeholders can provide useful information on how they currently conduct covert tests in non- aviation settings, and systematically coordinating with these organizations could prove useful for TSA.

National aviation covert tests have identified vulnerabilities in the commercial aviation security system. However, TSA could better use the covert testing program to mitigate these vulnerabilities by promptly evaluating and responding to OI’s findings and recommendations. We recognize that TSA must balance a number of competing interests when considering whether to make changes to TSO training, screening procedures, and screening equipment within the commercial aviation security system, including cost and customer service, in addition to security concerns. We further recognize that, in some cases, it may not be feasible or appropriate to implement all of OI’s recommendations. However, without a systematic process in place to consider OI’s recommendations, evaluate whether corrective action is needed to mitigate identified vulnerabilities, and evaluate whether the corrective action effectively addressed the vulnerability, OSO is limited in the extent to which it can use the results of covert tests to improve the security of the commercial aviation system.

Recommendations for Executive Action

To help ensure that the results of covert tests are more fully used to mitigate vulnerabilities identified in the transportation security system, we recommended in our May 2008 classified report that the Assistant Secretary of Homeland Security for TSA take the following five actions: Require OI inspectors to document the specific causes of all national covert testing failures—including documenting failures related to TSOs, screening procedures, and equipment—in the covert testing database to help TSA better identify areas for improvement, such as additional TSO training or revisions to screening procedures.

Develop a process for collecting, analyzing, and disseminating information on practices in place at those airports that perform well during national and local covert tests in order to assist TSA managers in improving the effectiveness of checkpoint screening operations.

As TSA explores the use of covert testing in non-aviation modes of transportation, develop a process to systematically coordinate with domestic and foreign transportation organizations that already conduct these tests to learn from their experiences.

Develop a systematic process to ensure that OSO considers all recommendations made by OI in a timely manner as a result of covert tests, and document its rationale for either taking or not taking action to address these recommendations.

Require OSO to develop a process for evaluating whether the action taken to implement OI’s recommendations mitigated the vulnerability identified during covert tests, such as using follow-up national or local covert tests to determine if these actions were effective.

Agency Comments and Our Evaluation

We provided a draft of this report to DHS for review and comment. On April 24, 2008, we received written comments on the draft report, which are reproduced in full in appendix II. DHS and TSA concurred with the findings and recommendations, and stated that the report will be useful in strengthening TSA’s covert testing programs. In addition, TSA provided technical comments, which we incorporated as appropriate.

Regarding our recommendation that OI document the specific causes of all national covert testing failures related to TSOs, screening procedures, and equipment in the covert testing database, DHS stated that TSA’s Office of Inspection (OI) plans to expand the covert testing database to all causes of test failures. DHS further stated that the specific causes of all OI covert testing failures are documented in data collection instruments used during covert tests and within a comment field in the covert testing database when the cause can be determined. However, TSA acknowledged that covert test failures caused by screening equipment not working properly are not recorded in the database in a systematic manner. Documenting test failures caused by equipment should help OI better analyze the specific causes of all national covert testing failures and assist TSA management in identifying corrective actions to mitigate identified vulnerabilities.

Concerning our recommendation that OI develop a process for collecting, analyzing, and disseminating information on practices in place at those airports that perform well during national and local covert tests in order to assist TSA managers in improving the effectiveness of checkpoint screening operations, DHS stated that it recognizes the value in identifying factors that may lead to improved screening performance. TSA officials stated that, while OI or ASAP test results can be used to establish a national baseline for screening performance at individual airports, the results are not statistically significant. As a result, additional assessments would be required to provide a statistical measure for individual airports. According to DHS, OI plans to develop a more formal process for collecting and analyzing test results to identify best practices that may lead to test passes. Officials stated that when specific screening practices indicate a positive effect on screening performance, TSA plans to share and institutionalize best practices in the form of management advisories to appropriate TSA managers. Developing a more formal process for collecting and analyzing test results to identify best practices that may lead to test passes should address the intent of this recommendation.

In response to our recommendation that TSA develop a process to systematically coordinate with domestic and foreign transportation organizations as the agency explores the use of covert testing in non- aviation modes of transportation to learn from their experiences, DHS stated that it is taking a number of actions. Specifically, according to DHS, TSNM has worked closely with transit agencies and internal TSA covert testing experts during red team testing exercises and is currently exploring programs in which covert testing may be used to evaluate the effectiveness of security measures. For example, TSNM is considering incorporating covert testing as a part of its Intermodal Security Training and Exercise Program. While considering the use of covert testing in its programs should help TSA evaluate the effectiveness of security measures, it is also important that TSA establish a systematic process for coordinating with domestic and foreign organizations that already conduct testing in non-aviation modes of transportation to learn from their experiences.

DHS further stated that it plans to take action to address our recommendation that the agency develop a systematic process to ensure that OSO considers all recommendations made by OI as a result of covert tests in a timely manner, and documents its rationale for either taking or not taking action to address these recommendations. Specifically, DHS stated that OSO is coordinating with OI to develop a directive requiring that OI’s covert testing recommendations be formally reviewed and approved by TSA management, and OSO is establishing a database to track all OI recommendations and determine what action, if any, has been taken to address the recommendation. Taking these steps should address the intent of this recommendation and help TSA to more systematically record whether OI’s covert testing recommendations have been addressed.

Concerning our recommendation that OSO develop a process to evaluate whether the action taken to implement OI’s recommendations mitigated the vulnerability identified during covert tests, such as using follow-up national or local covert tests or information collected through other methods, to determine if these actions were effective, DHS stated that OSO established a new program to study various aspects of TSO and screening performance in 2007 that considers recommendations originating from OI national covert tests and ASAP tests. According to DHS, after completing each study, recommendations resulting from this analysis will be provided to TSA leadership for consideration. DHS further stated that the results of ASAP tests will also likely be a focus of these future studies. While these actions should help to address the intent of this recommendation, it is also important that OSO assess whether the actions taken to mitigate the vulnerabilities identified by OI’s national covert tests are effective.

As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after its issue date. At that time, we will send copies of this report to the Secretary of Homeland Security, Assistant Secretary of DHS for the Transportation Security Administration, and the Ranking Member of the Committee on Homeland Security, House of Representatives, and other interested congressional committees as appropriate. We will also make this report available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3404 or at berrickc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Other key contributors to this report were John Hansen, Assistant Director; Chris Currie; Yanina Golburt; Samantha Goodman; Art James; Wendy Johnson; Thomas Lombardi; and Linda Miller.

Appendix I: Objectives, Scope, and Methodology

This report addresses the following questions: (1) what is the Transportation Security Administration’s (TSA) strategy for conducting covert testing of the transportation system, and to what extent has the agency designed and implemented its covert tests to achieve identified goals? and (2) what have been the results of TSA’s national aviation covert tests conducted from September 2002 to June 2007, and to what extent does TSA use the results of these tests to mitigate security vulnerabilities in the commercial aviation system?

To identify TSA’s strategy for conducting covert testing of the transportation system and the extent to which the agency has designed and implemented its covert tests to achieve identified goals, we reviewed applicable laws, regulations, policies, and procedures to determine the requirements for conducting covert testing in the transportation sector. To assess TSA’s strategy specifically in the aviation covert testing program, we interviewed TSA Office of Inspection (OI) officials responsible for conducting national covert tests and Office of Security Operations (OSO) officials responsible for local covert tests regarding the extent to which information on risks is included in the design and implementation of tests. We also interviewed the Transportation Security Officers (TSO), supervisors, screening managers, and Federal Security Directors (FSD) who participated in covert tests at each airport where we observed tests to discuss their experience with the national and local covert testing programs. We observed OI inspectors during covert tests at seven airports including airports with heavy passenger traffic and those with just a few flights per day, as well as airports with both federal and contract TSOs. During these observations, we accompanied OI inspectors during all phases of the covert test including planning and observations, testing, and post test reviews with TSOs, supervisors, and screening managers. While these seven airports represent reasonable variations in size and geographic locations, our observations of OI’s covert tests and the perspectives provided by TSA officials at these airports cannot be generalized across all commercial airports. However, our observations at the seven airports provided us an overall understanding of how OI conduct covert tests and useful insights provided by TSOs, their supervisors, and FSDs at these airports. We analyzed TSA documents including established protocols for national and local covert testing, procedures for screening passengers and checked baggage, and OI covert testing reports issued from 2002 to 2007 to identify procedures for designing and implementing TSA’s covert testing program. Furthermore, to determine the extent to which TSA met the goals of the program, we conducted a detailed analysis of the data collection instrument and methods that OI used to collect covert testing data for the seven airports where we observed covert tests.

We also assessed the adequacy of TSA’s internal controls for collecting and maintaining the results of covert tests by evaluating TSA’s processes for collecting covert testing data and inputting this data into its database. In assessing the adequacy of internal controls, we used the criteria in GAO’s Standards for Internal Control in the Federal Government, GAO/AIMD 00-21.3.1, dated November 1999. These standards, issued pursuant to the requirements of the Federal Managers’ Financial Integrity Act of 1982 (FMFIA), provide the overall framework for establishing and maintaining internal control in the federal government. Also pursuant to FMFIA, the Office of Management and Budget issued Circular A-123, revised December 21, 2004, to provide the specific requirements for assessing the reporting on internal controls. To assess TSA’s strategy for conducting covert tests in non-aviation modes of transportation, we interviewed officials from TSA’s Office of Transportation Sector Network Management (TSNM) regarding the extent to which TSA has conducted covert testing in non-aviation modes of transportation, the applicability and potential use of covert testing in other modes, and their future plans for conducting covert testing in other modes. To understand how other organizations and federal agencies have used covert testing in the non- aviation arena, we interviewed officials from selected federal agencies and organizations that conduct covert testing including Amtrak, the United Kingdom Department for Transport Security (TRANSEC), U.S. Customs and Border Protection (CBP), DHS Domestic Nuclear Detection Office (DNDO), and select transportation industry associations. We reviewed the president’s National Strategy for Homeland Security and TSA’s Transportation Systems Sector Specific plan, including individual plans for each mode of transportation, to determine the role and use of covert testing across the transportation system. We also reviewed the fiscal year 2008 DHS appropriations legislation, enacted as Division E of the Consolidated Appropriations Act, 2008, and associated committee reports and statements to identify any funding allocated to TSA to conduct covert testing in non-aviation modes.

To determine the results of TSA’s national covert tests and the extent to which TSA used the results of these tests to mitigate security vulnerabilities in the aviation system, we obtained and analyzed a database of the results of TSA’s national covert tests conducted from September 2002 to June 2007. We analyzed the test data according to airport category, threat item, and type of test conducted between September 2002 and June 2007. We also examined trends in pass and failure rates when required screening steps were or were not followed and examined differences in covert test results between private and federal airports. We assessed the reliability of TSA’s covert testing data by reviewing existing information about the data and the systems used to produce them, and by interviewing agency officials responsible for maintaining the database. We determined that the data were sufficiently reliable for our analysis and the purposes of this report. TSA provided us with a copy of their covert testing database which contained a table with one record, or entry, per test for all of the tests conducted between 2002 and 2007. In order to accurately interpret the data, we reviewed information provided by OI officials regarding each of the fields recorded in the database and information about how they enter test results into the database. We also conducted manual testing of the data, conducting searches for missing data and outliers. To further assess the reliability of the data, we reviewed the source documents used to initially collect the data as well as OI’s published reports. We also interviewed OI officials regarding how the results of covert tests are used in developing their recommendations to TSA management. We reviewed OI reports on the results of covert tests issued between March 2003 and June 2007 that were submitted to TSA’s Administrator and OSO to identify OI’s recommendations for mitigating the vulnerabilities identified during covert tests. We obtained and analyzed a summary of the actions that OSO had taken to address OI’s recommendations for mitigating vulnerabilities made from March 2003 to June 2007. We also asked officials to discuss the extent to which OSO has addressed and implemented recommendations made by OI based on covert test results, and we analyzed information provided by TSA regarding the status of each covert testing recommendation made by OI from 2003 to 2007.

We conducted this performance audit from October 2006 to May 2008, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on out audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Homeland Security