Secure Flight: TSA Should Take Additional Steps to Determine Program Effectiveness
Released: 2014-09-18T00:00:00

Letter

Background

Responsibility for Secure Flight Operations

Several entities located within TSA’s Office of Intelligence and Analysis share responsibility for administering the Secure Flight program. Among these are the Operations Strategy Mission Support Branch, which acts as the program’s lead office; the Secure Flight Operations Branch, which oversees passenger vetting and other operational activities; and the Systems Management and Operations Branch and the Secure Flight Technology Branch, both of which focus on technology-related issues. Collectively, these entities received about $93 million to carry out program operations in fiscal year 2014.

Overview of Secure Flight Matching and Screening Processes at Implementation

The Secure Flight program, as implemented pursuant to the 2008 Secure Flight Final Rule, requires U.S.- and foreign-based commercial aircraft operators traveling to, from, within, or overflying the United States, as well as U.S. commercial aircraft operators with international point-to-point flights, to collect information from passengers and transmit that information electronically to TSA. This information, known collectively as Secure Flight Passenger Data (SFPD), includes personally identifiable information, such as full name, gender, date of birth, passport information (if available), and certain nonpersonally identifiable information, such as itinerary information and the unique number associated with a travel record (record number locator).

Since implementation began in January 2009, the Secure Flight system has identified high-risk passengers by matching SFPD against the No Fly List and the Selectee List, subsets of the Terrorist Screening Database (TSDB), the U.S. government’s consolidated watchlist of known or suspected terrorists maintained by the Terrorist Screening Center, a multiagency organization administered by the Federal Bureau of Investigation (FBI). (We discuss screening activities initiated after TSA began implementing Secure Flight in 2009 later in this report.) To carry out this matching, the Secure Flight system conducts automated matching of passenger and watchlist data to identify a pool of passengers who are potential matches to the No Fly and Selectee Lists. Next, the system compares all potential matches against the TSA Cleared List, a list of individuals who have applied to, and been cleared through, the DHS redress process. Passengers included on the TSA Cleared List must submit a redress number when making a reservation, which allows the Secure Flight system to recognize and clear them. After the system performs automated matching, Secure Flight analysts are to conduct manual reviews of potential matches, which may involve consulting other classified and unclassified data sources, to further rule out individuals who are not those included on the No Fly and Selectee Lists.

After the completion of manual reviews, TSA precludes passengers who remain potential matches to certain lists from receiving their boarding passes. These passengers, for whom air carriers receive a “passenger inhibited” message from Secure Flight, must undergo a resolution process at the airport. This process may involve air carriers sending updated passenger information back to Secure Flight for automated rematching or placing a call to Secure Flight for assistance in resolving the match. At the conclusion of automated and manual screening processes (including the airport resolution process) air carriers may not issue a boarding pass to a passenger until they receive from Secure Flight a final screening determination. These determinations include a “cleared” message, for passengers found not to match a watchlist, and a “selectee” message, for matches to the Selectee List who are to be to be designated by air carriers for enhanced screening. For passengers matching the No Fly List, Secure Flight’s initial “passenger inhibited” message is the final determination, and the air carrier may not issue a boarding pass (see fig. 1).

Passenger Screening at Airport Security Checkpoints

In general, passengers undergo one of three types of screening, based on the Secure Flight determinations shown on boarding passes— standard screening, enhanced screening for selectees, and expedited screening for low-risk passengers. Standard screening typically includes a walk-through metal detector or Advanced Imaging Technology screening, which is to identify objects or anomalies concealed under clothing, and X-ray screening for the passenger’s accessible property. In the event a walk-through metal detector triggers an alarm, the Advanced Imaging Technology identifies an anomaly, or the X-ray machine identifies a suspicious item, additional security measures, such as pat- downs, explosives trace detection searches (which involve a device certified by TSA to detect explosive particles), or additional physical searches may ensue as part of the resolution process. Enhanced screening includes, in addition to the procedures applied during a typical standard screening experience, a pat-down and an explosives trace detection search or physical search of the interior of the passenger’s accessible property, electronics, and footwear. Expedited screening typically includes walk-through metal detector screening and X-ray screening of the passenger’s accessible property, but unlike in standard screening, travelers do not have to, among other things, remove their belts, shoes, or light outerwear. Passengers not designated for enhanced or expedited screening generally receive standard screening unless, for example, identified by TSA for a different type of screening through the application of random and unpredictable security measures at the screening checkpoint.

Secure Flight Initially Identified Passengers on Terrorist Watchlists and Now Also Differentiates Passengers Based on Risk

Since January 2009, the Secure Flight program has changed from one that identifies high-risk passengers by matching them against the No Fly and Selectee Lists to one that assigns passengers a risk category: high risk, low risk, or unknown risk.passengers as high risk if they are matched to watchlists of known or suspected terrorists or other lists developed using certain high-risk criteria, as low risk if they are deemed eligible for expedited screening through TSA PreTM—a 2011 initiative to preapprove passengers for expedited screening—or through the application of low-risk rules, and as unknown risk if they do not fall within the other two risk categories. To separate passengers into these risk categories, TSA utilizes lists in addition to the No Fly and Selectee Lists, and TSA has adapted the Secure Flight system to perform risk assessments, a new system functionality that is distinct from both watchlist matching and matching against lists of known travelers. At airport checkpoints, those passengers identified as high risk receive enhanced screening, passengers identified Specifically, Secure Flight now identifies as low risk are eligible for expedited screening, and passengers identified as unknown risk generally receive standard screening.

Secure Flight Is Using New High-Risk Lists for Screening, Including Two Lists of Individuals Who Meet Various Threat Criteria, but Who May Not Be Known or Suspected Terrorists

Since January 2009, TSA has been using new high-risk lists for screening, including two lists to identify passengers who may not be known or suspected terrorists, but who—based on TSA’s application of threat criteria—should receive enhanced screening, and an expanded list of known or suspected terrorists in the TSDB. As initially implemented under the October 2008 Secure Flight Final Rule, the program matched the names of passengers against the No Fly and Selectee List components of the TSDB. According to the rule, comparing passenger information against the No Fly and Selectee components of the TSDB (versus the entire TSDB) would be generally satisfactory during normal security circumstances to counter the security threat. The rule also provides that TSA may use the larger set of watchlists maintained by the federal government as warranted by security considerations, for example, if TSA learns that flights on a particular route may be subject to an increased security risk.compare passenger information on some or all flights on that route against the full TSDB or other government databases, such as intelligence or law enforcement databases.

Rules-Based Watchlists

After the December 25, 2009, attempt to detonate a concealed explosive on board a U.S.-bound flight by an individual who was not a known or suspected terrorist in the TSDB, TSA sought to identify ways to mitigate unknown threats—individuals not in the TSDB for whom TSA has determined enhanced screening would be prudent. To that end, TSA worked with CBP to develop new lists for Secure Flight screening, and in April 2010, began using the lists to identify and designate for enhanced screening passengers who may represent unknown threats. To create these lists, TSA leveraged CBP’s access to additional data submitted by passengers traveling internationally and the capabilities of CBP’s Automated Targeting System-Passenger (ATS-P)—a tool originally created and used by CBP that targets passengers arriving at or departing the United States by comparing their information against law enforcement, intelligence, and other enforcement data using risk-based targeting scenarios and assessments. Specifically, analysts within the Intelligence and Analysis Division of TSA’s Office of Intelligence and Analysis review current intelligence to identify factors that may indicate an elevated risk for a passenger. TSA creates rules based on these factors and provides them to CBP. CBP then uses ATS-P to identify passengers who correspond with the rules and provides TSA information on them in the form of a list. Upon receiving the list, TSA creates another rules-based list—a subset of the larger rules-based list—based on additional criteria. Through Secure Flight screening, TSA designates passengers matching either rules-based list as selectees for enhanced screening.

The Expanded Selectee List

In addition to the two ATS-P-generated lists, Secure Flight incorporated an additional list derived from the TSDB into its screening activities in order to designate more passengers who are known or suspected terrorists as selectees for enhanced screening. Specifically, in April 2011, TSA began conducting watchlist matching against an Expanded Selectee List that includes all records in the TSDB with a full name (first name and surname) and full date of birth that meet the Terrorist Screening Center’s reasonable suspicion standard to be considered a known or suspected terrorist, but that are not already included on the No Fly or Selectee List. TSA began using the Expanded Selectee List in response to the December 25, 2009, attempted attack, as another measure to secure civil aviation. Collectively, the No Fly, Selectee, and Expanded Selectee Lists are used by Secure Flight to identify passengers from the government’s consolidated database of known or suspected terrorists.

Secure Flight Is Identifying Low-Risk Passengers by Screening against TSA PreTM Lists and Conducting Passenger Risk Assessments

Since October 2011, TSA has also begun using Secure Flight to identify passengers as low risk, and therefore eligible for expedited screening, through the use of new screening lists and by performing passenger risk assessments. According to TSA, identifying more passengers as eligible for expedited screening will permit TSA to reduce screening resources for low-risk travelers, thereby enabling TSA to concentrate screening resources on higher-risk passenger populations. In August 2013, TSA officials stated that this approach would support the agency’s goal of identifying 25 percent of airline passengers as eligible for expedited screening by the end of calendar year 2013. As of May 2014, TSA officials stated the goal had been revised to identify 50 percent of airline passengers as eligible for expedited screening by the end of calendar year 2014. According to officials within TSA’s Office of Chief Counsel, TSA’s efforts to identify low-risk travelers also fulfill a stated goal of the 2008 Secure Flight rule to implement a “known traveler” concept that would allow the federal government to assign a unique number to known travelers for whom the federal government had conducted a threat assessment and determined did not pose a security threat.

TSA PreTM Lists of Preapproved Low-Risk Travelers

We expect to issue a report on TSA’s PreTM program later this year. entry). Since then, TSA has established separate TSA PreTM lists for additional low-risk passenger populations, including members of the U.S. armed forces, Congressional Medal of Honor Society members, and members of the Homeland Security Advisory Council (see app. II for a full listing of TSA PreTM lists used by Secure Flight for screening). To identify these and other low-risk populations, TSA coordinated and entered into agreements with a lead agency or outside entity willing to compile and maintain the associated TSA PreTM list. Members of the list-based, low-risk populations participating in TSA PreTM are provided a unique known traveler number, and their personal identifying information (name and date of birth), along with the known traveler number, is included on lists used by Secure Flight for screening.

In addition to TSA PreTM lists sponsored by other agencies or entities, TSA created its own TSA PreTM list composed of individuals who apply to be preapproved as low-risk travelers through the TSA PreTM Application Program, an initiative launched in December 2013. The program is another DHS trusted traveler program, in which DHS collects a fee to conduct a background investigation for applicants. Applicants approved as low risk through the program receive a known traveler number and are included on an associated TSA PreTM Application Program list used by Secure Flight for screening. To be recognized as low risk by the Secure Flight system, individuals on TSA PreTM lists must submit their known traveler numbers when making a flight reservation. As of April 2014, there were about 5.6 million individuals who, through TSA PreTM program lists, were eligible for expedited screening.

TSA PreTM Risk Assessments

To further increase the number of passengers identified as low risk (and therefore TSA PreTM eligible), TSA adapted the Secure Flight system to begin assigning passengers risk scores to designate them as low risk for a specific flight. Beginning in 2011, TSA piloted a risk-based security program to identify certain members of participating airlines’ frequent flier programs as low risk, and therefore eligible for expedited screening for a specific flight. Specifically, TSA used the Secure Flight system to assess data submitted by these frequent fliers during the course of travel and assign them scores, which were then used to determine eligibility for expedited screening. In October 2013, TSA expanded the use of these assessments to all passengers, not just frequent fliers, and also began using other travel-related data to assess passengers. These assessments are conducted only if the passenger has not been designated as high risk by other Secure Flight screening activities or matched to one of the TSA PreTM Lists. The scores assigned to passengers correspond with a certain likelihood of being designated as eligible to receive expedited screening through TSA PreTM. According to officials within TSA’s Office of Chief Counsel, the assessments are not watchlist matching, rather they are a means to facilitate the secure travel of the public—a purpose of Secure Flight, as stated in the program’s final rule and in accordance with TSA’s statutory responsibilities to ensure the security of civil aviation.

As of May 2014, TSA uses PreTM risk assessments to determine a passenger’s low-risk status and resulting eligibility for TSA PreTM expedited screening, but according to TSA officials, TSA also has the capability to use this functionality to identify high-risk passengers for enhanced screening. TSA made adjustments to enable the Secure Flight system to perform TSA PreTM risk assessments to identify high-risk passengers in March 2013. However, TSA officials stated the agency has no immediate plans to use the assessments to identify high-risk passengers beyond those already included on watchlists.

New Secure Flight Screening Activities Allow TSA to Differentiate Passengers by Risk Category

Given the changes in the program since implementation, the current Secure Flight system screens passengers and returns one of four screening results to the air carriers for each passenger: TSA PreTM eligible (expedited screening), cleared to fly (standard screening), selectee (enhanced screening), or do not board (see fig. 2).

TSA Has Processes in Place to Implement Secure Flight Screening Determinations at Checkpoints, but Could Take Further Action to Address Screening Errors

TSA has developed processes to help ensure that individuals and their accessible property receive a level of screening at airport checkpoints that corresponds to the level of risk determined by Secure Flight. However, TSA could take additional actions to prevent TSO errors in implementing these risk determinations at the screening checkpoint. Furthermore, fraudulent identification or boarding passes could enable individuals to evade Secure Flight vetting, creating a potential vulnerability at the screening checkpoint. TSA’s planned technology solutions could reduce the risk posed by fraudulent documents at the screening checkpoint.

TSA Has Developed Processes to Implement Secure Flight Determinations at Airport Checkpoints

TSA has developed processes to help ensure that individuals and their accessible property receive a level of screening at airport checkpoints that corresponds to the level of risk determined by Secure Flight.are primarily responsible for ensuring that passengers receive the appropriate level of screening because they are to verify passengers’ identities and identify passengers’ screening designations. TSA requires passengers to present photo identification and a boarding pass at the TDCs screening checkpoint. Using lights and magnifiers, which allow the TDC to examine security features on the passenger’s identification documents, the TDC is to examine the identification and boarding pass to confirm that they appear genuine and pertain to the passenger. The TDC is also to confirm that the data included on the boarding pass and in the identity document match one another. According to TSA standard operating procedures, TDCs may accept minor name variations between the passenger’s boarding pass and identification.information on the identification varies significantly from the boarding pass, the TDC is to refer the passenger to another TSA representative for identity verification through TSA’s Identity Verification Call Center (IVCC). If the passenger’s information varies from the SFPD submitted to Secure Flight, the IVCC is to contact Secure Flight to vet the new information. If the identification or boarding pass appears fraudulent, the TDC is to contact law enforcement.

If the TDC finds that the The TDC is also required to review the passenger’s boarding pass to identify his or her Secure Flight passenger screening determination—that is, whether the passenger should receive standard, enhanced, or expedited screening. TDCs either examine the boarding pass manually or, where available, scan the boarding pass using an electronic boarding pass scanning system (BPSS). In addition, Secure Flight provides TSA officials in the airports with advance notice of upcoming selectees from the Selectee and Expanded Selectee Lists, as well as those on the No Fly List. Secure Flight provides this information to TSA officials at the passenger’s airport of departure via e-mail beginning 72 hours prior to flight departure for the No Fly and Selectee Lists, and via a shared electronic posting beginning 26 to 29 hours prior to flight departure for the Expanded Selectee List.

TSA also has requirements related to TDC performance. First, according to TSA officials, TSA designated the TDC a qualified position in February 2013, meaning that TSOs must complete training and pass a job knowledge test to qualify as TDCs. Second, TSA has documented processes to govern the screening checkpoint, such as standard operating procedures applicable to the TDC, the screening checkpoint, and expedited screening that specify responsibilities and lines of reporting. In March 2011, TSA also updated its Screening Management standard operating procedures to clarify that supervisory TSOs are required to monitor TSO performance to ensure compliance with all applicable standard operating procedures and correct improper or faulty application of screening procedures to ensure effective, vigilant, and courteous screening. According to officials in TSA’s Office of Inspection, many checkpoint failures resulted from a lack of supervision. These officials stated that when TDCs are not properly supervised, they are more likely to take shortcuts and miss steps in the standard operating procedures and that because working as a TDC can be tedious and repetitive, supervision and regular rotation are particularly important to ensure TDCs’ continued vigilance.

TSOs Have Made Errors in Implementing Secure Flight Screening Determinations at the Screening Checkpoint, and Additional Actions Could Reduce the Number of Screening Errors

Our analysis of TSA information from May 2012 through February 2014 found that TSOs have made errors in implementing Secure Flight risk determinations at the screening checkpoint. By evaluating the root causes of these errors and implementing corrective measures to address those root causes, TSA could reduce the risk posed by TSO error at the screening checkpoint. TSA officials we spoke with at five of the nine airports conduct after-action reviews of screening errors at the checkpoint and have used these reviews to take action to address the root causes of those errors. However, TSA does not have a systematic process for evaluating the root causes of screening errors at the checkpoint across airports, which could allow TSA to identify trends across airports and target nationwide efforts to address these issues.

TSA OSO officials told us that evaluating the root causes of screening errors would be helpful and could allow them to better target TSO training efforts. In January 2014, TSA OSO officials stated that they are in the early stages of forming a group to discuss these errors. However, TSA was not able to provide documentation of the group’s membership, purpose, goals, time frames, or methodology. Standards for Internal Control in the Federal Government states that managers should compare actual performance with expected results and analyze significant differences.will be important for TSA to develop a process for evaluating the root causes of screening errors at the checkpoint and identify and implement corrective measures, as needed, to address these root causes. Uncovering and addressing the root causes of screening errors could help TSA reduce the number of these errors at the checkpoint.

Fraudulent Documents Pose Risks at Airport Screening Checkpoints, and TSA’s Planned Technology Solutions Are in Early Stages

Fraudulent identification or boarding passes could enable individuals to evade Secure Flight vetting, creating a potential vulnerability at the screening checkpoint. TDCs are responsible for verifying the validity of identification documents and boarding passes presented by passengers. In June 2012, the TSA Assistant Administrator for the Office of Security Capabilities testified before Congress that the wide variety of identifications and boarding passes presented to TDCs poses challenges to effective manual verification of passenger identity, ticketing, and vetting status. He testified that there are at least 2,470 different variations of identification that could be presented at security checkpoints and stated that it is very difficult for a TSO to have a high level of proficiency for all of those identifications. From May 2012 through July 2013, TSA denied 1,384 individuals access to the sterile area as a result of identity checking procedures. These denials include travelers who did not appear to match the photo on their identification, who presented identification that appeared fraudulent or showed signs of tampering, and who were unwilling or unable to provide identifying information.time period, TDCs also made 852 referrals to airport law enforcement because of travelers who did not appear to match the photo on their identification, presented identification or boarding passes that appeared fraudulent or showed signs of tampering, or exhibited suspicious behaviors. However, TSA would not know how many travelers successfully flew with fraudulent documents unless those individuals came to TSA’s attention for another reason.

During this same We have previously reported on security vulnerabilities involving the identity verification process at the screening checkpoint. For example, in our May 2009 report on Secure Flight, we identified a vulnerability involving the Secure Flight system—namely, airline passengers could provide fraudulent information when making a flight reservation to avoid detection. In addition, in June 2012, we reported on several instances when passengers used fraudulent documentation to board flights. For example, we reported that in 2006, a university student created a website that enabled individuals to create fake boarding passes. In addition, in 2011, a man was convicted of stowing away aboard an aircraft after using an expired boarding pass with someone else’s name on it to fly from New York to Los Angeles. We also reported that news reports have highlighted the apparent ease of ordering high-quality counterfeit driver’s licenses from China.

TSA’s planned technology solutions could reduce the risk posed by fraudulent documents at the screening checkpoint. Boarding pass scanners are designed to verify the digital signature on these boarding passes, allowing TDCs to know that the boarding passes are genuine. The scanners are also to notify the TDC when a passenger is a selectee. In September 2013, TSA purchased 1,400 boarding pass scanners, at a cost of $2.6 million, and planned to deploy 1 for every TDC at airport security checkpoints, beginning with TDCs in TSA PreTM lines. According to TSA officials, as of March 2014, TSA had deployed all 1,400 scanners at airport security checkpoints.

In December 2013, TSA released a request for proposal for Credential Authentication Technology (CAT), which is a system that is designed to verify passenger identity, ticketing status, and Secure Flight risk determination at the screening checkpoint. CAT could address the risks of fraudulent identifications, as well as TSO error and reliance on air carriers to properly issue boarding passes. CAT is to verify the authenticity of identification documents presented at the screening checkpoint, confirm the passenger’s reservation, and provide the Secure Flight screening result for that traveler. TDCs would no longer need to examine passengers’ boarding passes to identify those who should receive enhanced screening, which could reduce the potential for error. In April 2014, TSA awarded a contract for the CAT technology solution.

TSA has faced long-standing challenges in acquiring CAT technology. In May 2009, we found that TSA had begun working to address the vulnerability posed by airline passengers providing fraudulent information when making a flight reservation to avoid detection. TSA has issued four previous requests for proposals for CAT/BPSS technology, two of which resulted in no vendors meeting minimum requirements. In 2012, TSA piloted a joint CAT/BPSS technology from three vendors at a cost of $4.4 million. According to TSA’s final report on the pilot, TSA decided not to move forward with these systems because of significant operability and performance difficulties. None of the units tested met TSA’s throughput requirements, creating delays at the screening checkpoint. According to TSA officials, after the joint CAT/BPSS pilot failed, TSA decided to separate CAT technology from BPSS technology and procure each separately.

TSA has also faced challenges in estimating the costs associated with the CAT system. In June 2012, we reported that we could not evaluate the credibility of TSA’s life-cycle cost estimate for CAT/BPSS because it did not include an independent cost estimate or an assessment of how changing key assumptions and other factors would affect the estimate. At that time, according to the life-cycle cost estimate for the Passenger Screening Program, of which CAT/BPSS is a part, the estimated 20-year life-cycle cost of CAT/BPSS was approximately $130 million based on a procurement of 4,000 units. As of April 2014, TSA had not approved a new life-cycle cost estimate for the CAT program, so we were unable to evaluate the extent to which TSA has addressed these challenges in its new estimate.

TSA Lacks Key Information to Determine whether the Secure Flight Program Is Achieving Its Goals

Secure Flight Measures Do Not Fully Assess Progress toward Goals

Secure Flight has six program goals that are relevant to the results of screening performed by the Secure Flight computer system and the program analysts who review computer-generated matches, including the following: goal 1: prevent individuals on the No Fly List from boarding an aircraft, goal 2: identify individuals on the Selectee List for enhanced goal 3: support TSA’s risk-based security mission by identifying high- risk passengers for appropriate security measures/actions and identifying low-risk passengers for expedited screening, goal 4: minimize misidentification of individuals as potential threats to goal 5: incorporate additional risk-based security capabilities to streamline processes and accommodate additional aviation populations, and goal 6: protect passengers’ personal information from unauthorized use and disclosure.

To assess progress with respect to these goals, the program has nine performance measures that it reports on externally (see app. III for the nine Secure Flight performance measures and performance results for fiscal years 2012 and 2013). In addition, Secure Flight has measures for a number of other program activities that it reports internally to program managers to keep them apprised of program performance with respect to the goals (such as the number of confirmed matches identified to the No Fly and Selectee Lists).

However, Secure Flight’s performance measures do not fully assess progress toward achieving its six program goals. For goals 1 through 4 and goal 6, we found that while TSA measured some aspects of performance related to these goals, it did not measure aspects of performance necessary to determine overall progress toward the goals. In addition, for goal 5, we could not identify any program measures that represented the type of performance required to make progress toward achieving the goal, in part because the goal itself did not specify how performance toward the goal should be measured. GPRA establishes a framework for strategic planning and performance measurement in the federal government. Part of that framework involves agencies establishing quantifiable performance measures to demonstrate how they intend to achieve their program goals and measure the extent to which they have done so. These measures should adequately indicate progress toward performance goals so that agencies can compare their programs’ actual results with desired results.

Our prior body of work has shown that measures adequately indicate progress toward performance goals when they represent the important dimensions of their performance goals and reflect the core functions of their related programs or activities. Further, when performance goals are not self-measuring, performance measures should translate those goals into concrete conditions that determine what data to collect in order to learn whether the program has made progress in achieving its goal.

Measures Addressing Accuracy (Goals 1 through 4)

With respect to the program’s first four goals, which address the Secure Flight system’s ability to accurately identify passengers on various watchlists for high- and low-risk screening, the program does not measure all aspects of performance that are essential to achieving these goals. To measure performance toward the first three goals, Secure Flight collects various types of data, including the number of passengers TSA identifies as matches to high- and low-risk lists (including the No Fly, Selectee, Expanded Selectee, rules-based, and TSA PreTM lists). However, Secure Flight has no measures to address the extent to which Secure Flight is missing passengers who are actual matches to these lists (see table 1).

TSA Secure Flight officials stated that measuring the extent to which the Secure Flight system may miss passengers on high-risk lists is difficult to perform in real time. However, our prior work and current program documentation show that the Secure Flight program has used proxy methods to assess the extent to which the system is missing passengers on watchlists. For example, we reported in May 2009 that when the Secure Flight system was under development, TSA conducted a series of tests—using a simulated passenger list and a simulated watchlist created by a TSA contractor with expertise in watchlist matching—to measure the extent to which Secure Flight did not identify all simulated watchlist records. In addition, for this review, we examined meeting minutes of the Secure Flight Match Review Board—a multidepartmental board that reviews system performance and recommends changes—for the period May 2010 through August 2013, to determine how the board assesses system performance. The minutes show that Secure Flight, when contemplating a change in the system’s search capabilities, measures the impacts of proposed changes on system performance, including the extent to which the changes result in failures to identify watchlisted individuals. To make these assessments, Secure Flight rematches historical passenger data and watchlist data under proposed system changes, and compares the results with prior Secure Flight screening outcomes to determine whether any previously identified individuals on high-risk lists were missed. While helpful for Match Review Board deliberations, the testing reflected in meeting minutes was performed on an ad hoc basis and therefore is not a substitute for ongoing performance measurement.

In addition, with respect to low-risk lists, TSA could measure the extent to which the Secure Flight system correctly identifies passengers submitting valid known traveler numbers (i.e., an actual number on a TSA PreTM list) and designates them for expedited screening. TSA officials have stated that variations in the way passengers enter information when making a reservation with a valid known traveler number can cause the system to fail to identify them as TSA PreTM eligible. For example, TSA Match Review Board documentation from December 2012 identified that the Secure Flight system had failed to identify participants on one TSA PreTM list because they used honorific titles (e.g., the Honorable and Senator) when making reservations, and, as a result, they were not eligible for expedited screening.TSA has a process in place to review and resolve inquiries from passengers who believe they should have received TSA PreTM but did not during a recent travel experience.helpful for addressing some TSA PreTM-related problems, the process does not provide information on the extent to which TSA is correctly identifying passengers on low-risk lists, because some passengers may not report problems.

TSA’s fourth goal (to minimize the number of passengers misidentified as threats on high-risk lists) also addresses system accuracy. The program’s related performance measure, its false positive rate, accounts for the number of passengers who have been misidentified as matches to some, but not all, high-risk lists and, thus does not fully assess performance toward the related goal (as shown above, in table 1). TSA’s false positive rate does not account for all misidentifications, because, under the current Secure Flight process, TSA has information on passengers misidentified to the No Fly and Selectee Lists, but does not have information on passengers misidentified to the Expanded Selectee or rules-based lists. TSA is currently implementing changes that will allow it to collect more information about passengers misidentified to other high-risk lists.positive measure, would allow TSA to more fully assess the program’s ability to minimize the misidentification of individuals as potential threats to aviation security.

Measures Addressing Risk- Based Security Capabilities (Goal 5)

TSA does not have any measures that clearly address its goal of incorporating additional risk-based security capabilities to streamline processes and accommodate additional aviation populations (goal 5). According to TSA officials, the goal addresses the program’s ability to adapt the Secure Flight system for risk-based screening initiatives, such as TSA PreTM and similar efforts that allow TSA to distinguish high-risk from low-risk passengers. TSA officials identified several measures that address this goal, including program measures for responding to a change in the national threat level, the system false positive rate, and the system availability measure. identified clearly relate to the goal of adapting the Secure Flight system for different risk-based screening activities, or specify what data should be collected to measure progress toward the goal.

Measures Addressing Privacy (Goal 6)

The Secure Flight system availability measure—a Key Performance Parameter and an OMB 300 Measure—tracks the total amount of time the Secure Flight system (within Secure Flight bounds) is available for matching activities. Secure Flight’s false positive measure was discussed previously. All Secure Flight measures are defined in app. III. implemented.scoring activities from the Secure Flight system, TSA ensures that passenger data do not remain in the system and thus will not be subject to unauthorized use or disclosure. Nevertheless, the measure does not assess other points in time in which the records could be subject to unauthorized use or disclosure, such as before the records are purged or when other government agencies request the results of Secure Flight screening for various purposes, such as an ongoing investigation. When the Secure Flight program was in development, TSA included among a list of possible measures for the fully implemented program a measure for privacy incident compliance (i.e., percentage of privacy incidents reported in compliance with DHS Privacy Incident Handling Guidance). According to TSA officials, since then, TSA has determined that such a measure is not needed because privacy incidents are tracked and publicly reported on at the department level. Nevertheless, additional measures, such as the percentage of government agencies’ requests for Secure Flight data that are handled consistently with program privacy requirements, would allow Secure Flight to determine the extent to which the program is appropriately handling passenger information before it is purged from the system.

By purging the results of Secure Flight matching and Secure Flight’s performance measures provide program managers some information on its progress with respect to its accuracy-related and privacy-related goals (goals 1 through 4 and 6), but do not measure all aspects of performance critical to achieving these goals. In addition, the measures do not provide information on progress toward the program’s risk-based security capabilities goal (goal 5). Additional measures that address key performance aspects related to program goals, and that clearly identify the activities necessary to achieve goals, would allow the program to more fully assess progress toward its goals. For example, the extent to which the Secure Flight system is missing individuals on the No Fly, Selectee, and other high- and low-risk lists is an important dimension of performance related to each of the accuracy-related goals and speaks to a core function of the Secure Flight program—namely to accurately identify passengers on lists. Without measures that provide a more complete understanding of Secure Flight’s performance, TSA cannot compare actual with desired results to understand how well the system is achieving these goals. Similarly, without a measure that reflects misidentifications to all high-risk lists, TSA cannot appropriately gauge its performance with respect to its goal of limiting such misidentifications. Likewise, with respect to its privacy-related goal, additional measures that address other key points in the Secure Flight process in which passenger records could be inappropriately accessed would allow Secure Flight to more fully assess the extent to which it is meeting its goal of protecting passenger information. Finally, establishing measures that clearly represent the performance necessary to achieve the program’s goal that addresses risk-based security capabilities (goal 5) will allow Secure Flight to determine the extent to which it is meeting its goal of adapting the Secure Flight system for different risk-based screening activities.

TSA Does Not Have Timely and Reliable Information on the Secure Flight System’s Matching Errors

TSA does not have timely and reliable information on past Secure Flight system matching errors. As previously discussed, preventing individuals on the No Fly List from boarding an aircraft and identifying individuals on the Selectee List for enhanced screening are key goals of the Secure Flight program. Standards for Internal Control in the Federal Government states that agencies must have relevant, reliable, and timely information to determine whether their operations are performing as expected, and that such information can assist agencies in taking any necessary corrective actions to achieve relevant goals. According to TSA officials, when TSA receives information related to matching errors of the Secure Flight system (i.e., the computerized matching and manual reviews conducted to identify matches of passenger and watchlist data), the Match Review Board reviews this information to determine if any actions could be taken to prevent similar errors from happening again. We reviewed meeting minutes and associated documentation for the 51 Match Review Board meetings held from March 2010 through August 2013, and found 16 meetings in which the Match Review Board discussed system matching errors; investigated possible actions to address these errors; and, when possible, implemented changes to strengthen system performance.

However, when we asked TSA for complete information on the extent and causes of system matching errors, we found that TSA does not have readily available or complete information. It took TSA over 6 months to compile a list of such errors, a process that, according to TSA officials, required a significant amount of manual investigation and review. Further, we found that the list was not complete because it did not reflect all system errors that were discussed at the Match Review Board meetings.discussion of a system error that was not included in the Match Review Board documentation. We also found that, for many incidents on the list, TSA’s description of the cause of the error was not sufficiently detailed to understand whether the Secure Flight system was at fault.

Information on the time frames of our request and the number of system matching errors TSA identified is considered sensitive information and cannot be included in a public report. all potential causes of these errors, and identify and implement sufficient corrective actions.

Conclusions

The Secure Flight program is one of TSA’s key tools for defending civil aviation against terrorist threats. Since TSA began implementing the program, in January 2009, Secure Flight has expanded from a system that matches airline passengers against watchlists of known or suspected terrorists to a system that uses additional high-risk lists and conducts risk- based screening assessments of passengers. Specifically, through the use of new high-risk screening lists, the program now identifies a broader range of high-risk travelers—including ones who may not be on lists of known and suspected terrorists but who nevertheless correspond with known threat criteria. TSA has also begun using Secure Flight to identify low-risk passengers eligible for expedited screening through TSA PreTM. Given Secure Flight’s importance to securing civil aviation and achieving TSA’s risk-based screening goals, the extent to which passengers are being accurately identified by the system (including computerized matching and manual reviews) for standard, expedited, and enhanced screening is critically important. More broadly, to fully realize the security benefits of the Secure Flight program, it is critical that TSA checkpoint personnel correctly identify and appropriately screen travelers according to Secure Flight determinations. Better information on both system and checkpoint performance, therefore, would provide TSA with greater assurance that Secure Flight is achieving its desired purpose to correctly identify passengers for standard, expedited, and enhanced checkpoint screening.

Specifically, TSA would have better assurance that all passengers are screened in accordance with their Secure Flight risk determinations by investigating checkpoint errors and taking appropriate corrective action. Evaluating the root causes of screening errors across all airport checkpoints would provide TSA more complete information on such cases and serve as the basis for policies to ensure the checkpoint is correctly processing passengers. In addition, implementing corrective measures to address the root causes that TSA identifies through its evaluation process would help strengthen checkpoint operations. Furthermore, establishing measures that cover all activities necessary to achieve Secure Flight program goals would allow TSA to more fully assess progress toward these goals. Finally, when TSA learns of Secure Flight system matching errors, a mechanism to systematically document the number and causes of these errors would help ensure that TSA had timely and reliable information to take any corrective action to strengthen system performance.

Recommendations for Executive Action

We recommend that the Transportation Security Administration’s Administrator take the following four actions: to further improve the implementation of Secure Flight risk determinations at the screening checkpoint, develop a process for regularly evaluating the root causes of screening errors across airports so that corrective measures can be identified; to address the root causes of screening errors at the checkpoint, thereby strengthening checkpoint operations, implement the corrective measures TSA identifies through a root cause evaluation process; to assess the progress of the Secure Flight program toward achieving its goals, develop additional measures to address key performance aspects related to each program goal, and ensure these measures clearly identify the activities necessary to achieve progress toward the goal; and to provide Secure Flight program managers with timely and reliable information on cases in which TSA learns of Secure Flight system matching errors, develop a mechanism to systematically document the number and causes of such cases, for the purpose of improving program performance.

Agency Comments and Our Evaluation

We provided a draft of this report to DHS and the Department of Justice for their review and comment. DHS provided written comments on August 25, 2014, which are summarized below and reproduced in full in appendix IV. DHS concurred with all four of our recommendations and described actions under way or planned to address them. In addition, DHS provided written technical comments, which we incorporated into the report as appropriate.

DHS concurred with our first recommendation, that TSA develop a process for regularly evaluating the root causes of checkpoint screening errors across airports so that corrective measures can be identified. DHS stated that TSA is collecting data on the root causes of checkpoint screening errors in its Security Incident Reporting Tool (SIRT) and that TSA OSO’s Operations Performance Division will develop a process for regularly evaluating the root causes of checkpoint screening errors across airports and identify corrective measures. DHS estimates that this will be completed by September 30, 2014. These actions, if implemented effectively, should address the intent of our recommendation.

Regarding our second recommendation, that TSA implement the corrective measures it identifies through a root cause evaluation process, DHS concurred. DHS stated that TSA OSO’s Operations Performance Division will evaluate the data gathered from airports through SIRT to identify root causes of checkpoint screening errors and on the basis of the root cause, work with the appropriate TSA program office to implement corrective measures. Such actions could help to reduce the likelihood that TSA will fail to appropriately screen passengers at the screening checkpoint.

Additionally, DHS concurred with our third recommendation, that TSA develop additional measures to address key performance aspects related to each program goal and ensure these measures clearly identify the activities necessary to achieve progress toward the goal. DHS stated that TSA's Office of Intelligence and Analysis will evaluate its current Secure Flight performance goals and measures and develop new performance measures as necessary. DHS further stated that TSA will explore the possibility of implementing analyses to measure match effectiveness through the use of test data sets. Such actions could help TSA better monitor the performance of the Secure Flight program.

DHS also concurred with our fourth recommendation, that TSA develop a mechanism to systematically document the number and causes of cases in which TSA learns that the Secure Flight system has made a matching error. DHS stated that TSA's Office of Intelligence and Analysis will develop a more robust process to track all known cases in which the Secure Flight system has made a matching error, and that the Secure Flight Match Review Board will conduct reviews to identify potential system improvement measures on a quarterly basis. TSA plans to implement these efforts by December 31, 2014. These actions, if implemented effectively, should address the intent of our recommendation. We will continue to monitor DHS’s efforts.

The Department of Justice did not have formal comments on our draft report, but provided technical comments, which we incorporated as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretary of Homeland Security, the TSA Administrator, the United States Attorney General, and interested congressional committees as appropriate. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

Should you or your staff have any questions about this report, please contact Jennifer A. Grover at 202-512-7141 or groverj@gao.gov. Key contributors to this report are acknowledged in appendix IV. Key points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report.

Appendix I: Objectives, Scope, and Methodology

This report addresses the following questions: 1. How, if at all, has Secure Flight changed since implementation began in January 2009? 2. To what extent does the Transportation Security Administration (TSA) ensure that Secure Flight screening determinations for passengers are fully implemented at airport security checkpoints? 3. To what extent do TSA’s performance measures appropriately assess progress toward achieving the Secure Flight program goals?

To identify how the Secure Flight program has changed since implementation began, we analyzed TSA documentation related to new agency initiatives involving Secure Flight screening since January 2009, including the Secure Flight program concept of operations, privacy notices TSA issued from 2008 (in preparation to begin program implementation) through 2013, and TSA memorandums describing the rationale for new agency initiatives involving the Secure Flight system. We also submitted questions on how Secure Flight has changed to TSA’s Office of Chief Counsel and reviewed its responses. To clarify our understanding of a new agency initiative to identify high-risk passengers not already included in the Terrorist Screening Database (TSDB)—the U.S. government’s consolidated list of known or suspected terrorists—we spoke with relevant officials in the Intelligence and Analysis Division of TSA’s Office of Intelligence and Analysis, who are responsible for the initiative, and with officials from U.S. Customs and Border Protection, who facilitate the generation of one rules-based list. In addition, to understand new initiatives involving Secure Flight screening to identify low-risk travelers, we spoke with Secure Flight program officials and with officials in TSA’s Office of Risk Based Security who oversee TSA PreTM, a 2011 program that allows TSA to designate preapproved passengers as low risk, and TSA Pre✓™ risk assessments, another initiative to identify passengers as low risk for a specific flight.

To determine the extent to which TSA ensures that the Secure Flight vetting results are fully implemented at airport security checkpoints, we analyzed TSA documents governing the screening checkpoint, such as standard operating procedures for checkpoint screening operations and Travel Document Checkers (TDC) and reviewed reports about the performance of Transportation Security Officers (TSO) at the checkpoint by TSA’s Office of Inspections and GAO. To determine the extent to which TSA made errors at the screening checkpoint, we analyzed certain TSA data on TSO performance at the screening checkpoint from May 2012, when TSA began tracking these data, through February 2014, when we conducted the analysis. We examined documentation about these data and interviewed knowledgeable officials, and determined that the data were sufficiently reliable for our purposes. In addition, to clarify our understanding of TSA’s checkpoint operations and inform our analysis, we interviewed officials within TSA’s Office of Security Operations, which is responsible for checkpoint operations, and TSA officials at nine airports. We selected these nine airports based on a variety of factors, such as volume of passengers screened and geographic dispersion. The results of these interviews cannot be generalized to all airports, but provide insight into TSA’s challenges to correctly identify and screen passengers at checkpoints. To better understand how TSA ensures that all passengers have been appropriately screened by Secure Flight, we visited TSA’s Identity Verification Call Center to interview officials and observe their identity verification procedures. We compared TSA’s checkpoint procedures against Standards for Internal Control in the Federal Government.Finally, to determine the extent to which TSA’s planned technology solutions could address checkpoint errors, we analyzed documents, such as requests for proposals, related to TSA’s planned technology solutions and interviewed knowledgeable TSA officials.

To determine the extent to which Secure Flight performance measures appropriately assess progress toward achieving the program goals, we reviewed documentation of TSA’s program goals and performance measures for fiscal years 2012 and 2013—including the measures Secure Flight reports externally to the Department of Homeland Security and the Office of Management and Budget (OMB), as well as other internal performance measures Secure Flight officials use for program management purposes—and discussed these measures with Secure Flight officials. We assessed these measures against provisions of the Government Performance and Results Act (GPRA) of 1993 and the GPRA Modernization Act of 2010 requiring agencies to compare actual results with performance goals. Although GPRA’s requirements apply at the agency level, in our prior work, we have reported that these requirements can serve as leading practices at lower levels within an organization, such as individual programs or initiatives, through a review of our related products, OMB guidance, and studies by the National Academy of Public Administration and the Urban Institute. We also interviewed relevant TSA officials about the current performance measures for the Secure Flight program and the adequacy of these measures in assessing TSA’s progress in achieving program goals.

In addition, to understand how TSA uses Secure Flight-related performance data, we reviewed documentation related to all meetings that TSA identified of the Secure Flight Match Review Board—a multidepartmental entity established to, among other things, review performance measures and recommend changes to improve system performance—from the time was the board was initiated, in March 2010, through August 2013, a total of 51 meetings. To identify the extent to which TSA monitors and evaluates the reasons for any Secure Flight system matching errors, we analyzed a list of such errors that occurred from November 2010 (the point at which the Secure Flight program was implemented for all covered domestic and foreign air carriers) through July 2013 that TSA compiled at our request. To assess the accuracy and completeness of the list TSA provided, we also checked to see if system matching errors we identified in documentation from the Match Review Board meetings were included in TSA’s list. We evaluated TSA’s efforts to document system matching errors against standards for information and communications identified in GAO’s Standards for Internal Control in the Federal Government.

We conducted this performance audit from March 2013 to September 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions.

Appendix II: Secure Flight Screening Lists and Activities

In January 2009, the Transportation Security Administration (TSA) began implementing the Secure Flight program to facilitate the identification of high-risk passengers who may pose security risks to civil aviation, and designate them for additional screening at airport checkpoints. Since then, TSA has begun using Secure Flight to identify low-risk passengers eligible for more efficient processing at the checkpoint. This appendix presents an overview of the lists and other activities as of July 2013 that Secure Flight uses to identify passengers as high risk or low risk. The Secure Flight program, as implemented pursuant to the 2008 Secure Flight Final Rule, requires commercial aircraft operators traveling to, from, within, or overflying the United States to collect information from passengers and transmit that information electronically to TSA. The Secure Flight system uses this information to screen passengers by conducting computerized matching against government lists and other risk assessment activities. As a result of this screening, passengers identified as high risk receive enhanced screening, which includes, in addition to the procedures applied during a typical standard screening experience, a pat-down and either an explosive trace detection search involving a device certified by TSA to detect explosive particles or a physical search of the interior of the passenger’s accessible property, electronics, and footwear. Those passengers Secure Flight identifies as low risk are eligible to receive expedited screening, which unlike standard screening, affords travelers certain conveniences, such as not having to remove their belts, shoes, or light outerwear when screened.

Figure 3 provides information on the lists Secure Flight uses to identify high-risk passengers. Figure 4 describes Secure Flight’s activities to identify low-risk passengers, including screening against lists associated with the TSA PreTM Program, a 2011 initiative that allows TSA to designate preapproved passengers as low risk, and TSA Pre✓™ risk assessments, which assess passengers’ risk using data submitted to Secure Flight for screening. Figure 5 describes two additional lists Secure Flight uses for passenger screening that, depending on the list, exempt passengers from being designated as low or high risk.

Appendix III: Secure Flight Performance Data for Fiscal Years 2012 and 2013

This appendix presents data on the Secure Flight program’s performance measures and associated performance results that the Transportation Security Administration (TSA) reported externally to the Department of Homeland Security (DHS) and the Office of Management and Budget (OMB). Specifically, table 2 displays data on six Secure Flight Key Performance Parameters—key system capabilities that must be met in order for a system to meet its operational goals—that TSA management reported to DHS for fiscal years 2012 and 2013. Table 3 displays data on five Secure Flight program measures that TSA management reported to OMB for fiscal years 2012 and 2013. The OMB measures are part of the program’s yearly exhibit 300, also called the Capital Asset Plan and Business Case, a document that agencies submit to OMB to justify resource requests for major information technology investments. TSA reports performance data for all these measures on a monthly basis, and for each measure, we have provided the range of the performance measurement results for each fiscal year.

Overview of the Secure Flight Screening Process

The Secure Flight program, as implemented pursuant to the 2008 Secure Flight Final Rule, requires commercial aircraft operators traveling to, from, or overflying the United States to collect information from passengers and transmit that information electronically to TSA. This information, known collectively as Secure Flight Passenger Data (SFPD), includes personally identifiable information, such as full name, gender, date of birth, passport information (if available), and certain nonpersonally identifiable information, such as itinerary information and the unique number associated with a travel record (record number locator).

The Secure Flight program designates passengers for risk-appropriate screening by matching SFPD against various lists composed of individuals who should be identified, for the purpose of checkpoint screening, as either high risk or low risk. With respect to matching passengers against lists, the Secure Flight computer system first conducts automated matching of passenger and watchlist data to identify a pool of passengers who are potential matches to various lists. Next, the system compares all potential matches against the TSA Cleared List, a list of individuals who have applied to, and been cleared through, the DHS redress process. Passengers included on the TSA Cleared List submit a redress number when making a reservation, which allows the Secure Flight system to recognize and clear them.performs automated matching, Secure Flight analysts conduct manual reviews of potential matches to further rule out individuals who are not included on the No Fly and Selectee Lists.

After the completion of manual reviews, TSA precludes passengers who remain potential matches to certain lists from receiving their boarding passes. These passengers, for whom air carriers receive a “passenger inhibited” message from Secure Flight, must undergo a resolution process at the airport. This process may involve air carriers sending updated passenger information back to Secure Flight for automated rematching or placing a call to Secure Flight for assistance in resolving the match. At the conclusion of automated and manual screening processes, Secure Flight provides air carriers with a final screening determination for each passenger. At airport checkpoints, those passengers identified as high risk receive enhanced screening and those identified as low risk are eligible for expedited screening.

Appendix IV: Comments from the Department of Homeland Security

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Maria Strudwick (Assistant Director), Mona Nichols Blake (Analyst-in-Charge), John de Ferrari, Michele Fejfar, Imoni Hampton, Eric Hauswirth, Susan Hsu, Richard Hung, Justine Lazaro, Benjamin Licht, Tom Lombardi, Linda Miller, David Plocher, and Ashley Vaughan made key contributions to this report.