Aviation Security: Efforts to Validate TSA's Passenger Screening Behavior Detection Program Underway, but Opportunities Exist to Strengthen Validation and Address Operational Challenges
Released: 2010-05-20T00:00:00

Letter

Background

The Aviation and Transportation Security Act established TSA as the federal agency with primary responsibility for securing the nation’s civil aviation system, which includes the screening of all passenger and property transported by commercial passenger aircraft. TSA currently has direct responsibility for, or oversees the performance of, security operations at approximately 457 TSA-regulated airports in the United States implementing security requirements in accordance with TSA- approved security programs and other TSA direction. At TSA-regulated airports, prior to boarding an aircraft, all passengers, their accessible property, and their checked baggage are screened pursuant to TSA- established procedures, which include, for example, passengers passing through security checkpoints where they and their identification documents are checked by TSOs and Travel Document Checkers, or by Screening Partnership Program employees.

TSA uses multiple layers of security to deter, detect, and disrupt persons posing a potential risk to aviation security. These layers include three principal types of screening employees at airport checkpoints—Travel Document Checkers, who examine tickets, passports, and other forms of identification; TSOs, who examine property, including checked baggage, and persons using x-ray equipment and magnetometers, as well as other devices; and BDOs, using SPOT to assess passenger behaviors and appearance. BDOs are the only type of TSA screening employees not deployed to all TSA-regulated airports and all checkpoints within the airports where it is deployed on a regular basis. TSA deployed SPOT as an added layer of security to help deter terrorists attempting to exploit TSA’s focus on prohibited items and other potential security weaknesses. Other security layers cited by TSA include intelligence gathering and analysis; passenger prescreening; random canine team searches at airports; federal air marshals; reinforced cockpit doors; federal flight deck officers; the passengers themselves; as well as other measures both visible and invisible to the public. Figure 1 shows TSA’s 20 aviation security layers.

The grey area in figure 1 highlights four layers that apply to passengers and their property as they seek to board an aircraft. Airport LEOs, another layer of security cited by TSA, do not report to TSA and may not maintain a physical presence at smaller TSA-regulated airports. According to TSA, each one of these layers alone is capable of stopping a terrorist attack. In combination, TSA states that their security value is multiplied, creating a much stronger system, and that a terrorist who has to overcome multiple security layers in order to carry out an attack is more likely to be preempted, deterred, or to fail during the attempt.

SPOT Uses Behavior Detection Techniques to Assess Passenger Behaviors and Appearances

The SPOT program utilizes behavior observation and analysis techniques to identify potentially high-risk passengers. Individuals who exhibit suspicious behaviors, including both physical and appearance indicators, may be required to undergo additional screening. Field agents and law enforcement officers of other federal agencies and entities—such as the FBI, the Secret Service, CBP, and FAMS—utilize elements of behavior detection analysis as a part of their work. In addition, some foreign entities, such as Israel’s El Al airlines, use behavior detection and analysis techniques as part of their security efforts. However, TSA emphasized to us that the SPOT program is unique among these entities because it uses a point system to help identify suspicious persons on the basis of their behavior and appearance and because behavior detection and analysis are the central focus of SPOT. Officials from the other agencies stated that their field personnel incorporate behavior detection as one of many skills used in their work; in contrast, behavior detection is the primary element of the BDOs’ work.

SPOT trains BDOs to look for and recognize facial expressions, body language, and appearance that indicate the possibility that an individual is engaged in some form of deception and fears discovery. These behaviors and appearances are listed on a SPOT score sheet used in SPOT training.

Passenger behavior and appearance are to be compared by the BDOs— who typically work in two-person teams. BDOs are expected to “walk the line”—that is, to initiate casual conversations with passengers waiting in line, particularly if their observations led them to question someone exhibiting behaviors or appearances on the SPOT checklist. As the BDOs walk the line, and the passenger with SPOT indicators is reached, a casual conversation is used to determine if there is a basis for observed behaviors or appearances on the checklist. In most instances, these conversations provide information to the BDOs that permits them to consider the issue resolved, and hence not a security concern. Figure 2 below illustrates the first step of the three-step SPOT process, the BDO-passenger interaction at a checkpoint prior to the passenger passing through a magnetometer.

As shown in figure 2, passenger behavior and appearance are observed by the BDOs as passengers wait in line for screening at a security checkpoint. Even if the checkpoint is busy, the BDOs must attempt to visually scan all the passengers waiting in line, as well as persons near the checkpoint, to determine if any are showing behaviors or appearances on the SPOT checklist. According to TSA, on average a BDO has approximately 30 seconds to assess each passenger while the passenger waits in line. For passengers exhibiting indicators above baseline conditions, the BDOs are to (mentally) add up the points assigned to each indicator they observe. Both BDO team members must agree that observed indicators have exceeded the predetermined numerical threshold, although they do not have to identify the same indicators the passenger exhibited. In instances when a passenger’s SPOT indicators place them above the numerical threshold, and the passenger has placed their property on the conveyor belt for x-raying, and has walked through the magnetometer or equivalent screening device for passengers, he or she will be directed to the second step of SPOT, referral screening. This involves additional questioning and physical search of their person and property by BDOs and TSOs. This referral screening occurs in the checkpoint area.

If the passenger’s behavior escalates further—accumulating more points based on the SPOT checklist—the BDOs are to refer the passenger to a LEO. A referral to a LEO is a potential third step in the SPOT process. BDOs are not LEOs—they do not conduct criminal investigations, carry weapons, or make arrests.

After a passenger has been referred by the BDOs to a LEO, the LEO is then expected to independently determine, through additional investigation, such as questioning the passenger and, if appropriate, by conducting an identity verification and background check through the FBI’s National Crime Information Center (NCIC), whether sufficient grounds exist to take further action, such as detaining or arresting the passenger. TSA officials who are LEOs also have access to NCIC, such as an airport’s Assistant Federal Security Director for Law Enforcement or federal air marshals. NCIC is the FBI’s computerized index of criminal justice information (i.e., criminal record history information, fugitives, stolen properties, and missing persons), available to federal, state, and local law enforcement and other criminal justice agencies at all times. Similarly, other federal LEOs also have such access, including CBP, and Drug Enforcement Agency (DEA) personnel. However, since both local and federal LEOs have other responsibilities, and may not be present at each operating checkpoint, BDOs may have to seek them out to request an NCIC check. According to TSA, aside from requiring that an airport maintain a law enforcement presence, it exercises no jurisdiction over the law enforcement activities of non-TSA officers or entities at an airport; thus, it cannot require LEOs to conduct an NCIC check or to provide BDOs with information about the ultimate disposition of cases referred by them to LEOs.

Once the LEO concludes his or her investigation and determines whether the passenger will be arrested or detained, TSA officials are to evaluate the security concerns to determine whether to allow the passenger to proceed to the boarding gate. (In some instances, a LEO might choose not to arrest or detain a passenger; TSA would then decide whether the infraction was sufficiently serious to necessitate barring the passenger from boarding.) After a referral incident has been resolved, BDOs are to enter information about the incident into TSA’s SPOT referral database. The data entered are to include time, date, location of the incident, behaviors witnessed, prohibited items found (if any), and information on the LEO’s response (if applicable), such as whether the LEO questioned the passenger, arrested the individual, or released the passenger. The SPOT referral database contains no personal identifying information about passengers.

SPOT Has Been Deployed in Phases

The SPOT program began with pilot tests in 2003 and 2004 at several New England airports, in which TSA began using uniformed BDOs at airport checkpoints. After some initial pilot projects and test deployments, 644 BDOs were deployed to 42 airports in the first phase of the program from November 2006 through June 2007. As of March 2010, about 3,000 BDOs utilizing SPOT were deployed at 161 of 457 TSA-regulated airports.

BDO eligibility is restricted to TSOs with at least 12 months of TSO experience, or others with related security experience. Applicants must apply and be accepted into the BDO training program. The training includes 4 days of classroom courses, followed by 3 days of on-the-job training. BDOs must memorize all of the behaviors and appearances on the SPOT checklist, as well as the point value assigned to each, in order to be able to add these up to determine if a passenger should be sent to SPOT referral screening. BDO applicants must also pass a job knowledge test at the conclusion of the training. The test includes related multiple choice questions, true or false statements, and case-based scenarios.

DHS Is Taking Action to Validate the Scientific Basis of TSA’s SPOT Program but Opportunities Exist to Help Inform Future Program Decisions

Although DHS is in the process of validating the way in which the SPOT program utilizes the science of behavior detection in an airport environment, TSA deployed SPOT nationwide before first determining whether there was a scientifically valid basis for using behavior and appearance indicators as a means for reliably identifying passengers as potential threats in airports. TSA reported that it deployed SPOT before a scientific validation of the program was completed in response to the need to address potential threats to the aviation system that would not necessarily be detected by existing layers of aviation security. TSA stated that no other large-scale U.S. or international screening program incorporating behavior- and appearance-based indicators has ever been rigorously scientifically validated. While TSA deployed SPOT on the basis of some risk-related factors, such as threat information and airport passenger volume, it did not use a comprehensive risk assessment to guide its strategy of selectively deploying SPOT to 161 of the nation’s 457 TSA- regulated airports. TSA also expanded the SPOT program over the last 3 years without the benefit of a cost-benefit analysis of SPOT. Additionally, TSA’s strategic plan for SPOT could be improved by the inclusion of desirable characteristics identified in our prior work, such as risk assessment information, cost and resources analysis, and a means for collaboration with other key entities.

TSA Is in the Process of Validating the Scientific Basis Used to Identify Passengers with SPOT Behaviors

TSA proceeded with deploying SPOT on a nationwide basis before determining whether the list of passenger behaviors and appearances underpinning the SPOT program were scientifically validated, and whether these techniques could be applied for counterterrorism purposes in an airport environment. In 2008, a report issued by the National Research Council of the National Academy of Sciences noted that behavior and appearances monitoring might be able to play a useful role in counterterrorism efforts but stated that a scientific consensus does not exist regarding whether any behavioral surveillance or physiological monitoring techniques are ready for use in the counterterrorist context given the present state of the science. The report also stated that the scientific evidence for behavioral monitoring is preliminary in nature. According to the report, an information-based program, such as a behavior detection program, should first determine if a scientific foundation exists and use scientifically valid criteria to evaluate its effectiveness before going forward. The report added that programs should have a sound experimental basis and documentation on the program’s effectiveness should be reviewed by an independent entity capable of evaluating the supporting scientific evidence. The report also stated that often scientists and other experts can help independently assess the scientific evidence on the effectiveness of a program. A contributor to the National Research Council report also stated that no conclusive research has been conducted to determine if behavior detection can be reliably used on a larger scale, such as in an airport setting, to identify persons intending to cause harm to the aviation system.

While TSA and DHS’s Science and Technology (S&T) Directorate officials agreed that SPOT was deployed before its scientific underpinnings were fully validated, they stated that no large-scale U.S. or international operational screening program incorporating behavior- and appearance- based indicators has been rigorously scientifically validated. These officials also questioned the findings of the National Research Council report and stated that the study lacked sufficient information for its conclusions because it did not consider recent findings from unpublished DHS, defense, and intelligence community studies. However, National Research Council officials stated that an agency should be cautious about relying on the results of unpublished research that has not been peer reviewed, such as that generated by DHS and the defense and intelligence community, and using unpublished work as a basis for proceeding with a process, method, or program. Moreover, we have previously reported that peer review is widely accepted as an important quality control mechanism that helps prevent the dissemination of potentially erroneous information.

In addition to the unpublished research, TSA told us that the SPOT program was based on operational best practices from law enforcement, defense, and the intelligence communities. According to TSA officials, the agency based its choice of SPOT behavior, appearance, and deception indicators on existing research and training programs. For example, TSA cited research on emotions and their behavior indicators by Dr. Paul Ekman, interviewing and interrogation by Stan Walters, and nonverbal indicators by Dr. David Givens and Dr. Mark Frank as support for the choice of several of the behavior indicators. According to TSA, its development of the SPOT program was based on related DHS research and information from the training curricula of other federal agencies, such as the Federal Transit Administration and the Bureau of Alcohol, Tobacco, Firearms, and Explosives.

As with the SPOT behavior indicators, TSA told us that it sought input in creating the SPOT point scoring system from subject matter experts and from participants in TSA’s SPOT working group, which consisted of law enforcement officials from agencies such as FBI, DEA, and local law enforcement officials. While TSA officials said that they coordinated with relevant subject matter experts, such as Dr. Ekman, and based the SPOT scoring system on existing research and training programs, no validation of the behavior, appearance, and deception indicators was conducted prior to the deployment of SPOT in November 2006. According to TSA officials, they used professional judgment in developing the SPOT point system and stated that the purpose of developing the scoring system was to increase the objectivity of the SPOT process.

Dr. Ekman stated that, in his opinion, and after reviewing the scoring system and observing the program in operation, it was not clear whether the SPOT behaviors and appearances, and the related point system, could be used effectively in an airport environment because no credible validation research on this issue had been conducted. He noted, for example, that research is needed to identify how many BDOs are required to observe a given number of passengers moving at a given rate per day in an airport environment, or the length of time that such observation can be conducted before observation fatigue affects the effectiveness of the personnel. He commented that observation fatigue is a well-known phenomenon among workers whose work involves intense observation, and that it is essential to determine the duration of effective observation and to ensure consistency and reliability among the personnel carrying out the observations.

DHS has recognized the need to conduct additional research to scientifically validate the use of the SPOT behavioral indicators in an airport environment. DHS’s S&T Directorate began research in 2007 to determine if there is a statistically significant correlation between the SPOT behaviors exhibited by airport passengers and finding airport passengers with prohibited items (such as weapons), false documents, and illegal drugs or who pose a potential risk to aviation security. According to S&T, this research is expected to be completed in fiscal year 2011 and is to include three key elements. First, the study’s purpose is to assess the reliability of the SPOT program by analyzing TSA’s SPOT database to determine patterns of BDO scoring to measure consistency across BDOs, teams, locations, and other variables. Second, the study aims to compare the current implementation of SPOT to random passenger screening. Specifically, according to S&T officials, 130,000 passengers are to be randomly selected for additional SPOT referral screening. The study’s design states that data collected about these passengers will be compared to data for passengers screened through the normal SPOT process. S&T officials expect that the results of this element of the study will provide a better understanding of how SPOT compares to random selection, as well as providing a baseline of each indicator present in the traveling public. Third, the study also aims to utilize live and video data, as available, to measure SPOT score ratings by BDOs of behaviors exhibited by passengers against ratings of the same passengers by subject matter experts. This element of the study could help determine whether BDOs are using, or are continuing to use, the SPOT score sheet correctly as time passes after their initial training. According to S&T officials, the study is to form the basis for BDO performance and training requirements.

The S&T Directorate reported some preliminary findings associated with this research in February 2008. The Directorate reported that although some of the existing literature supported the possibility of using behavioral and physiological cues, the results are not methodologically strong enough to support standardized applications in an operational setting. The preliminary findings also noted that it is not known whether behavioral and physiological cues linked to deception in planning a hostile action will be the same or different as those indicators linked to deception by an individual after they have already engaged in a hostile action. However, an S&T program director stated that although early literature can be characterized as methodologically weak, more recent unpublished research sponsored by DHS, the Department of Defense, and the intelligence community is promising in that it has demonstrated some linkages between behavioral and physiological indicators and deception.

In March 2009, the Under Secretary (Acting) for DHS’s S&T Directorate testified that the Directorate had performed an initial validation of the behavior indicators used by BDOs. The Under Secretary stated that this analysis provided statistically significant support that persons demonstrating select behavioral indicators are more likely to possess prohibited items and that behaviors can distinguish deceptive from nondeceptive individuals. According to S&T, this validation was the result of statistical analyses performed by S&T using operational data from the SPOT program database. However, we identified weaknesses in TSA’s process for maintaining these data. For example, controls over the SPOT database to help ensure the completeness and accuracy of the data were missing. Specifically, the SPOT database did not have computerized edit checks built into the system to review the format, existence, and reasonableness of data. For example, we found that discrepancies existed between the number of passengers arrested by local law enforcement at the screening checkpoints and the number of screened passengers recorded as arrested. In another example, we found that the total number of LEO referrals differed from the number of passenger records with information on the reasons for LEO referral. Internal control standards state that controls should be installed at an application’s interfaces with other systems to ensure that all inputs are received and are valid and that outputs are correct and properly distributed. TSA officials explained these issues as data anomalies and planned to change instructions to staff entering data to reduce these problems. Although TSA is taking steps to update the SPOT database, which are discussed later in this report, the data used by S&T to conduct its preliminary validation of related behaviors lacked such controls. In addition, BDOs could not input all behaviors observed in the SPOT database because the database limits entry to eight behaviors, six signs of deception, and four types of prohibited items per passenger referred for additional screening. Because of these data-related issues, meaningful analyses could not be conducted to determine if there is an association between certain behaviors and the likelihood that a person displaying certain behaviors would be referred to a LEO or whether any behavior or combination of behaviors could be used to distinguish deceptive from nondeceptive individuals. As a result, TSA lacks assurance that the SPOT data can be used effectively to determine that the person poses a risk to aviation security. S&T has recognized weaknesses in the procedures for collecting data on passengers screened by SPOT and plans to more systematically collect data during its study by, for example, requiring BDOs to record more complete and accurate information related to a passenger referral immediately following resolution.

The S&T study is an important step to determine whether SPOT is more effective at identifying passengers who may be threats to the aviation system than random screening. However, S&T’s current research plan is not designed to fully validate whether behavior detection and appearances can be effectively used to reliably identify individuals in an airport terminal environment who pose a risk to the aviation system. For example, research on other issues, such as determining the number of individuals needed to observe a given number of passengers moving at a given rate per day in an airport environment or the duration that such observation can be conducted by BDOs before observation fatigue affects effectiveness, could provide additional information on the extent to which SPOT can be effectively implemented in airports. In another example, Dr. Ekman told us that additional research could help determine the need for periodic refresher training since no research has yet determined whether behavior detection is easily forgotten or can be potentially degraded with time or lack of use. While S&T officials agree on the need to validate the science of behavior detection programs, they told us that some of these other issues could be examined in the future but are not part of the current plan due to time and budgetary constraints. According to S&T, some additional analysis is underway to inform the current BDO selection process. This analysis is intended to provide information on the knowledge, skills, abilities, and other characteristics of successful BDOs. Since the analysis is scheduled for completion in May 2010, it remains unclear to what extent the findings will help to validate the science related to SPOT. While we recognize the potential benefits of these efforts, we believe that an assessment by an independent panel of experts of the planned methodology of DHS’s study could help DHS assess the costs and benefits associated with a more comprehensive methodology designed to fully validate the science related to SPOT. Our prior work has recommended the use of such independent panels for comprehensive, objective reviews of complex issues. In addition, according to the National Research Council, an independent panel could provide an objective assessment of the methodology and findings of DHS’s study to better ensure that SPOT is based on validated science. Thus, an independent panel of experts could help DHS develop a comprehensive methodology to determine if the SPOT program is based on valid scientific principles that can be effectively applied in an airport environment for counterterrorism purposes.

SPOT Was Deployed Nationwide on Basis of Threat, but Without a Comprehensive Risk Assessment

According to DHS’s National Infrastructure Protection Plan (NIPP), risk assessments are to be documented, reproducible (so that others can verify the results), defensible (technically sound and free of significant errors), and complete. The NIPP states that comprehensive risk assessments are necessary for determining which assets or systems face the highest risk, for prioritizing risk mitigation efforts and the allocation of resources, and for effectively measuring how security programs reduce risks. For a risk assessment to be considered complete, the NIPP states that it must specifically assess threat, vulnerability, and consequence; after these three components have been assessed, they are to be combined to produce a risk estimate.

According to TSA, SPOT was deployed to TSA-regulated airports on the basis of threat information in TSA’s Current Airport Threat Assessment list. TSA deployed SPOT to 161 of 457 TSA-regulated airports. TSA officials told us that this selective deployment creates unpredictability for persons seeking to cause harm to the aviation system because they would not know which airports had BDO teams and because BDOs are occasionally sent out to the smaller airports that do not have BDOs on a permanent basis. Although TSA’s selective deployment of SPOT was based on threat information, TSA did not conduct vulnerability and consequence assessments to inform the deployment of BDOs. As a result, it could not combine the results to conduct a comprehensive risk assessment to inform the deployment of BDOs to those airports with the highest risks.

TSA officials told us that while they have not completed a comprehensive risk assessment for airport security, they have prepared and are currently reviewing a draft of a comprehensive, scenario-based Aviation Modal Risk Assessment—known as the AMRA—which is to serve as a comprehensive risk assessment for aviation security. According to TSA officials, the AMRA is to address all three elements of risk for domestic commercial aviation, general aviation, and air cargo. Although TSA planned to release the AMRA in February 2008, it now expects to finalize the AMRA in 2010. According to TSA, the AMRA may help provide information for the prioritization of BDO deployment within airports, but could not provide specifics on how it would do so. Further, TSA officials noted that information from AMRA would inform BDO deployment in conjunction with other TSA priorities not related to SPOT. Since the AMRA is not yet complete, it is not clear whether it will provide the risk analysis— including assessments of vulnerability and consequence—needed to inform TSA’s decisions and planning for any revisions or future deployment of SPOT. If AMRA lacks information relevant to the deployment of SPOT and further research determines that SPOT has a scientifically validated basis for using behavior detection for counterterrorism purposes in the airport environment, then conducting a comprehensive risk assessment of airports could strengthen TSA’s ability to establish priorities and make cost-effective resource decisions regarding the deployment of BDOs to those airports deemed to have the highest priority risks.

TSA Deployed SPOT Nationwide Without Conducting a Cost-Benefit Analysis but Such an Analysis Could Help Inform Program Decisions Moving Forward

DHS and other federal guidance recommend conducting a cost-benefit analysis before implementing new programs to avoid unnecessary costs and identify the best way to achieve goals at the lowest costs among potential alternatives. Our prior work has also supported the use of cost- benefit analyses during retrospective reviews to validate the agency’s original assumptions regarding costs and benefits. In addition, the DHS February 2006 Cost-Benefit Analysis Guidebook and OMB guidance both recommend the use of cost-benefit analysis, both in the planning stage for a program, and when significant milestones or financial options are to be assessed. The DHS Guidebook states that a cost-benefit analysis is designed to identify optimal financial solutions among competing alternatives. OMB guidance also identifies cost-benefit analysis as one of the key principles to be considered when making capital expenditures, that expected benefits of proposed actions should be explained, and that a baseline should be identified discussing costs and benefits in comparison with clearly defined alternatives. DHS’s 2006 and 2009 NIPPs also state that priority is to be given to those protective measures that provide the greatest mitigation of risk for the resources that are available. The DHS NIPPs add that effective protective programs seek to use resources efficiently by focusing on actions that offer the greatest mitigation of risk for any given expenditure. In addition, measuring cost effectiveness of SPOT was a key TSA goal in an October 2005 version of the SPOT strategic plan.

Although the DHS and OMB guidance recommend that a cost-benefit analysis be conducted prior to deploying a program nationwide—and potentially incurring substantial costs—TSA did not conduct such an analysis of SPOT to inform its pilot testing prior to full-scale nationwide deployment. In early 2003, TSA began conducting a pilot test of the SPOT program at Boston Logan airport to better understand the benefits of the program. According to Boston Logan’s Federal Security Director, the primary purpose of this pilot test was to understand the potential of the program, not to validate its success. TSA officials stated that the program had several benefits, one of which was its “negligible cost.” However, TSA did not analyze the pilot test results to determine if SPOT was more cost effective than other alternatives, such as random screening of passengers. In October 2004, TSA implemented additional pilot programs in Providence, Rhode Island and Portland, Maine with the goal of providing Federal Security Directors with an additional layer of security to identify high-risk passengers for additional screening using behavior detection techniques. TSA concluded that the pilot program was successful and cited several security benefits of these pilots. For example, TSA personnel in Providence identified two individuals in possession of illegal drugs, who were then arrested. Law enforcement also arrested another individual referred to them for providing a fraudulent passport. In another example, BDOs in Portland discovered a passenger with multiple passports and a hidden luggage compartment. The passenger was interviewed by LEOs and later released.

TSA determined that these initial pilot tests at three airports were successful without comparing pilot test data to other possible security alternatives. For example, the results of random screening of passengers at the pilot airports could have provided TSA with objective baseline data. Specifically, these data could have been compared to data collected during the SPOT pilots to determine if SPOT was more effective than random screening in detecting passengers who pose a potential risk to aviation security. TSA concluded that the pilot tests were successful because pilot airports were able to easily incorporate SPOT into their security program, train personnel in SPOT, and implement procedures for an additional layer of security according to TSA.

TSA conducted additional pilot tests at the Minneapolis-St. Paul, Minnesota and Bangor, Maine airports in October 2005. TSA also deployed the program to nine additional airports in response to TSA’s holiday preparedness plan in December 2005 to further operationally test the program. Senior SPOT program officials explained that TSA did not conduct an analysis of the pilot testing because the program was in its infancy and officials were focused on deploying SPOT to additional airports. Since that time, TSA has not conducted a cost-benefit analysis, which could help the agency establish the value of the program relative to other layers of aviation security. Moreover, a cost-benefit analysis could also be useful considering recent program growth. For example, from fiscal year 2007 through fiscal year 2009, TSA allotted about $383 million for SPOT. During this period, SPOT’s share of TSA’s total screening operations budget increased from 1 percent to 5 percent. The conference report accompanying the fiscal year 2010 DHS appropriations act designates $212 million of the appropriated aviation security funding for the SPOT program. A cost-benefit analysis could have provided TSA management with analysis on whether this allocation was a prudent investment, as well as whether this level of investment in SPOT is appropriate. Figure 3 shows the growth in the budget and personnel numbers for SPOT from fiscal years 2007 through 2010.

SPOT’s Strategic Plan Could be Strengthened by Addressing Key Characteristics of a Successful Strategy

Our previous work, and the Government Performance and Results Act, set forth several key elements of a strategic plan. Such plans can guide agencies in planning and implementing an effective government program. Table 1 summarizes the desirable characteristics of an effective strategic plan, as identified in our prior work. In April 2009, we reported that these characteristics are the starting point for developing a strategic plan.

TSA officials at Boston Logan airport told us that they completed the first strategic plan for SPOT in 2006. The strategic plan was last updated in March 2007. The March 2007 plan includes some of the desirable characteristics described above, such as an overall purpose. However, incorporating additional characteristics of an effective strategic plan could enhance the plan’s usefulness in program management and resource allocation decisions to effectively manage the deployment of SPOT if TSA determines that the program has a scientifically valid basis. TSA officials stated that they believed the plan was sufficiently comprehensive to develop a national program, such as SPOT. However, these officials told us that the plan was not updated after TSA expanded the program in 2008 and 2009. They also stated that the program’s focus remained on deploying SPOT to additional airports. Our assessment of the extent to which the SPOT strategic plan addresses these characteristics is presented below.

Purpose, scope, and methodology: The SPOT strategic plan addresses why the plan was developed (i.e., purpose) and the scope of its coverage. Specifically, the plan describes a strategy to utilize behavior detection screening as an additional layer of security. The plan also notes that the primary focus is to expand SPOT in the aviation environment while also developing a capability to deploy BDOs to support security efforts in all modes of transportation. However, the plan does not discuss the process by which it was developed (i.e., methodology). According to TSA, officials responsible for developing the plan received input from relevant stakeholders at Boston Logan airport and TSA headquarters. We believe incorporating the methodology into the plan could make the document more useful to TSA and other organizations, such as local law enforcement, responsible for implementing the plan.

Problem definition and risk assessment: The plan addresses the particular threat it is directed towards. Specifically, the plan describes the need to implement SPOT to counter terrorist activities, improve security, and incorporate additional layers of protection within aviation security. However, the plan does not incorporate risk assessment information to identify priorities or guide program implementation because TSA has not conducted a comprehensive risk assessment related to the deployment of SPOT. Using available risk assessment information to inform the development of a strategic plan would help ensure that clear priorities are established and focused on the areas of greatest need. Specifically, incorporating the results of a risk assessment in the program’s strategic plan could help inform TSA’s decisions such as whether to deploy SPOT to additional TSA-regulated airports, to shift SPOT teams from one airport to another, or to remove SPOT at airports where the benefit of addressing the risk does not outweigh the costs, as well as to identify and communicate the risks to aviation security if SPOT was not deployed to all TSA- regulated airports.

Goals, subordinate objectives, activities, and performance measures: The plan outlines several goals, objectives, and activities for the SPOT program to achieve. For example, the plan outlines a goal to develop multimodal partnerships, including at the local level, to support SPOT. An associated objective for this goal includes identifying and fostering advocates within each mode of transportation by developing transportation, intelligence, and law enforcement working groups with relevant officials to share information and foster cooperation. The plan also includes a goal to develop and implement performance measures for SPOT. However, the plan did not include performance measures for SPOT. Incorporating performance measures into the plan could help TSA officials measure progress in implementing the plan’s goals, objectives, and activities.

Resources, investments, and risk management: The plan does not identify the costs and resources needed to achieve program objectives discussed in the plan. Incorporating information about cost and resources would facilitate TSA’s ability to allocate resources across programs according to priorities and constraints, track costs and performance, and shift such investments and resources as appropriate.

Organizational roles, responsibilities, and coordination: The SPOT program relies on a close partnership with law enforcement officers at airports. TSA provides briefings to law enforcement on the SPOT program, and TSA officials conduct outreach efforts to local law enforcement as needed. The SPOT SOP guidance and SPOT training include guidance about ensuring that LEOs receive complete and accurate information about each SPOT referral. However, while the strategic plan identifies TSA officials and offices as responsible parties for implementing the strategic plan, it does not provide guidance on how to effectively link the roles, responsibilities, and capabilities of federal, state, and local officials providing program support. Moreover, although SPOT SOP guidance discusses the need for BDOs to coordinate with other TSA personnel, such as TSOs and TDCs, TSA does not identify their roles and responsibilities in regards to the SPOT program in the program’s strategic plan. Integrating these elements into the strategic plan could help to clarify the relationships between these various implementing parties, which would thereby increase accountability and improve the effectiveness of implementation.

Integration and implementation: The SPOT strategic plan does not discuss how its scope complements, expands upon, or overlaps with other related strategic documents. For example, TSA’s April 2008 Office of Security Operations Organizational Business Plan for Fiscal Year 2010 describes how its goals—including those for SPOT—relate to DHS and TSA strategic goals. However, TSA does not link goals in the SPOT strategic plan with other related strategic documents, such as the Aviation Implementation Plan of DHS’s Transportation Systems Sector-Specific Plan and the Passenger Checkpoint Screening Program Strategic Plan. By linking goals in its SPOT strategic plan to other TSA efforts, TSA could better ensure that the program’s objectives are integrated with other TSA security programs and that resources are used effectively by minimizing any unnecessary duplication with these other actions.

More Fully and Consistently Utilizing Available Information Technology Could Enhance TSA’s Ability to Identify Threats to the Aviation System

Inconsistencies in the use of available information technology to aid in the collection and recording of data on passengers by BDOs during referrals to LEOs, lack of guidance on, or a mechanism for, BDOs to request the TSA’s Transportation Security Operations Center to run the names of passengers exhibiting suspicious behaviors against law enforcement and intelligence databases, and the Center’s not checking all of the databases available to it—have limited TSA’s ability to identify potential terrorist threats to the aviation system. Among other information, these databases include terrorism-related watch lists.

Systematic Collection of Data on Passengers Identified Through the SPOT Program Could be Improved to Better Identify Activity Potentially Harmful to the Aviation System

TSA is not fully utilizing the resources it has available to systematically collect the information obtained by BDOs on passengers whose behaviors and appearances resulted in either SPOT referral screening, or in a referral to LEOs, and who thus may pose a risk to the aviation system. TSA’s July 2008 Privacy Impact Assessment on the TSA Transportation Security Operations Center, and its August 2008 Privacy Impact Assessment on SPOT, state that information may be obtained by BDOs to check an individual’s identity against intelligence, terrorist, and law enforcement databases and to permit intelligence analysts to conduct trend analysis.

The August 2008 SPOT Privacy Impact Assessment states that information about a passenger who has exceeded the SPOT behavior threshold, leading to LEO referral, may be collected and entered into DHS’s Transportation Information Sharing System. According to the SPOT Privacy Impact Assessment, information collected may be submitted to the Transportation Information Sharing System database for analysis, and, through it to other linked intelligence databases and the intelligence analysts who study them, to detect, deter, and defeat a criminal or terrorist act in the transportation domain before it occurs. The SPOT Privacy Impact Assessment notes that terrorist acts that threaten transportation security are most vulnerable in the planning stages and that the timely passage of SPOT referral information may assist in identifying such efforts before they become operational. A June 2008 Transportation Information Sharing System Privacy Impact Assessment similarly states that one goal is to use the system data to find trends and patterns that may indicate preoperational terrorist or criminal activity—that is, to “connect the dots” about a planned terrorist attack or criminal enterprise. Information in TSA’s Transportation Information Sharing System is primarily activity or behavioral information but may also contain personal information regarding the individuals identified by the BDO through SPOT. According to TSA, BDOs do not analyze the data obtained during referrals; if they have the appropriate training, they may enter the data by computer into the Transportation Information Sharing System, where they can be analyzed by intelligence analysts. Other appropriately trained and officially designated TSA officials, such as Federal Security Directors, may also enter data into the system.

According to TSA, a 2008 pilot program it conducted that involved BDOs entering data into the Transportation Information Sharing System database was “invaluable,” in part because over 40 referrals have since been passed on to other LEO organizations for further investigation, most of which came from BDO input. A February 2006 TSA memorandum describes the Transportation Information Sharing System as “a critical element in the success of SPOT” because it provides the necessary platform for the reporting of information obtained as a result of SPOT referrals. TSA noted that through the use of the Transportation Information Sharing System, two different BDO teams had separately identified and selected the “same extremist” for secondary questioning. TSA officials also told us about an incident in which an individual sought to board an aircraft with a handgun on two separate occasions, at two different airports. Although the handgun was detected both times, the individual was released after providing what seemed to be a credible explanation. After the second incident, however, intelligence analysts who reviewed the system information saw that this individual had tried twice in 2 weeks to bring a weapon onto an aircraft. A LEO was dispatched to the person’s home, and an arrest was made. Without the data inputted into the system both times, no pattern would have been detected by the analysts, according to TSA. Although the pilot program illustrated the benefits of BDOs entering data into the system, access to the system was not expanded to all SPOT airports in 2008 or 2009.

Internal control standards call for management to develop policies, procedures, and techniques to help enforce management directives. TSA does not provide official guidance on how or when BDOs or other TSA personnel should enter data into the Transportation Information Sharing System or which data should be entered. Official guidance on what data should be entered into the system on passengers could better position TSA personnel to be able to consistently collect information to facilitate synthesis and analysis in “connecting the dots” with regard to persons who may pose a threat to the aviation system.

On March 18, 2010, TSA officials told us that TSA recognizes the value of recording SPOT incidents for the purposes of intelligence gathering. As a result, TSA decided that certain data would be entered into the Transportation Information Sharing System, and would, in turn, be analyzed as a way to potentially “connect the dots” with other transportation security incidents.

TSA officials said that the Federal Security Director at each SPOT airport has been given the discretion to decide which personnel should have access to the Transportation Information Sharing System. However, TSA has not developed a plan detailing how many personnel would have access to the system, or when they would have access at SPOT airports. TSA officials said that training is currently being provided to personnel responsible for using the system at all SPOT airports although they did not provide information on the number being trained.

Standard practices for defining, designing, and executing programs include developing a road map, or program plan, to establish an order for executing specific projects needed to obtain defined programmatic results within a specified time frame. However, TSA stated that it has not developed a schedule or milestones by which database access will be deployed to SPOT airports, or a date by which access at all SPOT airports will be completed. Setting milestones for expanding Transportation Information Sharing System access to all SPOT airports, and setting a date by which the expansion will be completed, could better position TSA to identify threats to the aviation system that may otherwise go undetected and help TSA track its progress in expanding Transportation Information Sharing System access as management intended.

Guidance on and a Mechanism for Running Names of Referred Passengers Through the Databases Available to the Transportation Security Operations Center Could Help Improve SPOT Practices

Internal control standards state that policies, procedures, techniques, and other mechanisms are essential to help ensure that actions are taken to address program risks. The current process makes the BDOs dependent on the LEOs with regard to the timeliness that LEOs respond to BDO calls for service, as well as with regard to whether the LEOs choose to question the passengers referred to them or conduct a background check. Our analysis of the SPOT referral database found a wide variation in the percent of times that LEOs responded to calls for service at SPOT airports. Moreover, if a local LEO decides to run a background check on a passenger referred to them, they would be accessing the FBI’s NCIC and not other intelligence and law enforcement databases.

Although LEOs may not always respond to calls for service, question passengers, or check passenger names against databases available to TSA, TSA has not developed a mechanism allowing BDOs to send information to the Transportation Security Operations Center about passengers whose behavior indicates that they may be a possible threat to aviation security. According to TSA’s July 2008 Transportation Security Operations Center Privacy Impact Assessment, passenger information may be submitted to the Transportation Security Operations Center to ascertain, as quickly as possible, the individual’s identity, whether they are already the subject of a terrorist or criminal investigation, or to analyze suspicious behavior that may signal some form of preoperational surveillance or activity.

Our survey of Federal Security Directors at SPOT airports found a notable inconsistency in the rates at which BDOs at different airports contacted the Transportation Security Operations Center. Developing additional guidance in the SPOT operating procedures could help improve consistency in the extent to which BDOs utilize Transportation Security Operations Center resources. Given the range of responses we received from SPOT airports about whether the BDOs contact the Transportation Security Operations Center to verify passenger identities and run their names against terrorist and intelligence databases and the inconsistencies identified related to LEO responses to BDO requests for service, developing a standard mechanism and providing BDOs with additional guidance could help TSA achieve greater consistency in the SPOT process. Such a mechanism would provide designated TSA officials with a means of verifying passenger identities and help them determine whether a passenger was the subject of a terrorist or criminal investigation and thus posed a risk to aviation security.

Standards for internal control state that effectively using available resources, including key information databases, is one element of functioning internal controls. In this connection, it is widely recognized among intelligence entities and police forces that a capability to “run” names against databases that contain criminal and other records is a potentially powerful tool to both identify those with outstanding warrants and to help discover an ongoing criminal or security-related incident. Additionally, TSA recommended in an April 2008 Organizational Business plan for its Office of Security Operations that the SPOT program should establish a mechanism and policy for allowing real-time checks of federal records for individuals whose behavior indicates they may be a threat to security. The Office of Security Operations plan also states that BDOs should communicate the data to U.S. intelligence centers, with the purpose of permitting rapid communication of this information to local LEOs to take action. However, TSA officials told us that because of safety concerns, the Transportation Security Operations Center does not provide information from database checks directly to BDOs because BDOs are not LEOs, are unarmed, and do not have the training needed to deal with potentially violent persons. If the mechanism discussed in the Office of Security Operations business plan were implemented, it would allow the Transportation Security Operations Center to use BDO information to conduct real-time record checks of passengers and communicate the results to LEOs for action. Such a mechanism could increase the chances to detect ongoing criminal or terror plans.

TSA’s Transportation Security Operations Center Does Not Use All Database Resources When Contacted

The final report of the National Commission on Terrorist Attacks Upon the United States (the “9/11 Commission Report”) recommends that in carrying out its goal of protecting aviation, TSA should utilize the larger set of information maintained by the federal government, that is, the entire Terrorist Screening Database—the U.S. government’s consolidated watch list that contains information on known or suspected international and domestic terrorists—as well as other government databases, such as intelligence or law enforcement databases. However, the Transportation Security Operations Center is not using all the resources at its disposal to support BDOs in verifying potential risks to the aviation system. This reduces the opportunities to “connect the dots” that would increase the chances of detecting terrorist attacks in their planning stage, which the SPOT Privacy Impact Assessment states is when they are the most vulnerable.

According to TSA, the Transportation Security Operations Center has access to multiple law enforcement and intelligence databases that can be used to verify the identity of airline passengers; these include among others: 1. the Selectee list, which identifies persons who must undergo enhanced screening at the checkpoint prior to boarding; 2. the No-Fly list, which lists persons prohibited from boarding aircraft; and 3. the Terrorist Identity Datamark Environment terrorist list.

TSA stated that the Transportation Security Operations Center checks passenger names submitted to it against these three databases if the passenger has been referred by a BDO to a LEO, but has not been arrested. Of the three databases that the Transportation Security Operations Center is to check in the case of a referral, passengers would have already been screened against two—the Selectee and No-Fly lists—in accordance with TSA passenger prescreening procedures when purchasing a ticket. The third database checked—the Terrorist Identity Datamark Environment— tracks terrorists but not persons wanted for other crimes. The FBI’s NCIC information system would contain names of such persons, but is not among the three databases checked for nonarrest referrals. If the passenger has been arrested, the Transportation Security Operations Center will run the passenger’s name against the additional law enforcement and intelligence databases available to it.

In addition, TSA told us that the Operations Center does not have direct electronic access to the Terrorist Screening Database and must call the FBI’s Terrorist Screening Center to provide it with a name to verify. TSA stated that this is done if a passenger’s identity could not be verified using the Operations Center databases. In effect, if a passenger has been referred to a LEO, but not arrested, the Operations Center is to check the three databases shown above to verify the passenger’s identity. If a passenger has been arrested, but the three databases do not list the person, the Center can check the additional databases available to it. If none of these databases can verify the person’s identity, the Operations Center can contact the Terrorist Screening Center by telephone to request further screening. screening database (the watch list) and to provide for the use of watch-list records during security-related screening processes. See GAO-08-136T, Aviation Security: TSA Is Enhancing Its Oversight of Air Carrier Efforts to Screen Passengers against Terrorist Watch-List Records, but Expects Ultimate Solution to Be Implementation of Secure Flight (Washington, D.C.: Sept. 9, 2008).

For passengers who have risen to the level of a LEO referral at an airport checkpoint, having the Transportation Security Operations Center consistently check their names against all the databases available to it could potentially help TSA identify threats to the aviation system and aid in “connecting the dots.” TSA indicated that there are no obstacles to rapidly checking all databases rather than the three listed. We did not analyze the extent to which the law enforcement and intelligence databases available to TSA may contain overlapping information.

TSA Lacks Program Effectiveness Measures for SPOT but Is Taking Steps to Improve Evaluation Capabilities

TSA has established some performance measures by tracking SPOT referral and arrest data, but lacks the measures needed to evaluate the effectiveness of the SPOT program and, as a result, has not been able to fully assess SPOT’s contribution to improving aviation security. TSA emphasized the difficulty of developing performance measures for deterrence-based programs, but stated that it is developing additional measures to quantify the effectiveness of the program. The SPOT program uses teams to assess BDO proficiency, provide individual and team guidance, and address issues related to the interaction of BDOs with TSA checkpoint personnel. However, TSA does not systematically track the teams’ recommendations or the frequency of the teams’ airport visits. TSA states that it is working to address these issues and plans to do so by the end of fiscal year 2010.

TSA Has Taken Action to Collect Data for Some Performance Measures, but Work Remains to Assess Progress Towards Achieving Strategic Goals

TSA agreed that the SPOT program lacked sufficient performance measures in the past, but stated that it has some performance measures in place including tracking data on passengers referred for additional screening and the resolution of this screening, such as if prohibited items were found or if law enforcement arrested the passenger and the reason for the arrest. TSA is also working to improve its evaluation capabilities to better assess the effectiveness of the program. DHS’s NIPP, internal controls standards, and our previous work on program assessment state that performance metrics and associated program evaluations are needed to determine if a program works and to identify adjustments that may improve its results. Moreover, standard practices in program management for defining, designing, and executing programs include developing a road map, or program plan, to establish an order for executing specific projects needed to obtain defined programmatic results within a specified time frame. Congress also needs information on whether and in what respects a program is working well or poorly to support its oversight of agencies and their budgets; and agencies’ stakeholders need performance information to accurately judge program effectiveness. For example, in the Senate Appropriations Committee report accompanying the fiscal year 2010 DHS appropriations bill, the committee noted that while TSA has dramatically increased the size and scope of SPOT, resources were not tied to specific program goals and objectives. In addition, the conference report accompanying the fiscal year 2010 DHS appropriations act requires TSA to report to Congress, within 60 days of enactment, on the effectiveness of the program in meeting its goals and objectives, among other things. This report was completed on March 15, 2010.

Although TSA tracks data related to SPOT activities including prohibited items, law enforcement arrests related to SPOT referrals, and reasons for the arrests (output measures), it has not yet developed measures to gauge SPOT’s effectiveness in meeting TSA strategic goals (outcome measures), such as identifying individuals who may pose a threat to the transportation system. OMB encourages the use of outcome measures because they are more meaningful than output measures, which tend to be more process- oriented or means to an end. For example, TSA’s Assistant General Manager for the Office of Operation Process and Performance Metrics told us that SPOT staffing levels are currently used as one performance metric. The official said that since the SPOT is an added layer of security, additional SPOT staffing would add to security effectiveness. While staffing levels may help gauge how fast the program is growing, they do not measure the effectiveness in meeting strategic goals.

Similarly, TSA also cited the number of prohibited items discovered by BDOs in SPOT metrics reports as a measure of program success. However, TSA told us that possession of a prohibited item is often an oversight and not an intentional act; moreover, other checkpoint screening layers are intended to find such items, such as the TSOs and the property screening equipment. TSA also cited measures of BDO job performance as some of the existing measures of program effectiveness, but noted that these are “pass/fail” assessments of individual BDOs, rather than overall program measures.

TSA notes that one purpose of the SPOT program is to deter terrorists, but that proving that it has succeeded at deterring terrorists is difficult because the lack of data has presented challenges for the SPOT program office when developing performance measures. We agree that developing performance measures, especially outcome measures, for programs with a deterrent focus is difficult. Nevertheless, such measures are an important tool to communicate what a program has accomplished and provide information for budget decisions. TSA uses proxy measures—indirect measures or indicators that approximate or represent the direct measure—to address deterrence, other security goals, or a combination of both. For example, TSA tracks the number of prohibited items found and individuals arrested as a result of SPOT referrals. According to OMB, proxy measures are to be correlated to an improved security outcome, and the program should be able to demonstrate—such as through the use of modeling—how the proxies tie to the eventual outcome.In using a variety of proxy measures, failure in any one of the identified measures could provide an indication on the overall risk to security. However, developing a plan that includes objectives, milestones, and time frames to develop outcome-based performance measures could better position TSA in assessing the effectiveness of the SPOT program.

With regard to more readily quantifiable output performance measures, such as the number of referrals by BDOs, or the ratio of arrests to referrals, TSA was limited in its ability to analyze the data related to these measures. The SPOT database includes information on all passengers referred by BDOs for additional SPOT screening including the behaviors of the passengers that led to the additional screening, as well as the resolution of the screening process (e.g., no further action taken, law enforcement notification, law enforcement investigation, arrested, and reason for arrest). However, TSA reported that any analysis of the data had to be done manually.

In March 2010, TSA migrated the SPOT referral data to its Performance Management Information System, allowing for more statistical and other analyses. According to TSA, migrating the SPOT referral database will enhance the SPOT program’s analytic capabilities. For example, TSA stated that it would be able to conduct trend analyses, better segregate data, and create specific reports for certain data. This includes better tracking of performance data at specific airports, analyzing by categories of airports (threat or geographic location), and tracking the performance data of individual BDOs, such as number of referrals, number of arrests, arrest to referral ratios, and other analyses. However, since these changes to the database were not complete at the time of our audit, we could not assess whether the problems we identified with the database had been corrected.

Over 4 Years, SPOT Resulted in About 1,100 Arrests Out of Almost 14,000 Referrals to Law Enforcement

The SPOT referral database records the total number of SPOT referrals since May 29, 2004, how many were resolved, how many passengers BDOs referred to LEOs, the recorded reasons for the referral, and how many referrals led to arrests, among other things. As shown in figure 4, we analyzed the SPOT referral data for the period May 29, 2004, to August 31, 2008.

Figure 4 shows that approximately 2 billion passengers boarded aircraft at SPOT airports from May 29, 2004, through August 31, 2008. Of these, 151,943 (less than 1/100th of 1 percent) were sent to SPOT referral screening, and of these, 14,104 (9.3 percent) were then referred to LEOs. These LEO referrals resulted in 1,083 arrests, or 7.6 percent of those referred, and less than 1 percent of all SPOT referrals (0.7 percent of 151,943).

We also analyzed the reasons for arrests resulting from SPOT referrals, for the May 29, 2004, through August 31, 2008, period. Table 2 shows, in descending order, the reasons for the arrests.

While SPOT personnel did not determine a specific reason for arrest for 128 cases categorized as “other” or 16 other cases categorized as “no reason given,” our analysis of the SPOT database found that a specific reason for arrest could have been determined for these cases by using the LEO resolution notes included in the database. For example, we identified 43 additional arrests related to fraudulent documents, illegal aliens, and suspect documents, among others. The remaining 101 arrests originally characterized as “other” or “no reason given” included arrests for reasons such as intoxication, unruly behavior, theft, domestic violence, and possession of prohibited items. Many of the arrests resulting from BDO referrals would typically fall under the jurisdiction of various local, state, and federal agencies and are not directly related to threats to aviation security. For example, the 427 individuals arrested as illegal aliens, and the 166 arrested for possession of fraudulent documents, are subject to the enforcement responsibilities shared by U.S. Immigration and Customs Enforcement (ICE) and CBP. Although outstanding warrants and the possession of fraudulent or suspect documents could be associated with a terrorist threat, TSA officials did not identify any direct links to terrorism or any threat to the aviation system in any of these cases.

According to TSA, anecdotal examples of BDO actions at airports show the value added by SPOT to securing the aviation system. However, because the SPOT program has not been scientifically validated, it cannot be determined if the anecdotal results cited by TSA were better than if passengers had been pulled aside at random, rather than as a consequence of being identified for further screening by BDOs. Some of the incidents cited by TSA include the following.

A BDO referred two passengers who were traveling together to referral screening due to suspicious behavior. During secondary screening, one passenger presented fraudulent travel documents. The other could not produce any documentation of his citizenship and it was determined he was in the United States illegally. ICE responded and interviewed both passengers. ICE stated one passenger was also in possession of $10,000 dollars which alarmed positive for narcotics when swept by a K-9 team. ICE arrested one passenger on a federal charge of possession of fraudulent identification documents and entry without inspection. ICE stated charges are still pending for the possession of $10,000. The second passenger was charged with a federal charge of entry without inspection.

A BDO referred a passenger to referral screening for exhibiting suspicious behavior. Port Authority of Portland (Oregon) Police responded and interviewed the passenger who did not give a statement. LEOs conducted an NCIC check which revealed that there was an outstanding warrant for the failure to appear for a theft charge. LEOs arrested the passenger on a state charge for an outstanding warrant for the failure to appear for theft.

A BDO referred a passenger for referral screening due to suspicious behavior. During the referral, the passenger admitted that he was unlawfully present in the United States. The Orlando (Florida) Police Department and CBP responded and interviewed the passenger who stated he had $100,000 in his checked baggage, which was confirmed by CBP. The passenger was arrested on a federal charge of illegal entry.

Because these are anecdotal examples, they cannot be used to reliably generalize about the SPOT program’s overall effectiveness or success rate. Our analysis of the SPOT referral database found that the referral data do not indicate if any of the passengers sent to referral screening, or those arrested by LEOs after being referred to them, intended to harm the aircraft, its passengers, or other components of the aviation system. Additionally, SPOT officials told us that it is not known if the SPOT program has ever resulted in the arrest of anyone who is a terrorist, or who was planning to engage in terrorist-related activity.

Reviewing Airport Video Recordings of Individuals Later Arrested or Who Pleaded Guilty for Engaging in Terrorism- Related Activities Could Help TSA Better Identify Terrorist- Related Behaviors

Studying airport video recordings of the behaviors exhibited by persons waiting in line and moving through airport checkpoints and who were later charged with or pleaded guilty to terrorism-related offenses could provide insights about behaviors that may be common among terrorists or could demonstrate that terrorists do not generally display any identifying behaviors. TSA officials agreed that examining video recordings of individuals who were later charged with or pleaded guilty to terrorism- related offenses, as they used the aviation system to travel to overseas locations allegedly to receive terrorist training or to execute attacks, may help inform the SPOT program’s identification of behavioral indicators. In addition, such images could help determine if BDOs are looking for the right behaviors or seeing the behaviors they have been trained to observe.

Using CBP and Department of Justice information, we examined the travel of key individuals allegedly involved in six terrorist plots that have been uncovered by law enforcement agencies. We determined that at least 16 of the individuals allegedly involved in these plots moved through 8 different airports where the SPOT program had been implemented. Six of the 8 airports were among the 10 highest risk airports, as rated by TSA in its Current Airport Threat Assessment. In total, these individuals moved through SPOT airports on at least 23 different occasions. For example, according to Department of Justice documents, in December 2007 an individual who later pleaded guilty to providing material support to Somali terrorists boarded a plane at the Minneapolis-Saint Paul International Airport en route to Somalia to join terrorists there and engage in jihad. Similarly, in August 2008 an individual who later pleaded guilty to providing material support to Al-Qaeda boarded a plane at Newark Liberty International Airport en route to Pakistan to receive terrorist training to support his efforts to attack the New York subway system.

Our survey of Federal Security Directors at 161 SPOT airports indicated most checkpoints at SPOT airports have surveillance cameras installed. As we previously reported, best practices for project management call for conducting feasibility studies to assess issues related to technical and economic feasibility, among other things. In addition, Standards for Internal Control state that effectively using available resources is one element of functioning internal controls. TSA may be able to utilize the installed video infrastructure at the nation’s airports to study the behavior of persons who were later charged with or pleaded guilty to terrorism- related offenses, and determine whether BDOs saw the behaviors. The Director of Special Operations in TSA’s Office of Inspection told us that video recordings could be used as a teaching tool to show the BDOs which behaviors or activities they did or did not observe. In addition, TSA indicated that although the airports may have cameras at the security screening checkpoints, the cameras are not owned by TSA, and in many cases, they are not accessible to TSA. However, TSA officials lack information on the scope of these potential limitations because prior to our work TSA did not have information on the number of checkpoints equipped with video surveillance. We obtained this information as part of our survey of Federal Security Directors at SPOT airports. While TSA officials noted several possible limitations of the use of the existing video surveillance equipment, these images provide TSA a means of acquiring information about terrorist behaviors in the checkpoint environment that is not available elsewhere. If current research determines that the SPOT program has a scientifically validated basis for using behavior detection for counterterrorism purposes in the airport environment, then conducting a study to determine the feasibility of using images captured by video cameras could better position TSA in identifying behaviors to observe.

Standardization Teams Assess BDO Proficiency in SPOT Activities and Provide Guidance and Mentoring to BDOs

TSA sends standardization teams to SPOT airports on a periodic basis to conduct activities related to quality control. Teams observe SPOT operations at an airport for several days, working side by side with the BDOs, on multiple shifts, observing their performance, offering guidance, and providing training when required. According to TSA, the purpose of a standardization team visit is to provide operational support to the BDOs, which includes additional training, mentoring, and guidance to help maintain a successful SPOT program.

The standardization teams are comprised of at least two G-Band, or Expert BDOs who have received an additional week of training on SPOT behaviors and mentoring skills. SPOT officials stated that the SPOT program uses its standardization teams to assess overall BDO proficiency by observing BDOs, reviewing SPOT score sheet data, and other relevant data. Standardization teams may also provide a Behavior Observation and Analysis review class to refresh BDOs if the team determines that such a class is needed. The SPOT program director also said that the standardization teams aim to monitor the airport’s compliance with the SPOT program’s Standard Operating Procedures. As part of this mentoring approach, the standardization teams provide individual and team guidance to the BDOs, offer assistance in program management, and cover issues related to the interaction of BDOs with other TSA checkpoint personnel.

TSA reported to us that it does not systematically track the standardization teams’ recommendations or the frequency of the teams’ airport visits. Standards for Internal Control state that programs should have controls in place to assess the quality of performance over time and ensure that the findings of audits and other reviews are promptly resolved. Managers are to (1) promptly evaluate findings from audits and other reviews, including those showing deficiencies and recommendations reported by auditors and others who evaluate agencies’ operations; (2) determine proper actions in response to findings and recommendations from audits and reviews; and (3) complete, within established time frames, all actions that correct or otherwise resolve the matters brought to management’s attention. Although the standardization teams may provide an airport Federal Security Director with recommendations on how to improve SPOT operations, the SPOT program director stated that Federal Security Directors are not required to document whether they have implemented the team recommendations. TSA officials told us that standardization teams can follow up on recommendations made during previous visits. However, TSA did not track whether corrective actions were implemented or the frequency of the team’s airport visits to ensure the implementation of the airport’s SPOT program. TSA officials stated that they are currently examining ways to compile data to address this issue, and expect to have a system in place in fiscal year 2010.

TSA Developed and Deployed SPOT Training but Further Action Could Enhance Its Effectiveness

Although TSA has taken steps to incorporate all four elements of an effective training program by planning, designing, implementing, and evaluating training for BDOs, further action could help enhance the training’s effectiveness. TSA initially consulted outside experts for help in the training’s development, which began as a half-day course and has grown to include classroom, on-the-job, and advanced training. TSA also has efforts underway to improve its training program, such as the deployment of SPOT recurrent training. However, TSA evaluations of SPOT program instructors found mixed quality among them, from 2006 onwards. Additionally, TSA has ongoing plans to evaluate the SPOT training for effectiveness, but has not yet developed time frames and milestones for completing the evaluation.

TSA Has Taken Actions to Develop and Deploy SPOT Training

TSA’s SPOT Training Evolved Over Time

In 2003, TSA officials at Boston Logan International airport developed the initial half-day training course for SPOT based on an existing course developed for the Massachusetts State Police. Their goal was to take the behavior detection program designed for law enforcement and apply it to screeners at airport checkpoints. According to TSA officials at Boston Logan, after they recognized that the lecture-style course they originally designed was not effective, they tasked an instructional system designer from TSA’s Workplace Performance and Training (the former name of TSA’s Operational and Technical Training Division) and an industrial psychologist from the Office of Human Capital to redesign and expand the course, which was piloted in 2005. The 2007 SPOT strategic plan included training objectives for the SPOT program as follows: reviewing existing behavior observation training providers, establishing and prioritizing multimodal training and assistance efforts based on threat assessments and critical infrastructure, establishing a Center of Excellence for Behavior Detection Program training that would continually enhance the quantity and quality of training to selected candidates, and developing a recurrent training program designed to refresh and hone skills needed for an effective Behavior Detection Program.

Since that time, the SPOT program implemented, or is in the process of implementing, some of these objectives. For example, in 2008, as part of its effort towards establishing a center for excellence in behavior detection training (third objective), the SPOT program participated in a meeting with behavior detection training officials from various DHS components facilitated by DHS’s Screening Coordination Office to promote the sharing of information about behavior detection training and foster future collaboration. Additionally, the SPOT program worked with TSA’s Operational and Technical Training Division to create a recurrent training component for BDOs (fourth objective). For example, in 2008, the SPOT program office added a course on detecting microfacial expressions called Additional Behavior Detection Techniques. This 3-day course builds on the behavior detection skills taught in basic training, by teaching BDOs how to detect microfacial expressions. After pilot testing, the course began implementation nationwide in January 2009.

TSA Consulted with Some Experts on Developing SPOT Training

In developing an effective training program, we previously reported that consultation with subject matter experts and expert entities is a core characteristic of the strategic training and development process. TSA SPOT program staff told us that they consulted with experts on behavior detection and observed existing behavior detection courses before deploying the SPOT training program. According to SPOT program officials, a TSA staff member from Boston Logan International Airport attended other training programs offered by other federal agencies and private training organizations to inform the design of SPOT training. TSA officials told us that information from the training courses was used to help develop the list of behaviors or “stress elevators” for the program, and that the point system used to identify passengers for referral screening was based in part on consultations with several subject-matter experts.

TSA documentation also notes that a SPOT working group created in February 2004 consulted with the FBI’s Behavioral Science Unit. The Behavioral Science Unit specializes in developing and facilitating training, research, and consultation in the behavioral sciences for the FBI, law enforcement, intelligence, and military communities. While TSA officials from Boston Logan told us that the FBI was included in this initial SPOT working group, these officials agree that coordination with the FBI lapsed until June 2009 when the SPOT Program Office reengaged with the Behavioral Science Unit, and held a meeting with the unit at the FBI Academy in Quantico, Virginia. Since that meeting, a subject matter expert from the SPOT Program Office has been invited to be a member of the Terrorism Research and Analysis Project, which is an ongoing working group sponsored by the unit.

In July 2008, DHS’s Screening Coordination Office facilitated a collaborative discussion on behavior detection that included TSA, CBP, and Secret Service officials to better ensure that components within DHS share information regarding their efforts in behavior detection and provide a forum for components to have an informed and collaborative discussion on current capabilities, best practices, and lessons learned. According to TSA, no further contact has occurred between the DHS Behavior Detection Working Group and the SPOT program. Thus, the extent to which the working group’s expertise will be used to refine or augment SPOT training in the future is not yet clear.

SPOT Program Office Recently Deployed Recurrent Training

Along with basic and remedial training required by the Aviation and Transportation Security Act, TSA policy requires its screening force to regularly complete recurrent (refresher) training. TSA recognized that ongoing training of screeners on a frequent basis and effective supervisory training are critical to maintaining and enhancing skills learned during basic training. According to agency officials, TSA is currently working with DHS S&T to determine the necessary frequency for refresher training for each training course within the SPOT program. Furthermore, TSA plans to place BDOs under TSA’s Performance and Accountability Standards System (PASS) beginning in fiscal year 2010. This will include a recertification module.

In 2008, the SPOT program office began the process for developing recurrent SPOT training. Our internal control standards and training assessment guidance suggest that such refresher training should be considered integral to an effective training program from the start because work conditions and environments can be expected to change over time, and additional or updated training is essential to ensuring that the program mission continues to be accomplished. According to the SPOT program office, the recently deployed recurrent training will be semiannual. TSA’s Operational and Technical Training Division initially planned to pilot test recurrent training in April 2009 followed by full implementation of the course in approximately May 2009. Because the Operational and Technical Training Division focus was shifted to completing the revisions for the SPOT basic certification course, recurrent training was delayed until September 2009 when they released the training on TSA’s Online Learning Center.

Instructor Evaluations Found Mixed Quality; Issues with Program Management Led to Instructor Retraining

Our previous work on elements of effective training states that instructors must be both knowledgeable about the subject matter and issues involved, as well as able to effectively transfer these skills and knowledge to others. Moreover, internal control standards state that all personnel need to possess and maintain a level of competence that allows them to accomplish their assigned duties. Management needs to identify appropriate knowledge and skills needed for various jobs and provide needed training, as well as to ensure that those teaching the skills are themselves competent.

TSA conducted internal assessments of SPOT instructors episodically from 2006 through March 2008. These assessments involved a few instructors being rated at a time, and found a wide range of competency among the instructors. In January 2009, TSA’s Office of Inspections and Investigations began an investigation of the SPOT training manager, who resigned shortly thereafter. TSA investigators determined that the training manager and other trainers had created a hostile training environment that intimidated some trainees. To address this problem, TSA stated that the program office reexamined the SPOT training program nationally. This included recertifying 47 of 54 SPOT instructors in March 2009, which included evaluation by TSA’s Office of Human Capital, Quality Assurance assessors. Additionally, in July 2009, TSA centralized SPOT training at five permanent, regional training facilities in Orlando, Florida; Houston, Texas; Phoenix, Arizona; Denver, Colorado; and Philadelphia, Pennsylvania. According to the SPOT program director, this will allow the SPOT program office more oversight over training. Previously, training was provided at individual airports.

After the March 2009 recertification training, ratings scores of SPOT instructors showed less variation than did previous ratings. We reviewed the quality assurance instructor evaluations of two SPOT instructors conducted by TSA’s Office of Human Capital, Training Standards and Evaluation Branch, and the 167 SPOT program instructor evaluations of 54 SPOT instructors conducted by the SPOT program office and TSA’s Operational and Technical Training Division since the program started in October 2006. After the recertification training, 93 percent of instructors were rated as exceeding expectations, compared to 30 percent in the 2006 to September 2008 ratings. Table 3 shows the ratings of instructors for March 2009 compared to the period of 2006 to September 2008.

In addition to the variation in numeric scores and rating levels for the 2006 to September 2008 period, as shown in table 3, we found substantial variation in the comments about instructor competency for the same period. For example, in 32 out of 74 instructor evaluation forms that we reviewed where comments were made about the instructor prior to 2009, the comments ranged from superb to needs more experience as an instructor, as well as needs more time performing the job as a BDO to be able to teach others. In the comments on an instructor who was rated as “meets expectations,” the instructor was described as having “limited experience within the SPOT program,” that this was “a major concern,” and it was recommended that the instructor spend as much time as possible functioning as a BDO. In other cases, however, SPOT instructors were described as competent, solid, and outstanding. For example, one instructor who received a rating of “exceeds expectations” was described as a superb instructor who “is a valued member of the National Training Team.” As noted above, following the March 2009 recertification training, 93 percent of the instructors received a rating of “exceeds expectations” with only 1 percent “needing improvement.” Of the 94 instructor evaluations completed in March 2009, 82 contained written comments. Of these, multiple SPOT instructors were described as excellent, knowledgeable, and effective. For example, an instructor who received a rating of “exceeds expectations” was noted as demonstrating a high degree of material knowledge and great presentation skills. TSA attributed the increase in instructor ratings to two factors. The first is low turnover among SPOT instructors, which allows instructors to hone both their technical and instructor skills. The second factor cited by TSA is that TSA conducted a 2-day instructor refresher training immediately prior to the evaluations in March 2009. To ensure all instructors were reevaluated within a specific time frame, evaluations were scheduled and conducted in a controlled environment. Instructors knew in advance they were going to be evaluated and delivered modules of the BDO certification course to other BDO instructors.

TSA Has Taken Some Action, but Has Not Evaluated the SPOT Training Program for Effectiveness

We previously reported that evaluation is an integral part of training and development efforts, and that agencies need to systematically plan for and evaluate the effectiveness of training and development. Employing systematic monitoring and feedback processes can help by catching potential problems at an early stage, thereby saving valuable time and resources that a major redesign of training would likely entail. Similarly, in 2006, TSA’s Operational and Technical Training Division issued general evaluation standards for training programs, stating that training programs should be comprehensively evaluated on a periodic basis to identify program strengths and weaknesses. Moreover, standard practices in program management for defining, designing, and executing programs include developing a road map, or program plan, to establish an order for executing specific projects needed to obtain defined programmatic results within a specified time frame.

The former SPOT training manager told us that the SPOT program internally evaluates the effectiveness of SPOT training through the job knowledge tests that BDO candidates must pass following the classroom portion of the training and the SPOT Proficiency/On-the-Job Training Checklist following the on-the-job portion of the training. Furthermore, the former training manager told us that TSA knows that the SPOT training is effective because BDOs are able to recognize behaviors at the checkpoint, and because of BDOs’ demonstrated ability to identify criminals—such as drug couriers or people with outstanding arrest warrants—through the screening process.

Although TSA has not conducted a comprehensive analysis of the effectiveness of the SPOT training program, TSA’s Office of Human Capital, Training Standards and Evaluation Branch conducted training evaluations to assess how students use what they were taught in the SPOT basic training course. Specifically, from July through September 2008, the Training Standards and Evaluation Branch conducted evaluations at 5 of the 161 airports where the SPOT program is currently operating. Based on BDO feedback at the 5 airports, the Training Standards and Evaluation Branch’s final report contained a series of recommendations for improving the SPOT training program. These recommendations and TSA’s actions to address them are summarized in table 4.

Additionally, in conjunction with S&T, TSA conducted a training effectiveness evaluation on the Additional Behavior Detection Techniques course, which showed a statistically significant increase in knowledge and skills following completion of the course.

S&T is currently conducting a BDO job task analysis, which may be used to evaluate and update the SPOT training curriculum. Following the completion of the job task analysis—anticipated in mid-May 2010—TSA’s Operational and Technical Training Division intends to conduct an in- depth training gap analysis, which will take approximately 2 months to complete. Following completion of the training gap analysis, the agency will develop project plans, including milestones for future development efforts, to address any training concerns. However, to date, the agency does not have an evaluation plan including time frames and milestones for completion. According to the Operational and Technical Training Division, TSA will conduct periodic evaluations as the BDO position evolves. By conducting a comprehensive evaluation of the effectiveness of its training program, TSA will be in a better position to determine if BDOs are being taught the knowledge and skills they need to perform their job. Furthermore, by developing milestones and time frames for conducting such evaluations systematically, as well as on a periodic basis, TSA could help ensure that the SPOT training program is evaluated in accordance with its directives to help ensure that the program continues to provide BDOs with the necessary tools required to carry out their responsibilities.

Conclusions

TSA developed the SPOT program in the wake of September 11, 2001, in an effort to respond quickly to potential threats to aviation security by identifying individuals who may pose a threat to aviation security, including terrorists planning or executing an attack who were not likely to be identified by TSA’s other screening security measures. Because TSA did not ensure that SPOT’s underlying methodology and work methods were scientifically validated prior to its nationwide deployment, an independent panel of experts could help determine whether a scientific foundation exists for the way in which the SPOT program uses behavior detection analysis for counterterrorism purposes in the aviation environment.

With approximately $5.2 billion devoted to screening passengers and their property in fiscal year 2009, it is important that TSA provides effective stewardship of taxpayer funds ensuring a return on investment for each layer of its security system. As one layer of aviation security, the SPOT program has an estimated projected cost of about $1.2 billion over the next 5 years if the administration’s requested funding of $232 million for fiscal year 2011 remains at this level. The nation’s constrained fiscal environment makes it imperative that careful choices be made regarding which investments to pursue and which to discontinue. If an independent expert panel determines that DHS’s study is sufficiently comprehensive to determine whether the SPOT program is based on valid scientific principles that can be effectively applied in an airport environment for counterterrorism purposes, then conducting a comprehensive risk assessment including threat, vulnerability, and consequence could strengthen TSA’s ability in making resource allocation decisions and prioritizing its risk mitigation efforts. Moreover, conducting a cost-benefit analysis could help TSA determine whether SPOT provides benefits greater than or equal to other security alternatives and whether its level of investment in the SPOT program is appropriate. Revising its strategic plan for SPOT to incorporate risk assessment information, cost and resource analysis, and other essential components could enhance the plan’s usefulness to TSA in making program management and resource allocation decisions to effectively manage the deployment of SPOT.

Providing guidance on how to use TSA’s resources for running passenger names against intelligence and criminal databases available to the Transportation Security Operations Center and helping DHS to connect disparate pieces of information using the Transportation Information Sharing System and other related intelligence and crime database and data sources could better inform DHS and TSA regarding the identity and background of certain individuals and thereby enhance aviation security. In addition, implementing the steps called for in the TSA Office of Strategic Operations plan to provide BDOs with a real-time mechanism to verify passenger identities and backgrounds via TSA’s Transportation Security Operations Center could strengthen their ability to rapidly verify the identity and background of passengers who have caused concern, and increase the likelihood of detecting and disrupting potential terrorists intending to cause harm to the aviation system. Additionally, developing outcome-oriented performance measures, making improvements to the SPOT database, and studying the feasibility of utilizing video recordings of individuals as they transited checkpoints and who were later charged with or pleaded guilty to terrorism-related offenses, could help TSA evaluate the SPOT program, identify potential vulnerabilities, and assess the effectiveness of its BDOs. Further, developing a plan for systematic and periodic evaluation of the training provided to BDOs along with time frames and milestones for its completion could help ensure that the SPOT training program is evaluated in accordance with its directives to help ensure that the program continues to provide BDOs with the necessary tools required to carry out their responsibilities.

Recommendations for Executive Action

To help ensure that SPOT is based on valid scientific principles that can be effectively applied in an airport environment, we recommend that the Secretary of Homeland Security convene an independent panel of experts to review the methodology of the DHS S&T Directorate study on the SPOT program to determine whether the study’s methodology is sufficiently comprehensive to validate the SPOT program. This assessment should include appropriate input from other federal agencies with expertise in behavior detection and relevant subject matter experts.

If this research determines that the SPOT program has a scientifically validated basis for using behavior detection for counterterrorism purposes in the airport environment, then we recommend that the TSA Administrator take the following four actions: Conduct a comprehensive risk assessment to include threat, vulnerability, and consequence of airports nationwide to determine the effective deployment of SPOT if TSA’s ongoing Aviation Modal Risk Assessment lacks this information.

Perform a cost-benefit analysis of the SPOT program, including a comparison of the SPOT program with other security screening programs, such as random screening, or already existing security measures.

Revise and implement the SPOT strategic plan by incorporating risk assessment information, identifying cost and resources, linking it to other related TSA strategic documents, describing how SPOT is integrated and implemented with TSA’s other layers of aviation security, and providing guidance on how to effectively link the roles, responsibilities, and capabilities of federal, state, and local officials providing program support.

Study the feasibility of using airport checkpoint-surveillance video recordings of individuals transiting checkpoints who were later charged with or pleaded guilty to terrorism-related offenses to enhance understanding of terrorist behaviors in the airport checkpoint environment.

Concurrent with the DHS S&T Directorate study of SPOT, and an independent panel assessment of the soundness of the methodology of the S&T study, we recommend that the TSA Administrator take the following six actions to ensure the program’s effective implementation: To provide additional assurance that TSA utilizes available resources to support the goals of deterring, detecting, and preventing security threats to the aviation system, TSA should: Provide guidance in the SPOT Standard Operating Procedures or other TSA directive to BDOs, or other TSA personnel, on inputting data into the Transportation Information Sharing System and set milestones and a time frame for deploying Transportation Information Sharing System access to SPOT airports so that TSA and intelligence community entities have information from all SPOT LEO referrals readily available to assist in “connecting the dots” and identifying potential terror plots. Implement the steps called for in the TSA Office of Security Operations Business plan to develop a standardized process for allowing BDOs or other designated airport officials to send information to TSA’s Transportation Security Operations Center about passengers whose behavior indicates that they may pose a threat to security, and provide guidance on how designated TSA officials are to receive information back from the Transportation Security Operations Center.

Direct the TSA Transportation Security Operations Center to utilize all of the law enforcement and intelligence databases available to it when running passenger names, for passengers who have risen to the level of a LEO referral.

To better measure the effectiveness of the program and evaluate the performance of BDOs, TSA should: Establish a plan that includes objectives, milestones, and time frames to develop outcome-oriented performance measures to help refine the current methods used by Behavior Detection Officers for identifying individuals who may pose a risk to the aviation system.

Establish controls to help ensure completeness, accuracy, authorization, and validity of data collected during SPOT screening.

To help ensure that TSA provides BDOs with the knowledge and skills needed to perform their duties, TSA should: Establish time frames and milestones for its plan to systematically conduct evaluations of the SPOT training program on a periodic basis.

Agency Comments and Our Evaluation

We provided a draft of our report to DHS and TSA on March 19, 2010, for review and comment. On May 3, 2010, DHS provided written comments, which are reprinted in appendix II. In commenting on our report, DHS stated that it concurred with 10 of our recommendations and identified actions taken, planned, or under way to implement them. However, the actions DHS reported it plans to take and has underway do not fully address the intent of our first recommendation. DHS also concurred in principle with an eleventh recommendation stating that it had convened a working group to determine the feasibility of implementing it. DHS commented on the scientific basis underlying SPOT and on two statements in our report that it believed were inaccurate—specifically, DHS disagreed with our reliance on a 2008 National Research Council report published under the auspices of the National Academy of Sciences on issues related to behavior detection, and second, on issues related to unpublished research they had cited as a partial validation of some aspects of the SPOT program. Finally, DHS commented on our conclusion regarding the use of the SPOT referral data.

Regarding our first recommendation that DHS convene an independent panel of experts to review the methodology of DHS’s Science and Technology Directorate (S&T) study on SPOT, and to include appropriate input from other federal agencies with relevant expertise, DHS concurred and stated the current process includes an independent review of the program that will include input from other federal agencies and relevant experts. Although DHS has contracted with the American Institutes for Research to conduct its study, it remains unclear who will oversee this review and whether they are sufficiently independent from the current research process. DHS’s response also does not describe how the review currently planned is designed to determine whether the study’s methodology is sufficiently comprehensive to validate the SPOT program. As we noted in our report, research on other issues, such as determining the number of individuals needed to observe a given number of passengers moving at a given rate per day in an airport environment or the duration that such observation can be conducted by BDOs before observation fatigue affects effectiveness, could provide additional information on the extent to which SPOT can be effectively implemented in airports. Dr. Paul Ekman, a leading research scientist in the field of behavior detection, told us that additional research could help determine the need for periodic refresher training since no research has yet determined whether behavior detection is easily forgotten or can be potentially degraded with time or lack of use. Thus, questions exist as to whether behavior detection principles can be reliably and effectively used for counterterrorism purposes in airport settings to identify individuals who may pose a risk to the aviation system. To help ensure an objective assessment of the study’s methodology and findings, DHS could benefit from convening an independent panel of experts from outside DHS to determine whether the study’s methodology is sufficiently comprehensive to validate the SPOT program.

DHS also concurred with our second recommendation to conduct a comprehensive risk assessment to determine the effective deployment of SPOT. DHS stated that TSA’s Aviation Modal Risk Assessment is designed to evaluate overall transportation security risk, not deployment strategies. However, DHS noted that TSA is in the process of conducting an initial risk analysis using its risk management analysis tool and plans to update this analysis in the future. However, it is not clear from DHS’s comments how this analysis will incorporate an assessment of TSA’s deployment strategy for SPOT.

DHS also concurred with our third recommendation to perform a cost- benefit analysis of SPOT. DHS noted that TSA is developing an initial cost- benefit analysis and that the flexibility of behavior detection officers already suggests that behavior detection is cost-effective. However, it is not clear from DHS’s comments whether its cost-benefit analysis will include a comparison of the SPOT program with other security screening programs, such as random screening, or already existing security measures as we recommended. Completing its cost-benefit analysis and comparing it to other screening programs should help establish whether the SPOT program is cost-effective compared to other layers of security.

With regard to our fourth recommendation to revise and implement the SPOT strategic plan using risk assessment information, DHS concurred and noted that analysis facilitated by the risk management analysis tool will allow the program to revise the SPOT strategic plan to incorporate the elements identified in our recommendation.

DHS also concurred with our fifth recommendation to study the feasibility of using airport checkpoint-surveillance video recordings to enhance its understanding of terrorist behaviors. DHS noted that TSA agrees this could be a useful tool and is working with DHS’s S&T Directorate to utilize video case studies of terrorists, if possible. These cases studies could help TSA determine what behaviors had been demonstrated by these persons convicted of terrorist-related offenses who went through SPOT airports, and what could be learned from the observed behaviors.

DHS concurred with our sixth recommendation that TSA provide guidance in the SPOT SOP or other directives to BDOs, or other TSA personnel, on how to input data into the Transportation Information Sharing System database. DHS stated that the SPOT SOP is undergoing revision, and that the revised version will provide guidance directing the input of BDO data into the Transportation Information Sharing System. DHS anticipates release of the updated SPOT SOP in fiscal year 2010. DHS also agreed that TSA should set milestones and a time frame for deploying Transportation Information Sharing System access to SPOT airports so that TSA and intelligence community entities have information from all SPOT LEO referrals readily available to assist in “connecting the dots” and identifying potential terror plots. DHS stated that TSA is currently drafting a plan to include milestones and a time frame for deploying System access to all SPOT airports.

DHS concurred with our seventh recommendation to develop a standardized process to allow BDOs or other designated airport officials to send information to TSA’s Transportation Security Operations Center about passengers whose behavior indicates they may pose a threat to security, and to provide guidance on how designated TSA officials are to receive information back from the Center. DHS stated that TSA has convened a working group to address this recommendation. Moreover, TSA is developing a system and procedure for sending and receiving information from the Center and stated that it anticipates having a system in place later in fiscal year 2010.

DHS concurred in principle with regard to our eighth recommendation that the Transportation Security Operations Center utilize all of the databases available to it when conducting checks on passengers who rise to the level of a LEO referral against intelligence and criminal databases. DHS stated that TSA has convened a working group to address this recommendation. According to DHS, this group will conduct a study during fiscal year 2010 to determine the feasibility of fully implementing this recommendation. As such, the study is to review the various authorities, permissions, and limitations of each of the databases or systems cited in our report. DHS stated that access to some of the systems, requires more justification than a BDO referral. Further, according to DHS, because some of the databases or systems contain classified information, TSA will also need to adopt a communication strategy to transmit the passenger information between the BDO and Transportation Security Operations Center. DHS stated that TSA will work on a process to collect the passenger information, verify the passenger’s identity, through checks of databases, and analyze that information to determine if the passenger is the subject of an investigation and may pose a risk to aviation security.

With regard to our ninth recommendation to establish a plan with objectives, milestones, and time frames to develop outcome-oriented performance measures for BDOs, DHS concurred and stated that TSA intends to consult with experts to develop outcome-oriented performance measures.

DHS also concurred with our tenth recommendation to establish controls for SPOT data. DHS noted that TSA established additional controls as part of the SPOT database migration to TSA’s Performance Management Information System and is exploring an additional technology solution to reduce possible errors. As noted in our report, since these changes to the database were not complete at the time of our audit, we could not assess whether the problems we identified with the database had been corrected.

Regarding our eleventh recommendation to establish time frames and milestones to systematically evaluate the SPOT training program on a periodic basis, DHS concurred and stated that TSA intends to develop such a plan following completion of DHS’s S&T Directorate’s BDO Job Task Analysis, and TSA’s training gap analysis, which identifies gaps in the training curriculum.

DHS also commented on the scientific basis underlying SPOT. Specifically, DHS stated that decades of scientific research has shown the SPOT behaviors to be “universal in their manifestation.” However, according to DHS, its S&T Directorate is examining the extent to which behavior indicators are appropriate for screening purposes and lead to appropriate and correct security decisions. DHS also commented that the results of this work, which is currently underway, will establish a scientific basis of the extent to which the SPOT program instruments and methods are valid. Thus, DHS’s comments suggest that additional research is needed to determine whether these behaviors can be used in an airport environment for screening passengers to identify threats to the aviation system.

Moreover, DHS took issue with our use of a report from the National Research Council of the National Academy of Sciences stating that we improperly relied upon this report. We disagree. DHS questioned the findings of the National Research Council report and stated that it lacked sufficient information for its conclusions because it principally focused on privacy as it relates to data mining and behavioral surveillance and was not intended to represent an exhaustive or definitive review of the research or operational literature on behavioral screening, including recent unpublished DHS, defense, and intelligence community studies. DHS also stated that the National Research Council report did not study the SPOT program and that the researchers did not conduct interviews with SPOT personnel.

As we noted in our report, although the National Research Council report addresses broader issues related to privacy and data mining, a senior Council official—and one of the authors of the study—stated that the committee included behavior detection as a focus because any behavior detection program could have privacy implications. This official added that the primary objective of the report was to develop a framework for sound decision making for programs, such as SPOT, and help ensure a sound scientific and legal basis. According to this official, the National Academy of Sciences’ Committee on Technical and Privacy Dimensions of Information for Terrorism Prevention and Other National Goals—which had oversight of the report—was briefed on the SPOT program as part of the study. The Committee also conducted meetings with three experts in behavior detection as part of their research. During the course of our review, we interviewed three Committee members responsible for developing the report’s findings, as well as four other behavior detection experts, including the three who participated in the National Research Council study. Our discussions with these experts corroborated the report’s findings. Thus, we believe that our use of the Council report was an appropriate and a necessary part of our review.

However, the National Research Council report was only one of many sources that we analyzed with regard to the science of behavioral and physiological screening, and its applicability to an airport environment. As we noted in the description of our methodology, our study included interviews with officials from DHS as well as several of its components and other U.S. government agencies—each of which use elements of behavior detection in their daily work. We also interviewed El Al airline officials, a former director of security at Israel’s Ben-Gurion airport, and seven nationally recognized experts in behavior detection as part of our review. Moreover, as we explained in the discussion of our scope and methodology, we conducted a survey about the SPOT program of all 118 Federal Security Directors for all SPOT airports, and conducted site visits to 15 SPOT airports. In addition, we analyzed the SPOT referral database, to the extent the data permitted, covering a 4-year period and the results from 2 billion passengers passing through SPOT airports. Moreover, we attended both the basic and advanced training courses in behavior detection provided by TSA to BDOs, in order to better understand how the program is carried out. Therefore, our analysis of the program was not derived from or based on a single study by the National Research Council as DHS suggested, but rather is based on all of the information we gathered and synthesized from multiple, diverse, expert sources, each of which provided different perspectives about the program, as well as about behavior detection in general.

DHS also disagreed with the accuracy of a statement included in our report that noted DHS S&T could not provide us with specific contacts related to sources of information for certain research it cited as support for the SPOT program. In its comments, DHS stated that it had provided us with all requested documents that represent DHS’s S&T Directorate- sponsored research. We agree. However, DHS did not provide us with contact information for the sources of unpublished studies by the Department of Defense and other intelligence community studies that DHS S&T had cited as support for the SPOT program. Without such information, we are unable to verify the contents of these unpublished studies.

Finally, DHS stated that while we were unable to use the SPOT referral data to assess whether any behavior or combination of SPOT behaviors could be used to reliably predict the final outcome of an incident involving the use of SPOT, it was able to analyze the SPOT referral database successfully after working with TSA to verify scores assigned to different indicators. Our concern with the data did not involve the question of whether some behaviors were entered erroneously, nor whether errors in coding were excessive or non-random. Rather, we were concerned with whether the data on behaviors were complete. Specifically, it cannot be determined from the SPOT referral database whether all behaviors observed were included for each referred passenger by each BDO or whether only the behaviors that were sufficient for a LEO referral were recorded into the database. It is not possible to determine from the database if the number of observed behaviors entered for a given passenger was the total number of observed behaviors, or whether additional behaviors were observed. A rigorous analysis of the relative effects of the different behaviors on the outcomes of the use of SPOT would require each BDO to record, for each of the observable behaviors, whether it was or was not observed.

TSA also provided technical comments that we incorporated as appropriate.

We will send copies of this report to the Secretary of Homeland Security; the TSA Administrator (Acting); and interested congressional committees as appropriate. The report will also be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4379 or lords@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are acknowledged in appendix III.

Appendix I: Scope and Methodology

To determine the extent to which the Transportation Security Administration (TSA) determined whether the Screening of Passengers By Observation Techniques (SPOT) program had a scientifically-validated basis for identifying passengers before deploying it, we reviewed literature on behavior analysis by subject matter experts, interviewed seven experts in behavior analysis, interviewed other federal agencies and entities about how they use behavior detection techniques, and analyzed relevant reports and books on the topic. These included a 2008 study by the National Research Council of the National Academy of Sciences that has a discussion regarding deception and behavioral surveillance, as well as other issues related to behavioral analysis. We interviewed Dr. Herbert S. Lin, who was a primary author of the report, as well as Dr. Robert W. Levenson, and Dr. Stephen E. Fienberg, both members of the Academy committee that oversaw the report, about the report’s findings with regard to behavior detection, and the extent to which behavior detection in a complex environment, such as an airport terminal, has been validated with regard to its effectiveness in identifying persons who may be a risk to aviation security. Other behavior detection experts we consulted were Dr. Paul Ekman; Dr. Mark Frank; Dr. David Givens; Dr. David Matsumoto; and Mr. Rafi Ron, former director of security at Israel’s Ben-Gurion Airport. Dr. Ekman, Dr. Frank, and Mr. Ron provided expert advice for the National Research Council study. Dr. Givens was identified by TSA as having been their principal source for the nonverbal behavior indicators used by the SPOT program. We also interviewed Dr. Lawrence M. Wein, an expert in emergency responses to terror attacks and mathematical models in operations management. In addition, we interviewed officials from the Department of Homeland Security’s (DHS) Science and Technology (S&T) Directorate regarding their ongoing research into behavior detection. Although the views of these experts cannot be generalized across all experts in behavior analysis because we selected individuals based on their publications on behavioral analysis or related topics, their recognized accomplishments and expertise, and, in some cases, TSA’s use of their work or expertise to design and review the SPOT program’s behaviors, they provided us with an overall understanding of the fundamentals of behavior analysis, and how it could be applied.

To determine the basis for TSA’s strategy to develop and deploy SPOT and evaluate to what extent SPOT was informed by a cost-benefit analysis and a strategic plan, we reviewed program documentation, including briefings prepared by the SPOT program office during the course of developing and fielding SPOT, two versions of a strategic plan for SPOT, and the 2009 SPOT standard operating procedures guidance. We compared the plans and analyses used by TSA to develop and implement SPOT to criteria on how to develop and implement programs in DHS’s 2006 Cost Benefit Analysis Guidebook, as well as to Office of Management and Budget guidance on the utility of cost-benefit analyses in program implementation. We also analyzed the development of SPOT in light of the standards and criteria cited in DHS’s 2006 National Infrastructure Protection Plan. We met with relevant TSA officials to discuss these issues. To assess whether DHS developed an effective strategic plan for SPOT prior to implementing the program, we interviewed TSA officials involved in development of the SPOT strategic plan. We analyzed whether the SPOT plan incorporated the desirable characteristics of an effective strategic plan as identified by previous GAO work on what strategic plans should include to be considered effective, such as a risk assessment, cost and resources analysis, and a means for collaboration with other key entities. We also examined it in light of the requirements of the Government Performance and Results Act of 1993, which specifies the elements of strategic plans for government programs. We assessed whether the SPOT strategic plan was followed by TSA. As part of our analysis of the planning for SPOT before it was implemented on a nationwide basis, we reviewed TSA documentation related to the development and pilot testing of SPOT, such as a TSA white paper on SPOT, and interviewed key program officials from both headquarters and field offices.

We also interviewed cognizant officials from other U.S. government agencies and agency entities that utilize behavior detection in their work, including U.S. Customs and Border Protection (CBP), the U.S. Secret Service, the TSA’s Federal Air Marshal Service (FAMS) component, and the Federal Bureau of Investigation (FBI). We sought their views on the utility of various behavior detection methods, their experience with practicing behavior detection, and asked them about the extent to which TSA had consulted with them in developing and implementing the SPOT program.

To better understand how SPOT incorporated expertise about the use of behavior detection in an airport setting, we interviewed officials from Israel’s El Al Airlines, which is cited by TSA as having provided part of the basis of the SPOT program. We asked about El Al’s methods to ensure the security of its passenger aircraft, and also interviewed a former head of security at Israel’s Ben-Gurion airport, who has advised TSA on security issues. We asked TSA and SPOT program officials about their consultations with El Al, and about the ways in which they had utilized El Al’s expertise, as well as about any other entities whose expertise they may have adopted into SPOT.

To determine the challenges, if any, that emerged during implementation of the SPOT program, we interviewed headquarters and field personnel about how the program has utilized the resources available to it to ensure that it is effective. These resources included the support of law enforcement officers (LEOs), to whom passengers are referred by Behavior Detection Officers (BDOs) for additional questioning. In addition, we interviewed SPOT program and TSA officials about the databases available to them at TSA’s Transportation Security Operations Center to determine if a suspect passenger is being sought by other U.S. law enforcement or intelligence entities, and whether there is guidance for BDOs on when and how to contact the Transportation Security Operations Center. We also asked about whether there is guidance and training for BDOs on how to access TSA’s Transportation Information Sharing System database, which is owned by FAMS, and is available through the Transportation Security Operations Center. To determine if any management challenges had emerged related to management controls in developing and implementing SPOT, we compared TSA’s approach for implementing and managing the SPOT program with GAO’s Standards for Internal Control in the Federal Government and with risk management principles we had previously identified. Our legal counsel office reviewed court decisions relevant to the SPOT program. In addition, we interviewed attorneys from the American Civil Liberties Union, and obtained and reviewed TSA’s Privacy Impact Assessments for SPOT, the Transportation Security Operations Center, and the Transportation Information Sharing System. We also met with and discussed relevant privacy and legal issues with TSA’s Offices of Privacy and Civil Rights/Civil Liberties. To obtain data about certain aspects of the SPOT program that the SPOT program office did not have, we conducted a survey of Federal Security Directorswhose responsibilities included security at all 161 SPOT airports at the time of our survey. (Some Federal Security Directors have responsibility for more than one airport.) We obtained a 100 percent response rate. This survey asked, among other things, about whether there were cameras at security checkpoints that record the interactions of Transportation Security Officers (TSO), BDOs, and passengers; if the airport authority had an agreement with TSA that specifies certain law enforcement actions during a SPOT referral; and if there was an agreement, or any other comparable guidance that specified a time limit for LEOs to come to checkpoints after being called for help by BDOs.

To determine the extent to which TSA has measured SPOT’s effect on aviation security, we obtained and analyzed the TSA SPOT referral database, which records all incidents in which BDOs refer passengers to secondary, more intensive questioning, and which also records all incidents in which BDOs chose to refer passengers to LEOs. We found that the SPOT database was sufficiently reliable to count the number of arrests resulting from referrals from BDOs to LEOs, for examining the reasons for each arrest, and for counting the percentage of times that LEOs responded to BDO calls for service, and the length of time required. Use of these data required us to resolve apparent contradictions and anomalies in the database to make the data useable. Because of data problems, we were unable to conduct analyses to assess whether any behavior or combination of behaviors could be used to predict the final outcome of an incident involving the use of SPOT. In addition, we reviewed relevant standardization team reports and observed a standardization team visit in operation.

In addition, we spoke with BDO managers, Federal Security Directors, and Assistant Federal Security Directors to determine how BDOs are evaluated. To do so, we conducted site visits to 15 commercial airports at which BDOs and SPOT have been deployed, or almost 10 percent of the 161 airports with SPOT. We chose these airports taking into account the following criteria, among others: (1) each airport had BDOs deployed, and at each, the SPOT program had been in effect for no less than 3 months; (2) airports were chosen to provide a variety of sizes, as measured in annual passenger volume; physical location within the country (northeast, southwest, central, Pacific Coast, rural, urban); and estimated risk of terrorist incident, using DHS’s Current Airports Threat Assessment list (visiting 6 that were in the top 10, and others much lower); (3) BDOs who are employed by contractors, rather than employed directly by TSA; and (4) airports with LEOs who were identified to us by TSA as having received some form of behavior detection training and airports where they were not known to have received such training. In addition, we took into account the location of the airports with regard their proximity to subject matter experts on behavior detection whom we wished to interview, as well as the time and cost required to reach certain airports.

At each of the airports we visited, we interviewed cognizant officials, including the Federal Security Director or Assistant assigned to the airport, the BDO program manager, one or two BDOs, and one or two LEOs who have interacted with BDOs. Since each of these airports differs in terms of passenger volume, physical size and layout, geographic location, and potential value as a target for terrorism, among other things, the results from these visits are not generalizable to other airports. However, these visits provided helpful insight into the operation of SPOT at airports.

In addition, to determine if individuals had transited SPOT airports who were later charged with or pleaded guilty to terrorism-related offenses, we reviewed information contained in (1) the Treasury Enforcement Communication System II database maintained by CBP; (2) Department of Justice information and court documents, including indictments and related documents; and (3) media accounts of individuals accused of terrorism-related activities. We compared information pertaining to these individuals’ dates of transit to the dates when SPOT was deployed to the various airports identified in the Treasury Enforcement Communication System and Justice Department data to determine if SPOT had been deployed at a given airport when the transits occurred. Further, we used our survey of Federal Security Directors at SPOT airports to determine the extent to which video surveillance cameras are present at checkpoints.

To assess the extent that SPOT training incorporates the attributes of an effective training program, we had training experts at TSA headquarters complete a training assessment tool that we developed using our prior work for assessing training courses and curricula. To address training- related issues, including to understand better how other entities train their employees in behavior detection, and what their curricula include, we conducted site visits to the Secret Service, FAMS, CBP, and the FBI, and also interviewed nongovernmental experts on behavior detection (our selection of these experts is discussed above). As part of our assessment of SPOT training, we attended the basic SPOT training course given to BDOs, as well as the advanced SPOT course on behavior detection. We interviewed BDOs and BDO managers about the SPOT training, as well as officials of El Al airlines, with regard to how El Al trains and tests its personnel who utilize behavior recognition and analysis as part of their assessment of El Al passengers.

We conducted this performance audit from May 2008 through May 2010, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: DHS Comments

Appendix III: GAO Contacts and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, David M. Bruno, Assistant Director, and Jonathan R. Tumin, managed this assignment. Ryan Consaul, Jeff C. Jensen, Kevin Remondini, and Julie E. Silvers made significant contributions to the work. Arthur James, Jr., Amanda Miller, and Douglas Sloane assisted with design, methodology, and data analysis. Chris Dionis assisted with issues related to training. Katherine Davis and Debra Sebastian provided assistance in report preparation; Tracey King and Tom Lombardi provided legal support; and Pille Anvelt and Barbara Hills developed the report graphics.