Aviation Security: TSA Is Taking Steps to Validate the Science Underlying Its Passenger Behavior Detection Program, but Efforts May Not Be Comprehensive
Published: 2011-04-06T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The attempted passenger aircraft bombing of Northwest flight 253 on December 25, 2009, provided a vivid reminder that the civil aviation system remains an attractive terrorist target. To enhance aviation security, in October 2003 the Department of Homeland Security's (DHS) Transportation Security Administration (TSA) began testing of its Screening of Passengers by Observation Techniques (SPOT) program to identify persons who may pose a risk to aviation security. The SPOT program utilizes behavior observation and analysis techniques to identify potentially high-risk passengers. This testimony provides information on (1) the extent to which TSA has validated the scientific basis for SPOT and (2) other operational challenges. This statement is based on a prior report GAO issued in May 2010 on SPOT, including selected updates made in March 2011. For the updates, GAO reviewed documentation on TSA's progress in implementing the report's recommendations.
section_title: What GAO Found
paragraphs: As GAO reported in May 2010, TSA deployed its behavior detection program nationwide before first determining whether there was a scientifically valid basis for the program. According to TSA, the program was deployed before a scientific validation of the program was completed in response to the need to address potential security threats. However, a scientific consensus does not exist on whether behavior detection principles can be reliably used for counterterrorism purposes, according to a 2008 report of the National Research Council of the National Academy of Sciences. DHS is conducting a study on the scientific basis of SPOT. Thus, in May 2010, GAO recommended that DHS convene an independent panel of experts to review the methodology of its study. DHS concurred and stated that it is convening an independent panel to review its current efforts to help validate the scientific basis for the program, which is expected to complete its work by early April 2011. Nonetheless, DHS's study to assess SPOT is not designed to fully validate whether behavior detection can be used to reliably identify individuals in an airport environment who pose a security risk. For example, factors such as the length of time behavior detection officers (BDO) can observe passengers without becoming fatigued are not part of the plan and could provide additional information on the extent to which SPOT can be effectively implemented. The results of a panel to review DHS's methodology could help ensure a rigorous, scientific validation of SPOT. As GAO previously reported, TSA experienced SPOT operational challenges, including not systematically collecting and analyzing information obtained by BDOs on passengers who may pose a threat to the aviation system. Better utilizing existing resources would enhance TSA's ability to quickly verify passenger identity and could help TSA to more reliably "connect the dots" with regard to persons who pose a threat. Thus, GAO recommended that TSA clarify BDO guidance for inputting information into the database used to track suspicious activities, and develop a schedule to expand access to this database across all SPOT airports. TSA agreed and in March 2011 stated that it has revised the SPOT standard operating procedures on how BDOs are to input data into the database used to report suspicious activities. TSA plans to implement these revised procedures in April 2011. TSA also reported that all SPOT airports have access to this database as of March 2011. In addition, GAO reported that individuals allegedly involved in six terrorist plots transited SPOT airports. GAO recommended in May 2010 that TSA study the feasibility of using airport video recordings of the behaviors exhibited by persons transiting airport checkpoints who were later charged with or pleaded guilty to terrorism-related offenses. GAO reported that such recordings could provide insights about behaviors that may be common among terrorists or could demonstrate that terrorists do not generally display any identifying behaviors. TSA agreed that studying airport videos could be a useful tool in understanding terrorist behaviors in the airport environment and in March 2011 reported that it is exploring ways to better utilize such recordings.
section_title: What GAO Recommends
paragraphs: GAO has made recommendations in prior work to strengthen TSA's SPOT program. TSA generally concurred with the recommendations and has actions under way to address them. GAO provided the updated information to TSA. TSA had no comment.

section_title: Letter
paragraphs: I appreciate the opportunity to participate in today’s hearing to discuss the  Transportation Security Administration’s (TSA) behavior-based passenger  screening program known as the Screening of Passengers by Observation  Techniques (SPOT) program. The attempted U.S. passenger aircraft  bombing of Northwest flight 253 on December 25, 2009, provided a vivid  reminder that civil aviation remains an attractive terrorist target and  underscores the need for effective passenger screening. To help enhance  aviation security, in October 2003, the Department of Homeland Security’s  (DHS) TSA began testing its SPOT program to identify persons who may  pose a risk to aviation security. The SPOT program utilizes behavior  observation and analysis techniques to identify potentially high-risk  passengers. TSA designed SPOT to provide behavior detection officers  (BDO) with a means of identifying persons who may pose a potential  security risk at TSA-regulated airports by focusing on behaviors and  appearances that deviate from an established baseline and that may be  indicative of stress, fear, or deception.
In instances when a passenger’s SPOT indicators place him or her above a  numerical threshold, he or she will be directed to the second step of SPOT,  referral screening. This involves additional questioning and physical  search of his or her person and property by BDOs and transportation  security officers. This referral screening occurs in the checkpoint area. A  referral to a law enforcement officer (LEO) is a potential third step in the  SPOT process. After a passenger has been referred by the BDOs to a LEO,  the LEO is then expected to independently determine, through additional  investigation, such as questioning the passenger and, if appropriate,  conducting an identity verification and background check through the  Federal Bureau of Investigation’s (FBI) National Crime Information Center  (NCIC), whether sufficient grounds exist to take further action, such as  detaining or arresting the passenger. BDOs have been selectively deployed  to 161 of the 462 TSA-regulated airports in the United States. The  conference report accompanying the fiscal year 2010 DHS appropriations  act provided $211.9 million for the SPOT program. The administration has  requested $232 million for SPOT for fiscal year 2011, a $20.2 million (9.5  percent) increase over the fiscal year 2010 funding level, to support 3,350  BDOs. If this increase is appropriated, TSA will have invested over $800  million in the program since fiscal year 2007. In addition, DHS has  requested about $254 million, a $21.9 million increase, in fiscal year 2012  to support an additional 350 BDOs.
My statement today discusses TSA’s and DHS’s efforts to validate the  scientific basis of the SPOT program, as well as steps that TSA is taking to  address operational challenges in deploying SPOT to airports. My  comments are based primarily on our May 2010 report. It also includes  selective updates we obtained in March 2011. For our May 2010 report, we  reviewed relevant literature on behavior analysis by subject matter  experts. This included a 2008 study by the National Research Council of  the National Academy of Sciences that included a discussion section on  the issue of deception and behavioral surveillance, as well as other issues  related to behavioral analysis. We interviewed recognized experts in the  field, as well as cognizant officials from other U.S. government agencies  that utilize behavior analysis in their work, including U.S. Customs and  Border Protection (CBP), the U.S. Secret Service, the Federal Air Marshal  Service (FAMS), and the FBI. To better understand how SPOT  incorporated expertise on behavior analysis for aviation security, we also  interviewed current and retired officials of Israel’s El Al Airlines, whose  security processes TSA cites as providing part of the basis of the SPOT  program. To identify any challenges that emerged during implementation  of the SPOT program, we conducted field site visits to 15 TSA-regulated  airports with SPOT, which represent almost 10 percent of the 161 TSA- regulated airports with SPOT to observe operations and meet with key  program personnel. To obtain comparative data on how SPOT had been  implemented at different airports across the nation, we conducted a  survey of all federal security directors responsible for security operations  at TSA-regulated airports with SPOT. We obtained a 100 percent response  rate. In addition, to determine if individuals who were later charged with  or pleaded guilty to terrorism-related offenses had transited SPOT airports  and whether TSA could obtain information from these transits to enhance  its understanding of terrorist behaviors, we reviewed CBP and Department  of Justice information to (1) identify individuals who were charged with or  pleaded guilty to terrorism-related offenses and (2) determine if these  individuals had, prior to being charged, transited airports where SPOT had  been deployed. For the updates, we reviewed documentation from TSA on  the steps it has taken to implement the recommendations from our May  2010 report. More detailed information about our scope and methodology  is included in our May 2010 report. We conducted this work in accordance  with generally accepted government auditing standards.
subsections: section_title: TSA Did Not Validate the Science Underlying the SPOT Program before Deploying SPOT
paragraphs: As discussed in our May 2010 report, TSA deployed SPOT nationwide  before first determining whether there was a scientifically valid basis for  using behavior and appearance indicators as a means for reliably  identifying passengers who may pose a risk to the U.S. aviation system. A  validation study by DHS’s Science and Technology Directorate is under  way now, but questions exist regarding whether the study’s methodology  is sufficiently comprehensive to validate the SPOT program. Specifically,  DHS’s plan to assess SPOT is not designed to fully validate whether  behavior detection can be used to reliably identify individuals in an airport  environment who pose a security risk. The results of an independent  assessment are needed to determine whether current validation efforts are  sufficiently comprehensive to validate the program, and to support future  requests for increased funding.
According to TSA, SPOT was deployed before a scientific validation of the  program was completed, but TSA stated that this deployment was made in  response to the need to address potential threats to the aviation system,  such as suicide bombers. TSA also stated that the program was based  upon scientific research available at the time regarding human behaviors.
Moreover, TSA stated that no other large-scale U.S. or international  screening program incorporating behavior- and appearance-based  indicators has ever been rigorously scientifically validated.
However, a 2008 report issued by the National Research Council of the  National Academy of Sciences stated that the scientific evidence for  behavioral monitoring is preliminary in nature. The report also noted that  an information-based program, such as a behavior detection program,  should first determine if a scientific foundation exists and use  scientifically valid criteria to evaluate its effectiveness before deployment.  The report added that such programs should have a sound experimental  basis and that the documentation on the program’s effectiveness should be  reviewed by an independent entity capable of evaluating the supporting  scientific evidence.
As we reported in May 2010, an independent panel of experts could help  DHS develop a comprehensive methodology to determine if the SPOT  program is based on valid scientific principles that can be effectively  applied in an airport environment for counterterrorism purposes. Thus, we  recommended that the Secretary of Homeland Security convene an  independent panel of experts to review the methodology of the validation  study on the SPOT program being conducted to determine whether the  study’s methodology is sufficiently comprehensive to validate the SPOT  program. We also recommended that this assessment include appropriate  input from other federal agencies with expertise in behavior detection and  relevant subject matter experts. DHS concurred and stated that its  current validation study includes an independent review of the study th at  will include input from a broad range of federal and operational agencies  and relevant experts, including those from academia. According to DH S’s  Science and Technology Directorate, this independent review is expected  to be completed in early April 2011.
As discussed in our May 2010 report, DHS has contracted with the  American Institutes for Research to conduct its validation study. DHS  stated that the ongoing independent review will include, among other  things, recommendations on additional studies that should be undertaken  to more fully validate the science underlying the SPOT screening process.  As we noted in our report, research on other issues, such as determining  the number of individuals needed to observe a given number of passengers  moving at a given rate per day in an airport environment or the duration  that such observation can be conducted by BDOs before observation  fatigue affects effectiveness, could provide additional information on the  extent to which SPOT can be effectively implemented in airports.  Additional research could also help determine the need for periodic  refresher training for the BDOs since research has not yet determined  whether behavior detection is easily forgotten or can be potentially  degraded with time or lack of use. Because such questions exist, the  results of an independent panel of experts to assess the methodology of  the study could provide DHS with additional assurance regarding whether  the study’s methodology is sufficiently comprehensive to validate the  SPOT program.
Moreover, DHS stated that its current effort to validate the science  underlying SPOT includes 3 years of operational SPOT referral data and  preliminary results indicate that it is supportive of SPOT. However, in May  2010, we reported weaknesses in TSA’s process for maintaining  operational data from the SPOT program database. Because of these data- related issues, we reported that meaningful analyses could not be  conducted to determine if there is an association between certain  behaviors and the likelihood that a person displaying certain behaviors  would be referred to a law enforcement officer or whether any behavior or  combination of behaviors could be used to distinguish deceptive from  nondeceptive individuals.
As we reported in March 2011, Congress may wish to consider limiting  program funding pending receipt of an independent assessment of TSA’s  SPOT program. We identified potential budget savings of about $20  million per year if funding were frozen at current levels until validation  efforts are complete. Specifically, in the near term, we reported that  Congress could consider freezing appropriation levels for the SPOT  program at the 2010 level until the validation effort is completed.  Assuming that TSA is planning to expand the program at a similar rate  each year, this action could result in possible savings of about $20 million  per year, or $100 million over 5 years, since TSA is seeking about a $20  million increase for SPOT in fiscal year 2011. We also reported that upon  completion of the validation effort, Congress may also wish to consider  the study’s results—including those on the program’s effectiveness in  using behavior-based screening techniques to detect terrorists in the  aviation environment—in making future funding decisions regarding the  program.
section_title: TSA Is Taking Steps to Address Operational Challenges in Implementing the SPOT Program
paragraphs: In May 2010, we reported that TSA is not fully utilizing the resources it has  available to systematically collect the information obtained by BDOs on  passengers whose behaviors and appearances resulted in either a referral  to a BDO or to a LEO, and who thus may pose a risk to the aviation  system. As we previously reported, TSA does not provide official guidance  on how or when BDOs or other TSA personnel should enter data into the  Transportation Information Sharing System or which data should be  entered. Official guidance on what data should be entered into the  system on passengers could better position TSA personnel to be able to  consistently collect information to facilitate synthesis and analysis in  “connecting the dots” with regard to persons who may pose a threat to the  aviation system.
Moreover, as of May 2010, TSA had not developed a schedule or  milestones by which database access would be deployed to SPOT airports,  or a date by which access at all SPOT airports would be completed.  Setting milestones for expanding Transportation Information Sharing  System access to all SPOT airports, and setting a date by which the  expansion will be completed, could better position TSA to identify threats  to the aviation system that may otherwise go undetected and help TSA  track its progress in expanding Transportation Information Sharing  System access as management intended. Thus, we previously  recommended that TSA provide guidance in the SPOT standard operating  procedures or other directives to BDOs, and to other TSA personnel as  appropriate, on how and when to input data into the Transportation  Information Sharing System database. In March 2011, TSA stated that it  has taken steps to implement our recommendation by revising SPOT  standard operating procedures to provide guidance directing the input of  BDO data into the Transportation Information Sharing System. TSA plans  to implement these revised procedures in April 2011. In addition, all SPOT  airports have access to the Transportation Information Sharing System as  of March 2011 according to TSA.
In addition, as we previously reported, studying airport video recordings  of the behaviors exhibited by persons transiting airport checkpoints who  were later charged with or pleaded guilty to terrorism-related offenses  could provide important insights about behaviors that may be common  among terrorists or could demonstrate that terrorists do not generally  display any identifying behaviors. In addition, such images could help  determine if BDOs are looking for the right behaviors or seeing the  behaviors they have been trained to observe.
Using CBP and Department of Justice information, we examined the travel  of key individuals allegedly involved in six terrorist plots that have been  uncovered by law enforcement agencies. We determined that at least 16  of the individuals allegedly involved in these plots moved through 8  different airports where the SPOT program had been implemented. Six of  the 8 airports were among the 10 highest-risk airports, as rated by TSA in  its Current Airport Threat Assessment. In total, these individuals moved  through SPOT airports on at least 23 different occasions. For example,  according to Department of Justice documents, in December 2007 an  individual who later pleaded guilty to providing material support to Somali  terrorists boarded a plane at the Minneapolis-Saint Paul International  Airport en route to Somalia. Similarly, in August 2008, an individual who  later pleaded guilty to providing material support to al Qaeda boarded a  plane at Newark Liberty International Airport en route to Pakistan to  receive terrorist training to support his efforts to attack the New York  subway system.
Our survey of federal security directors at 161 SPOT airports indicated  that most checkpoints at SPOT airports have surveillance cameras  installed. Thus, we reported that TSA may be able to utilize the  information collected from the video infrastructure at the nation’s airports  to study the behavior of persons who were later charged with or pleaded  guilty to terrorism-related offenses to help improve and refine the existing  SPOT program. As a result, in our May 2010 report, we recommended that  if the current validation effort determines that the SPOT program has a  scientifically validated basis for using behavior detection for  counterterrorism purposes in the airport environment, then TSA should  study the feasibility of using airport checkpoint surveillance video  recordings to enhance its understanding of terrorist behaviors. DHS  agreed with our recommendation and noted that TSA agrees this could be  a useful tool and is working with DHS’s Science and Technology  Directorate to utilize video case studies of terrorists, if possible. TSA  officials agreed that examining video recordings of individuals who were  later charged with or pleaded guilty to terrorism-related offenses, as they  used the aviation system to travel to overseas locations allegedly to  receive terrorist training or to execute attacks, could help inform the  SPOT program’s identification of behavioral indicators. In March 2011,  TSA stated that it is exploring ways to better utilize video recordings to  identify these behavioral indicators.
Chairman Broun, Ranking Member Edwards, and Members of the  Subcommittee, this concludes my statement. I look forward to answering  any questions that you may have at this time.
section_title: Contacts and Acknowledgments
paragraphs: For questions about this statement, please contact Stephen M. Lord at  (202) 512-4379 or lords@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this statement. Individuals making key contributions to this testimony  are David M. Bruno, Assistant Director; Ryan Consaul; Katherine Davis;  Emily Gunn; and Tracey King.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.