Aviation Security: TSA Is Taking Steps to Validate the Science Underlying Its Passenger Behavior Detection Program, but Efforts May Not Be Comprehensive
Released: 2011-04-06T00:00:00

Letter

TSA Did Not Validate the Science Underlying the SPOT Program before Deploying SPOT

As discussed in our May 2010 report, TSA deployed SPOT nationwide before first determining whether there was a scientifically valid basis for using behavior and appearance indicators as a means for reliably identifying passengers who may pose a risk to the U.S. aviation system. A validation study by DHS’s Science and Technology Directorate is under way now, but questions exist regarding whether the study’s methodology is sufficiently comprehensive to validate the SPOT program. Specifically, DHS’s plan to assess SPOT is not designed to fully validate whether behavior detection can be used to reliably identify individuals in an airport environment who pose a security risk. The results of an independent assessment are needed to determine whether current validation efforts are sufficiently comprehensive to validate the program, and to support future requests for increased funding.

According to TSA, SPOT was deployed before a scientific validation of the program was completed, but TSA stated that this deployment was made in response to the need to address potential threats to the aviation system, such as suicide bombers. TSA also stated that the program was based upon scientific research available at the time regarding human behaviors.

Moreover, TSA stated that no other large-scale U.S. or international screening program incorporating behavior- and appearance-based indicators has ever been rigorously scientifically validated.

However, a 2008 report issued by the National Research Council of the National Academy of Sciences stated that the scientific evidence for behavioral monitoring is preliminary in nature. The report also noted that an information-based program, such as a behavior detection program, should first determine if a scientific foundation exists and use scientifically valid criteria to evaluate its effectiveness before deployment. The report added that such programs should have a sound experimental basis and that the documentation on the program’s effectiveness should be reviewed by an independent entity capable of evaluating the supporting scientific evidence.

As we reported in May 2010, an independent panel of experts could help DHS develop a comprehensive methodology to determine if the SPOT program is based on valid scientific principles that can be effectively applied in an airport environment for counterterrorism purposes. Thus, we recommended that the Secretary of Homeland Security convene an independent panel of experts to review the methodology of the validation study on the SPOT program being conducted to determine whether the study’s methodology is sufficiently comprehensive to validate the SPOT program. We also recommended that this assessment include appropriate input from other federal agencies with expertise in behavior detection and relevant subject matter experts. DHS concurred and stated that its current validation study includes an independent review of the study th at will include input from a broad range of federal and operational agencies and relevant experts, including those from academia. According to DH S’s Science and Technology Directorate, this independent review is expected to be completed in early April 2011.

As discussed in our May 2010 report, DHS has contracted with the American Institutes for Research to conduct its validation study. DHS stated that the ongoing independent review will include, among other things, recommendations on additional studies that should be undertaken to more fully validate the science underlying the SPOT screening process. As we noted in our report, research on other issues, such as determining the number of individuals needed to observe a given number of passengers moving at a given rate per day in an airport environment or the duration that such observation can be conducted by BDOs before observation fatigue affects effectiveness, could provide additional information on the extent to which SPOT can be effectively implemented in airports. Additional research could also help determine the need for periodic refresher training for the BDOs since research has not yet determined whether behavior detection is easily forgotten or can be potentially degraded with time or lack of use. Because such questions exist, the results of an independent panel of experts to assess the methodology of the study could provide DHS with additional assurance regarding whether the study’s methodology is sufficiently comprehensive to validate the SPOT program.

Moreover, DHS stated that its current effort to validate the science underlying SPOT includes 3 years of operational SPOT referral data and preliminary results indicate that it is supportive of SPOT. However, in May 2010, we reported weaknesses in TSA’s process for maintaining operational data from the SPOT program database. Because of these data- related issues, we reported that meaningful analyses could not be conducted to determine if there is an association between certain behaviors and the likelihood that a person displaying certain behaviors would be referred to a law enforcement officer or whether any behavior or combination of behaviors could be used to distinguish deceptive from nondeceptive individuals.

As we reported in March 2011, Congress may wish to consider limiting program funding pending receipt of an independent assessment of TSA’s SPOT program. We identified potential budget savings of about $20 million per year if funding were frozen at current levels until validation efforts are complete. Specifically, in the near term, we reported that Congress could consider freezing appropriation levels for the SPOT program at the 2010 level until the validation effort is completed. Assuming that TSA is planning to expand the program at a similar rate each year, this action could result in possible savings of about $20 million per year, or $100 million over 5 years, since TSA is seeking about a $20 million increase for SPOT in fiscal year 2011. We also reported that upon completion of the validation effort, Congress may also wish to consider the study’s results—including those on the program’s effectiveness in using behavior-based screening techniques to detect terrorists in the aviation environment—in making future funding decisions regarding the program.

TSA Is Taking Steps to Address Operational Challenges in Implementing the SPOT Program

In May 2010, we reported that TSA is not fully utilizing the resources it has available to systematically collect the information obtained by BDOs on passengers whose behaviors and appearances resulted in either a referral to a BDO or to a LEO, and who thus may pose a risk to the aviation system. As we previously reported, TSA does not provide official guidance on how or when BDOs or other TSA personnel should enter data into the Transportation Information Sharing System or which data should be entered. Official guidance on what data should be entered into the system on passengers could better position TSA personnel to be able to consistently collect information to facilitate synthesis and analysis in “connecting the dots” with regard to persons who may pose a threat to the aviation system.

Moreover, as of May 2010, TSA had not developed a schedule or milestones by which database access would be deployed to SPOT airports, or a date by which access at all SPOT airports would be completed. Setting milestones for expanding Transportation Information Sharing System access to all SPOT airports, and setting a date by which the expansion will be completed, could better position TSA to identify threats to the aviation system that may otherwise go undetected and help TSA track its progress in expanding Transportation Information Sharing System access as management intended. Thus, we previously recommended that TSA provide guidance in the SPOT standard operating procedures or other directives to BDOs, and to other TSA personnel as appropriate, on how and when to input data into the Transportation Information Sharing System database. In March 2011, TSA stated that it has taken steps to implement our recommendation by revising SPOT standard operating procedures to provide guidance directing the input of BDO data into the Transportation Information Sharing System. TSA plans to implement these revised procedures in April 2011. In addition, all SPOT airports have access to the Transportation Information Sharing System as of March 2011 according to TSA.

In addition, as we previously reported, studying airport video recordings of the behaviors exhibited by persons transiting airport checkpoints who were later charged with or pleaded guilty to terrorism-related offenses could provide important insights about behaviors that may be common among terrorists or could demonstrate that terrorists do not generally display any identifying behaviors. In addition, such images could help determine if BDOs are looking for the right behaviors or seeing the behaviors they have been trained to observe.

Using CBP and Department of Justice information, we examined the travel of key individuals allegedly involved in six terrorist plots that have been uncovered by law enforcement agencies. We determined that at least 16 of the individuals allegedly involved in these plots moved through 8 different airports where the SPOT program had been implemented. Six of the 8 airports were among the 10 highest-risk airports, as rated by TSA in its Current Airport Threat Assessment. In total, these individuals moved through SPOT airports on at least 23 different occasions. For example, according to Department of Justice documents, in December 2007 an individual who later pleaded guilty to providing material support to Somali terrorists boarded a plane at the Minneapolis-Saint Paul International Airport en route to Somalia. Similarly, in August 2008, an individual who later pleaded guilty to providing material support to al Qaeda boarded a plane at Newark Liberty International Airport en route to Pakistan to receive terrorist training to support his efforts to attack the New York subway system.

Our survey of federal security directors at 161 SPOT airports indicated that most checkpoints at SPOT airports have surveillance cameras installed. Thus, we reported that TSA may be able to utilize the information collected from the video infrastructure at the nation’s airports to study the behavior of persons who were later charged with or pleaded guilty to terrorism-related offenses to help improve and refine the existing SPOT program. As a result, in our May 2010 report, we recommended that if the current validation effort determines that the SPOT program has a scientifically validated basis for using behavior detection for counterterrorism purposes in the airport environment, then TSA should study the feasibility of using airport checkpoint surveillance video recordings to enhance its understanding of terrorist behaviors. DHS agreed with our recommendation and noted that TSA agrees this could be a useful tool and is working with DHS’s Science and Technology Directorate to utilize video case studies of terrorists, if possible. TSA officials agreed that examining video recordings of individuals who were later charged with or pleaded guilty to terrorism-related offenses, as they used the aviation system to travel to overseas locations allegedly to receive terrorist training or to execute attacks, could help inform the SPOT program’s identification of behavioral indicators. In March 2011, TSA stated that it is exploring ways to better utilize video recordings to identify these behavioral indicators.

Chairman Broun, Ranking Member Edwards, and Members of the Subcommittee, this concludes my statement. I look forward to answering any questions that you may have at this time.

Contacts and Acknowledgments

For questions about this statement, please contact Stephen M. Lord at (202) 512-4379 or lords@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this testimony are David M. Bruno, Assistant Director; Ryan Consaul; Katherine Davis; Emily Gunn; and Tracey King.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.