Available Evidence Does Not Support Whether Behavioral Indicators Can Be Used to Identify Aviation Security Threats

In November 2013, we reported that (1) peer-reviewed, published  research we reviewed did not support whether nonverbal behavioral  indicators can be used to reliably identify deception, (2) methodological  issues limited the usefulness of DHS’s April 2011 SPOT validation study,  and (3) variation in referral rates raised questions about the use of  indicators.

		Published Research on Behavioral Indicators

In November 2013, we reported that our review of meta-analyses (studies  that analyze other studies and synthesize their findings) that included  findings from over 400 studies related to detecting deception conducted  over the past 60 years, other academic and government studies, and  interviews with experts in the field, called into question the use of  behavior observation techniques, that is, human observation unaided by  technology, as a means for reliably detecting deception. The meta- analyses we reviewed collectively found that the ability of human  observers to accurately identify deceptive behavior based on behavioral  cues or indicators is the same as or slightly better than chance (54  percent). We also reported on other studies that do not support the use  of behavioral indicators to identify mal-intent or threats to aviation. In  commenting on a draft of our November 2013 report, DHS stated that one  of these studies, a 2013 RAND report, provides evidence that supports  the SPOT program. However, the RAND report, which concludes that  there is current value and unrealized potential for using behavioral  indicators as part of a system to detect attacks, refers to behavioral  indicators that are defined and used significantly more broadly than those  in the SPOT program. The indicators reviewed in the RAND report are  not used in the SPOT program, and, according to the RAND report’s  findings, could not be used in real time in an airport environment.

		DHS’s Validation Study

Further, in November 2013, we found that DHS’s April 2011 validation  study does not demonstrate effectiveness of the SPOT behavioral  indicators because of methodological weaknesses. The validation study  found, among other things, that some SPOT indicators were predictive of  outcomes that represent high-risk passengers, and that SPOT  procedures, which rely on the SPOT behavioral indicators, were more  effective than a random selection protocol implemented by BDOs in  identifying outcomes that represent high-risk passengers. While the April  2011 SPOT validation study is a useful initial step and, in part, addressed  issues raised in our May 2010 report, methodological weaknesses limit its  usefulness. Specifically, as we reported in November 2013, these  weaknesses include, among other things, the use of potentially unreliable  data and issues related to one of the study’s outcome measures.
First, the data the study used to determine the extent to which the SPOT  behavioral indicators led to correct screening decisions at checkpoints  were from the SPOT database that we had previously found in May 2010  to be potentially unreliable. In 2010, we found, among other things, that  BDOs could not record all behaviors observed in the SPOT database  because the database limited entry to eight behaviors, six signs of  deception, and four types of serious prohibited items per passenger  referred for additional screening, though BDOs are trained to identify 94  total indicators.subsequent to our May 2010 report, the validation study used data that  were collected from 2006 through 2010, prior to TSA’s improvements to  the SPOT database. Consequently, the data were not sufficiently reliable  for use in conducting a statistical analysis of the association between the  indicators and high-risk passenger outcomes.
Although TSA made changes to the database  Second, our analysis of the validation study data regarding one of the  primary high-risk outcome measures—LEO arrests—suggests that the  screening process was different for passengers depending on whether  they were selected using SPOT procedures or the random selection  protocol. Specifically, different levels of criteria were used to determine  whether passengers in each group were referred to a LEO, which is a  necessary precondition for an arrest. Because of this discrepancy  between the study groups, the results related to the LEO arrest metric are  questionable and cannot be relied upon to demonstrate the effectiveness  of the SPOT program’s behavioral indicators. In November 2013, we also  reported on other methodological weaknesses, including design  limitations and monitoring weaknesses, that could have affected the  usefulness of the validation study’s results in determining the  effectiveness of the SPOT program’s behavioral indicators.

		Variation in Referral Rates and TSA Efforts to Study Indicators

In November 2013, we reported that variation in referral rates and  subjective interpretation of the behavioral indicators raise questions about  the use of indicators, but TSA has efforts under way to study the  indicators. Specifically, we found that SPOT referral data from fiscal years  2011 and 2012 indicate that SPOT and LEO referral rates vary  significantly across BDOs at some airports, which raises questions about  the use of SPOT behavioral indicators by BDOs. The rate at which  BDOs referred passengers for SPOT referral screening ranged from 0 to  26 referrals per 160 hours worked during the 2-year period we  reviewed. Similarly, the rate at which BDOs referred passengers to   In November 2013, we  LEOs ranged from 0 to 8 per 160 hours worked.also reported that BDOs and TSA officials we interviewed said that some  of the behavioral indicators are subjective and TSA has not demonstrated  that BDOs can consistently interpret the behavioral indicators. We found  that there is a statistically significant relationship between the length of  time an individual has been a BDO and the number of SPOT referrals the  individual makes. This suggests that different levels of experience may be  one reason why BDOs apply the behavioral indicators differently. TSA  has efforts underway to better define the behavioral indicators currently  used by BDOs, and to complete an inter-rater reliability study. The inter- rater reliability study could help TSA determine whether BDOs can  consistently and reliably interpret the behavioral indicators, which is a  critical component of validating the SPOT program’s results and ensuring  that the program is implemented consistently. According to TSA, the  current contract to study the indicators and the inter-rater reliability study  will be completed in 2014.

	TSA Has Limited Information to Evaluate SPOT Program Effectiveness, but Plans to Collect Additional Performance Data

In November 2013, we reported that TSA plans to collect and analyze  additional performance data needed to assess the effectiveness of its  behavior detection activities. In response to a recommendation in our May  2010 report to develop a plan for outcome-based performance measures,  TSA completed a performance metrics plan in November 2012. The  plan defined an ideal set of 40 metrics within three major categories that  TSA needs to collect to measure the performance of its behavior  detection activities. As of June 2013, TSA had collected some information  for 18 of 40 metrics the plan identified, but the agency was collecting little  to none of the data required to assess the performance and security  effectiveness of its behavior detection activities or the SPOT program  specifically. For example, TSA did not and does not currently collect the  data required to determine the number of passengers meaningfully  assessed by BDOs, BDOs’ level of fatigue, or the impact that fatigue has  on their performance.
To address these and other deficiencies, the performance metrics plan  identifies 22 initiatives that are under way or planned as of November  2012. For example, in May 2013, TSA began to implement a new data  collection system, BDO Efficiency and Accountability Metrics, designed to  track and analyze BDO daily operational data, including BDO locations  and time spent performing different activities. According to TSA officials,  these data will allow the agency to gain insight on how BDOs are utilized,  and improve analysis of the SPOT program. However, according to the  performance metrics plan, TSA will require at least an additional 3 years  and additional resources before it can begin to report on the performance  and security effectiveness of its behavior detection activities or the SPOT  program.
Without the data needed to assess the effectiveness of behavior  detection activities or the SPOT program, we reported in November 2013  that TSA uses SPOT referral, LEO referral, and arrest statistics to help  track the program’s activities. As shown in figure 1, of the approximately  61,000 SPOT referrals made during fiscal years 2011 and 2012 at the 49  airports we analyzed, approximately 8,700 (13.6 percent) resulted in a  referral to a LEO. Of the SPOT referrals that resulted in a LEO referral,  365 (4 percent) resulted in an arrest.
TSA has taken a positive step toward determining the effectiveness of its  behavior detection activities by developing the performance metrics plan,  as we recommended in May 2010. However, as we reported in November  2013, TSA cannot demonstrate the effectiveness of its behavior detection  activities, and available evidence does not support whether behavioral  indicators can be used to identify threats to aviation security. According to  Office of Management and Budget (OMB) guidance accompanying the  fiscal year 2014 budget, it is incumbent upon agencies to use resources  on programs that have been rigorously evaluated and determined to be  effective, and to fix or eliminate those programs that have not  demonstrated results. As we concluded in our November 2013 report,  until TSA can provide scientifically validated evidence demonstrating that  behavioral indicators can be used to identify passengers who may pose a  threat to aviation security, the agency risks funding activities that have not  been determined to be effective. Therefore, in our November 2013 report,  we recommended that TSA limit future funding for its behavior detection  activities. DHS did not concur with our recommendation.
The negatively and significantly related indicators were more commonly associated with  passengers who were not identified as high-risk, than with passengers who were identified  as high-risk. available. However, as described in the report, in addition to the meta- analyses of over 400 studies related to detecting deception conducted  over the past 60 years that we reviewed, we also reviewed several  documents on behavior detection research that DHS officials provided to  us, including documents from an unclassified and a classified literature  review that DHS had commissioned.
Finally, in stating its nonconcurrence with the recommendation to limit  future funding in support of its behavior detection activities, DHS stated  that TSA’s overall security program is composed of interrelated parts, and  to disrupt one piece of the multilayered approach may have an adverse  impact on other pieces. As we reported in November 2013, TSA has not  developed the performance measures that would allow it to assess the  effectiveness of its behavior detection activities compared with other  screening methods, such as physical screening. As a result, the impact of  behavior detection activities on TSA’s overall security program is  unknown. Further, not all screening methods are present at every airport,  and TSA has modified the screening procedures and equipment used at  airports over time. These modifications have included the discontinuance  of screening equipment that was determined to be unneeded or  ineffective. Therefore, we concluded that providing scientifically validated  evidence that demonstrates that behavioral indicators can be used to  identify passengers who may pose a threat to aviation security is critical  to the implementation of TSA’s behavior detection activities.  Consequently, we added a matter for congressional consideration to the  November 2013 report. Specifically, we suggested that Congress  consider the findings in the report regarding the absence of scientifically  validated evidence for using behavioral indicators to identify aviation  security threats when assessing the potential benefits of behavior  detection activities relative to their cost when making future funding  decisions related to aviation security. Such action should help ensure that  security-related funding is directed to programs that have demonstrated  their effectiveness.
Chairman Hudson, Ranking Member Richmond, and members of the  subcommittee, this concludes my prepared testimony. I look forward to  answering any questions that you may have.

	GAO Contact and Staff Acknowledgments

For questions about this statement, please contact Steve Lord at   (202) 512-4379 or lords@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this statement. Individuals making key contributions to this statement  include David Bruno (Assistant Director), Nancy Kawahara, Elizabeth  Kowalewski, Susanna Kuebler, Grant M. Mallie, Amanda K. Miller, Linda  S. Miller, and Douglas M. Sloane.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.

Summary:
Why GAO Did This Study

This testimony discusses GAO's November 2013 report assessing the Department of Homeland Security (DHS) Transportation Security Administration's (TSA) behavior detection activities, specifically the Screening of Passengers by Observation Technique (SPOT) program. The recent events at Los Angeles International Airport provide an unfortunate reminder of TSA's continued importance in providing security for the traveling public. TSA's behavior detection activities, in particular the SPOT program, are intended to identify high-risk passengers based on behavioral indicators that indicate mal-intent. In October 2003, TSA began testing the SPOT program, and by fiscal year 2012, about 3,000 behavior detection officers (BDO) had been deployed to 176 of the more than 450 TSA-regulated airports in the United States. TSA has expended a total of approximately $900 million on the program since it was fully deployed in 2007. This testimony highlights the key findings of GAO's November 8, 2013, report on TSA's behavior detection activities. Specifically, like the report, this statement will address (1) the extent to which available evidence supports the use of behavioral indicators to identify aviation security threats, and (2) whether TSA has data necessary to assess the effectiveness of the SPOT program in identifying threats to aviation security.

What GAO Found

In November 2013, GAO reported that (1) peer-reviewed, published research we reviewed did not support whether nonverbal behavioral indicators can be used to reliably identify deception, (2) methodological issues limited the usefulness of DHS's April 2011 SPOT validation study, and (3) variation in referral rates raised questions about the use of indicators. GAO reported that its review of meta-analyses (studies that analyze other studies and synthesize their findings) that included findings from over 400 studies related to detecting deception conducted over the past 60 years, other academic and government studies, and interviews with experts in the field, called into question the use of behavior observation techniques, that is, human observation unaided by technology, as a means for reliably detecting deception. The meta-analyses GAO reviewed collectively found that the ability of human observers to accurately identify deceptive behavior based on behavioral cues or indicators is the same as or slightly better than chance (54 percent). GAO also reported on other studies that do not support the use of behavioral indicators to identify mal-intent or threats to aviation.
GAO found that DHS's April 2011 validation study does not demonstrate effectiveness of the SPOT behavioral indicators because of methodological weaknesses. The validation study found, among other things, that some SPOT indicators were predictive of outcomes that represent high-risk passengers, and that SPOT procedures, which rely on the SPOT behavioral indicators, were more effective than a random selection protocol implemented by BDOs in identifying outcomes that represent high-risk passengers. While the April 2011 SPOT validation study is a useful initial step and, in part, addressed issues raised in GAO's May 2010 report, methodological weaknesses limit its usefulness. Specifically, as GAO reported in November 2013, these weaknesses include, among other things, the use of potentially unreliable data and issues related to one of the study's outcome measures.