Aviation Security: TSA Should Limit Future Funding for Behavior Detection Activities
Released: 2013-11-14T00:00:00

Letter

Available Evidence Does Not Support Whether Behavioral Indicators Can Be Used to Identify Aviation Security Threats

In November 2013, we reported that (1) peer-reviewed, published research we reviewed did not support whether nonverbal behavioral indicators can be used to reliably identify deception, (2) methodological issues limited the usefulness of DHS’s April 2011 SPOT validation study, and (3) variation in referral rates raised questions about the use of indicators.

Published Research on Behavioral Indicators

In November 2013, we reported that our review of meta-analyses (studies that analyze other studies and synthesize their findings) that included findings from over 400 studies related to detecting deception conducted over the past 60 years, other academic and government studies, and interviews with experts in the field, called into question the use of behavior observation techniques, that is, human observation unaided by technology, as a means for reliably detecting deception. The meta- analyses we reviewed collectively found that the ability of human observers to accurately identify deceptive behavior based on behavioral cues or indicators is the same as or slightly better than chance (54 percent). We also reported on other studies that do not support the use of behavioral indicators to identify mal-intent or threats to aviation. In commenting on a draft of our November 2013 report, DHS stated that one of these studies, a 2013 RAND report, provides evidence that supports the SPOT program. However, the RAND report, which concludes that there is current value and unrealized potential for using behavioral indicators as part of a system to detect attacks, refers to behavioral indicators that are defined and used significantly more broadly than those in the SPOT program. The indicators reviewed in the RAND report are not used in the SPOT program, and, according to the RAND report’s findings, could not be used in real time in an airport environment.

DHS’s Validation Study

Further, in November 2013, we found that DHS’s April 2011 validation study does not demonstrate effectiveness of the SPOT behavioral indicators because of methodological weaknesses. The validation study found, among other things, that some SPOT indicators were predictive of outcomes that represent high-risk passengers, and that SPOT procedures, which rely on the SPOT behavioral indicators, were more effective than a random selection protocol implemented by BDOs in identifying outcomes that represent high-risk passengers. While the April 2011 SPOT validation study is a useful initial step and, in part, addressed issues raised in our May 2010 report, methodological weaknesses limit its usefulness. Specifically, as we reported in November 2013, these weaknesses include, among other things, the use of potentially unreliable data and issues related to one of the study’s outcome measures.

First, the data the study used to determine the extent to which the SPOT behavioral indicators led to correct screening decisions at checkpoints were from the SPOT database that we had previously found in May 2010 to be potentially unreliable. In 2010, we found, among other things, that BDOs could not record all behaviors observed in the SPOT database because the database limited entry to eight behaviors, six signs of deception, and four types of serious prohibited items per passenger referred for additional screening, though BDOs are trained to identify 94 total indicators.subsequent to our May 2010 report, the validation study used data that were collected from 2006 through 2010, prior to TSA’s improvements to the SPOT database. Consequently, the data were not sufficiently reliable for use in conducting a statistical analysis of the association between the indicators and high-risk passenger outcomes.

Although TSA made changes to the database Second, our analysis of the validation study data regarding one of the primary high-risk outcome measures—LEO arrests—suggests that the screening process was different for passengers depending on whether they were selected using SPOT procedures or the random selection protocol. Specifically, different levels of criteria were used to determine whether passengers in each group were referred to a LEO, which is a necessary precondition for an arrest. Because of this discrepancy between the study groups, the results related to the LEO arrest metric are questionable and cannot be relied upon to demonstrate the effectiveness of the SPOT program’s behavioral indicators. In November 2013, we also reported on other methodological weaknesses, including design limitations and monitoring weaknesses, that could have affected the usefulness of the validation study’s results in determining the effectiveness of the SPOT program’s behavioral indicators.

Variation in Referral Rates and TSA Efforts to Study Indicators

In November 2013, we reported that variation in referral rates and subjective interpretation of the behavioral indicators raise questions about the use of indicators, but TSA has efforts under way to study the indicators. Specifically, we found that SPOT referral data from fiscal years 2011 and 2012 indicate that SPOT and LEO referral rates vary significantly across BDOs at some airports, which raises questions about the use of SPOT behavioral indicators by BDOs. The rate at which BDOs referred passengers for SPOT referral screening ranged from 0 to 26 referrals per 160 hours worked during the 2-year period we reviewed. Similarly, the rate at which BDOs referred passengers to In November 2013, we LEOs ranged from 0 to 8 per 160 hours worked.also reported that BDOs and TSA officials we interviewed said that some of the behavioral indicators are subjective and TSA has not demonstrated that BDOs can consistently interpret the behavioral indicators. We found that there is a statistically significant relationship between the length of time an individual has been a BDO and the number of SPOT referrals the individual makes. This suggests that different levels of experience may be one reason why BDOs apply the behavioral indicators differently. TSA has efforts underway to better define the behavioral indicators currently used by BDOs, and to complete an inter-rater reliability study. The inter- rater reliability study could help TSA determine whether BDOs can consistently and reliably interpret the behavioral indicators, which is a critical component of validating the SPOT program’s results and ensuring that the program is implemented consistently. According to TSA, the current contract to study the indicators and the inter-rater reliability study will be completed in 2014.

TSA Has Limited Information to Evaluate SPOT Program Effectiveness, but Plans to Collect Additional Performance Data

In November 2013, we reported that TSA plans to collect and analyze additional performance data needed to assess the effectiveness of its behavior detection activities. In response to a recommendation in our May 2010 report to develop a plan for outcome-based performance measures, TSA completed a performance metrics plan in November 2012. The plan defined an ideal set of 40 metrics within three major categories that TSA needs to collect to measure the performance of its behavior detection activities. As of June 2013, TSA had collected some information for 18 of 40 metrics the plan identified, but the agency was collecting little to none of the data required to assess the performance and security effectiveness of its behavior detection activities or the SPOT program specifically. For example, TSA did not and does not currently collect the data required to determine the number of passengers meaningfully assessed by BDOs, BDOs’ level of fatigue, or the impact that fatigue has on their performance.

To address these and other deficiencies, the performance metrics plan identifies 22 initiatives that are under way or planned as of November 2012. For example, in May 2013, TSA began to implement a new data collection system, BDO Efficiency and Accountability Metrics, designed to track and analyze BDO daily operational data, including BDO locations and time spent performing different activities. According to TSA officials, these data will allow the agency to gain insight on how BDOs are utilized, and improve analysis of the SPOT program. However, according to the performance metrics plan, TSA will require at least an additional 3 years and additional resources before it can begin to report on the performance and security effectiveness of its behavior detection activities or the SPOT program.

Without the data needed to assess the effectiveness of behavior detection activities or the SPOT program, we reported in November 2013 that TSA uses SPOT referral, LEO referral, and arrest statistics to help track the program’s activities. As shown in figure 1, of the approximately 61,000 SPOT referrals made during fiscal years 2011 and 2012 at the 49 airports we analyzed, approximately 8,700 (13.6 percent) resulted in a referral to a LEO. Of the SPOT referrals that resulted in a LEO referral, 365 (4 percent) resulted in an arrest.

TSA has taken a positive step toward determining the effectiveness of its behavior detection activities by developing the performance metrics plan, as we recommended in May 2010. However, as we reported in November 2013, TSA cannot demonstrate the effectiveness of its behavior detection activities, and available evidence does not support whether behavioral indicators can be used to identify threats to aviation security. According to Office of Management and Budget (OMB) guidance accompanying the fiscal year 2014 budget, it is incumbent upon agencies to use resources on programs that have been rigorously evaluated and determined to be effective, and to fix or eliminate those programs that have not demonstrated results. As we concluded in our November 2013 report, until TSA can provide scientifically validated evidence demonstrating that behavioral indicators can be used to identify passengers who may pose a threat to aviation security, the agency risks funding activities that have not been determined to be effective. Therefore, in our November 2013 report, we recommended that TSA limit future funding for its behavior detection activities. DHS did not concur with our recommendation.

The negatively and significantly related indicators were more commonly associated with passengers who were not identified as high-risk, than with passengers who were identified as high-risk. available. However, as described in the report, in addition to the meta- analyses of over 400 studies related to detecting deception conducted over the past 60 years that we reviewed, we also reviewed several documents on behavior detection research that DHS officials provided to us, including documents from an unclassified and a classified literature review that DHS had commissioned.

Finally, in stating its nonconcurrence with the recommendation to limit future funding in support of its behavior detection activities, DHS stated that TSA’s overall security program is composed of interrelated parts, and to disrupt one piece of the multilayered approach may have an adverse impact on other pieces. As we reported in November 2013, TSA has not developed the performance measures that would allow it to assess the effectiveness of its behavior detection activities compared with other screening methods, such as physical screening. As a result, the impact of behavior detection activities on TSA’s overall security program is unknown. Further, not all screening methods are present at every airport, and TSA has modified the screening procedures and equipment used at airports over time. These modifications have included the discontinuance of screening equipment that was determined to be unneeded or ineffective. Therefore, we concluded that providing scientifically validated evidence that demonstrates that behavioral indicators can be used to identify passengers who may pose a threat to aviation security is critical to the implementation of TSA’s behavior detection activities. Consequently, we added a matter for congressional consideration to the November 2013 report. Specifically, we suggested that Congress consider the findings in the report regarding the absence of scientifically validated evidence for using behavioral indicators to identify aviation security threats when assessing the potential benefits of behavior detection activities relative to their cost when making future funding decisions related to aviation security. Such action should help ensure that security-related funding is directed to programs that have demonstrated their effectiveness.

Chairman Hudson, Ranking Member Richmond, and members of the subcommittee, this concludes my prepared testimony. I look forward to answering any questions that you may have.

GAO Contact and Staff Acknowledgments

For questions about this statement, please contact Steve Lord at (202) 512-4379 or lords@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this statement include David Bruno (Assistant Director), Nancy Kawahara, Elizabeth Kowalewski, Susanna Kuebler, Grant M. Mallie, Amanda K. Miller, Linda S. Miller, and Douglas M. Sloane.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.