Aviation Security: TSA Should Limit Future Funding for Behavior Detection Activities
Published: 2013-11-08T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: TSA began deploying the SPOT program in fiscal year 2007--and has since spent about $900 million--to identify persons who may pose a risk to aviation security through the observation of behavioral indicators. In May 2010, GAO concluded, among other things, that TSA deployed SPOT without validating its scientific basis and SPOT lacked performance measures. GAO was asked to update its assessment. This report addresses the extent to which (1) available evidence supports the use of behavioral indicators to identify aviation security threats and (2) TSA has the data necessary to assess the SPOT program's effectiveness. GAO analyzed fiscal year 2011 and 2012 SPOT program data. GAO visited four SPOT airports, chosen on the basis of size, among other things, and interviewed TSA officials and a nonprobability sample of 25 randomly selected BDOs. These results are not generalizable, but provided insights.
section_title: What GAO Found
paragraphs: Available evidence does not support whether behavioral indicators, which are used in the Transportation Security Administration's (TSA) Screening of Passengers by Observation Techniques (SPOT) program, can be used to identify persons who may pose a risk to aviation security. GAO reviewed four meta-analyses (reviews that analyze other studies and synthesize their findings) that included over 400 studies from the past 60 years and found that the human ability to accurately identify deceptive behavior based on behavioral indicators is the same as or slightly better than chance. Further, the Department of Homeland Security's (DHS) April 2011 study conducted to validate SPOT's behavioral indicators did not demonstrate their effectiveness because of study limitations, including the use of unreliable data. Twenty-one of the 25 behavior detection officers (BDO) GAO interviewed at four airports said that some behavioral indicators are subjective. TSA officials agree, and said they are working to better define them. GAO analyzed data from fiscal years 2011 and 2012 on the rates at which BDOs referred passengers for additional screening based on behavioral indicators and found that BDOs' referral rates varied significantly across airports, raising questions about the use of behavioral indicators by BDOs. To help ensure consistency, TSA officials said they deployed teams nationally to verify compliance with SPOT procedures in August 2013. However, these teams are not designed to help ensure BDOs consistently interpret SPOT indicators.
TSA has limited information to evaluate SPOT's effectiveness, but plans to collect additional performance data. The April 2011 study found that SPOT was more likely to correctly identify outcomes representing a high-risk passenger--such as possession of a fraudulent document--than through a random selection process. However, the study results are inconclusive because of limitations in the design and data collection and cannot be used to demonstrate the effectiveness of SPOT. For example, TSA collected the study data unevenly. In December 2009, TSA began collecting data from 24 airports, added 1 airport after 3 months, and an additional 18 airports more than 7 months later when it determined that the airports were not collecting enough data to reach the study's required sample size. Since aviation activity and passenger demographics are not constant throughout the year, this uneven data collection may have conflated the effect of random versus SPOT selection methods. Further, BDOs knew if passengers they screened were selected using the random selection protocol or SPOT procedures, a fact that may have introduced bias into the study. TSA completed a performance metrics plan in November 2012 that details the performance measures required for TSA to determine whether its behavior detection activities are effective, as GAO recommended in May 2010. However, the plan notes that it will be 3 years before TSA can begin to report on the effectiveness of its behavior detection activities. Until TSA can provide scientifically validated evidence demonstrating that behavioral indicators can be used to identify passengers who may pose a threat to aviation security, the agency risks funding activities that have not been determined to be effective. This is a public version of a sensitive report that GAO issued in November 2013. Information that TSA deemed sensitive has been redacted.
section_title: What GAO Recommends
paragraphs: Congress should consider the absence of scientifically validated evidence for using behavioral indicators to identify threats to aviation security when assessing the potential benefits and cost in making future funding decisions for aviation security. GAO included this matter because DHS did not concur with GAOs recommendation that TSA limit future funding for these activities until it can provide such evidence, in part because DHS disagreed with GAOs analysis of indicators. GAO continues to believe the report findings and recommendation are valid.

section_title: Letter
paragraphs: The Department of Homeland Security’s (DHS) Transportation Security  Administration (TSA) fiscal year 2014 budget request amounts to  approximately $7.4 billion for programs and activities to secure the  nation’s transportation systems. This amount includes nearly $5 billion for  TSA’s Aviation Security account, a portion of which is requested to  support Screening of Passengers by Observation Techniques (SPOT)  within the Behavior Detection and Analysis (BDA) program, which seeks  to identify persons who may pose a risk to aviation security. Through the  SPOT program, TSA’s behavior detection officers (BDO) are to identify  passenger behaviors indicative of stress, fear, or deception and refer  passengers meeting certain criteria for additional screening of their  persons and carry-on baggage. During this SPOT referral screening, if  passengers exhibit additional behaviors, or if other events occur, such as  the discovery of a suspected fraudulent document, BDOs are to refer  these passengers to a law enforcement officer (LEO) for further  investigation, which could result in an arrest, among other outcomes.
In October 2003, TSA began testing its primary behavior detection  activity, the SPOT program, and during fiscal year 2007, TSA deployed  the program to 42 TSA-regulated airports. By fiscal year 2012, about  3,000 BDOs were deployed to 176 of the more than 450 TSA-regulated  airports in the United States. From fiscal years 2011 through 2012, an  estimated 1.3 billion people passed through checkpoints at the 176 SPOT  airports. TSA has expended approximately $200 million annually for the  SPOT program since fiscal year 2010, and a total of approximately $900  million since 2007. BDOs represent one of TSA’s layers of security. In  addition to BDOs, other layers of security include travel document  checkers, who examine tickets, passports, and other forms of  identification; transportation security officers (TSO), who are responsible  for screening passengers and their carry-on baggage at passenger  checkpoints using X-ray equipment, magnetometers, advanced imaging  technology, and other devices; as well as for screening checked  baggage; and random employee screening, among others.
In May 2010, we concluded on the basis of our work, among other things,  that TSA deployed SPOT nationwide without first validating the scientific  basis for identifying passengers who may pose a threat in an airport  environment. TSA piloted the SPOT program in 2003 and 2004 at  several New England airports. However, the pilot was not designed to  determine the effectiveness of using behavior detection techniques to  enhance aviation security; rather, the pilot was focused on the operational  feasibility of implementing the SPOT program at airports. In recognition of  the need to conduct additional research, DHS’s Science and Technology  Directorate (S&T) hired a contractor in 2007 to design and execute a  validation study to determine whether the primary screening instrument  used in the program—the SPOT referral report and its associated  indicators based on behavior or appearance factors—could be used to  correctly identify high-risk passengers. The validation study, published in  April 2011, found that the SPOT program identified substantially more  “high-risk” passengers—defined by the study as those passengers who,  for example, possessed fraudulent documents—as compared with  passengers who had been selected by BDOs according to a random  selection protocol. However, the validation study cited certain  methodological limitations, such as the potential for selection bias as a  result of BDOs participating in the study not following the random  selection protocols, among others. S&T concluded that the limitations  were minimal and that the results were reasonable and reliable. In May  2010, we recommended that S&T convene an independent panel of  experts to comment on and evaluate the methodology of the ongoing  validation study. In response, S&T established a Technical Advisory  Committee (TAC) of 12 researchers and issued a separate report in June  2011 summarizing TAC members’ recommendations and opinions on the  study results. The results of the validation study and TAC’s comments  and concerns are discussed later in this report.
We also concluded in May 2010 that TSA was experiencing challenges in  implementing the SPOT program at airports, such as not systematically  collecting and analyzing potentially useful passenger information obtained  by BDOs, and that the program lacked outcome-based performance  measures useful for assessing the program’s effectiveness. As a result,  we recommended that TSA take several actions to help assess SPOT’s  contribution to improving aviation security. Overall, TSA has taken action  on all of the 11 recommendations we made, and, as of October 2013, has  implemented 10 of the recommendations. For example, among other  things, TSA revised SPOT standard operating procedures to more clearly  instruct BDOs and other TSA personnel regarding how and when to enter  SPOT referral data into the Transportation Information Sharing System  (TISS). This would help enable the referral data to be shared with  federal, state, or local law enforcement entities. Further, in November  2012, TSA issued a plan to develop outcome-based performance  measures, such as the ability of BDOs to consistently identify SPOT  behavioral indicators, within 3 years to assess the effectiveness of the  SPOT program. This plan is discussed in more detail later in this report.
You requested an updated assessment of the SPOT program’s  effectiveness. Specifically, this report addresses the following questions:  1.  To what extent does available evidence support the use of behavioral  indicators to identify aviation security threats?  2.  To what extent does TSA have data necessary to assess the  effectiveness of the SPOT program in identifying threats to aviation  security?
In addition, we also reviewed information related to recent allegations   of profiling in the SPOT program. This information can be found in   appendix I.
To address the first question, we reviewed academic and government  research on behavior-based deception detection, which we identified  through a structured literature search and recommendations from experts  in the field. We assessed the reliability of this research against  established practices for study design, and through interviews with nine  experts we selected based on their published peer-reviewed research in  this area. While the results of these interviews cannot be used to  generalize about all research on behavior detection, they represent a mix  of views and subject matter expertise. We determined that the research  was sufficiently reliable for describing the evidence that existed regarding  the use of behavioral indicators to identify security threats. We also  analyzed documentation related to the April 2011 SPOT validation study,  including study protocols and the final reports, and assessed the study  against established practices for evaluation design and generally  accepted statistical principles. We interviewed headquarters TSA and  S&T officials responsible for the validation study and contractor officials.  We obtained the data that were used by these officials to reach the  conclusions in the validation study. To assess the soundness of the  methodology and conclusions in the validation study, we replicated some  of the analyses that were conducted by the contractor, based on the  methodology described in the final report. Generally, we replicated the  study’s results, and as an extra step, we extended the analyses using the  full sample of SPOT referrals to increase the power to detect significant  associations, as described in appendix II. We also analyzed data on  BDOs’ SPOT referrals, hours worked, and characteristics, such as race  and gender, from the SPOT program database, TISS, TSA’s Office of  Human Capital, and the National Finance Center for fiscal years 2011  and 2012 to determine the extent to which SPOT referrals varied across  airports and across BDOs with different characteristics. To assess the  reliability of these data, we reviewed relevant documentation, including  DHS privacy impact assessments and a 2012 data audit of the SPOT  database, and interviewed TSA officials about the controls in place to  maintain the integrity of the data. We determined that the data were  sufficiently reliable for us to use to standardize the referral data across  airports based on the number of hours each BDO spent performing  operational SPOT activities. In addition, we interviewed BDA program  managers at headquarters, and visited four airports where the SPOT  program was implemented in fiscal years 2011 and 2012, and where the  validation study was carried out. We selected the airports based on their  size, risk ranking, and participation in behavior detection programs. As  part of our visits, we interviewed 25 randomly selected BDOs, as well as  BDO managers and officials from the responsible local law enforcement  agency for each airport. While the results of these visits and interviews  are not generalizable to all SPOT airports or BDOs, they provided  additional BDO perspectives and helped corroborate the research and  statistical information we gathered through other means.
To address the second question, we analyzed documentation related to  the April 2011 validation study, including study protocols and the final  reports, and evaluated these efforts against established practices for  designing evaluations and generally accepted statistical principles. We  also reviewed financial data from fiscal years 2007 through 2012 to  determine the expenditures associated with the SPOT program, and  interviewed officials in DHS’s Office of the Inspector General (OIG) who  were working on a related audit of the SPOT program. We also  reviewed documentation associated with program oversight, including a  November 2012 performance metrics plan and evaluated TSA’s efforts to  collect and analyze data to provide oversight of BDA activities against  criteria outlined in Office of Management and Budget guidance, federal  government efficiency initiatives, and Standards for Internal Control in the  Federal Government. Finally, to demonstrate effectiveness of the BDA  program, including SPOT, we analyzed documentation such as a return- on-investment analysis and a risk-based allocation analysis, both from  December 2012. We interviewed headquarters TSA and S&T officials  responsible for the validation study and TSA field officials responsible for  collecting study data at the four airports we visited, as well as contractor  officials, and 8 of the 12 TAC members. We interviewed BDA officials in  the Offices of Security Capabilities and Security Operations, and TSA  officials in the Office of Human Capital on the extent to which they collect  and analyze data. In addition, to identify additional information about  recent allegations of passenger profiling in the SPOT program, we  reviewed documentation and data, and interviewed a nongeneralizable  sample of 25 randomly selected BDOs and an additional 7 BDOs who  contacted us directly. We also interviewed TSA headquarters and field  officials, such as federal security directors and BDO managers. Appendix  III provides additional details on our objectives, scope, and methodology.
This report is a public version of the prior sensitive report that we  provided to you. DHS and TSA deemed some of the information in the  report as sensitive security information, which must be protected from  public disclosure. Therefore, this report omits sensitive information about  specific SPOT behavioral indicators, the validation study findings, and the  results of our analysis on the extent to which SPOT referrals varied  across airports and across BDOs with different characteristics. Although  the information provided in this report is more limited in scope, it  addresses the same questions as the sensitive report. Also, the overall  methodology used for both reports is the same.
We conducted this performance audit from April 2012 to November 2013  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe the  evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: BDA and the SPOT Program
paragraphs: The Aviation and Transportation Security Act established TSA as the  federal agency with primary responsibility for securing the nation’s civil  aviation system, which includes the screening of all passengers and  property transported by commercial passenger aircraft. At the more than  450 TSA-regulated airports in the United States, all passengers, their  accessible property, and their checked baggage are screened prior to  boarding an aircraft or entering the sterile area of an airport pursuant to  statutory and regulatory requirements and TSA-established standard  operating procedures. BDA, and more specifically, the SPOT program,  constitutes one of multiple layers of security implemented within TSA- regulated airports. According to TSA’s strategic plan and other program  guidance for the BDA program released in December 2012, the goal of  the agency’s behavior detection activities, including the SPOT program, is  to identify high-risk passengers based on behavioral indicators that  indicate “mal-intent.” For example, the strategic plan notes that in concert  with other security measures, behavior detection activities “must be  dedicated to finding individuals with the intent to do harm, as well as  individuals with connections to terrorist networks that may be involved in  criminal activity supporting terrorism.”
TSA developed its primary behavior detection activity, the SPOT  program, in 2003 as an added layer of security to identify potentially high- risk passengers through behavior observation and analysis techniques.  The SPOT program’s standard operating procedures state that BDOs are  to observe and visually assess passengers, primarily at passenger  screening checkpoints, and identify those who display clusters of  behaviors indicative of stress, fear, or deception. The SPOT procedures  list a point system BDOs are to use to identify potentially high-risk  passengers on the basis of behavioral and appearance indicators, as  compared with baseline conditions where SPOT is being conducted. A  team of two BDOs is to observe passengers as they proceed through the  screening process. This process is depicted in figure 1.
According to TSA, it takes a BDO less than 30 seconds to meaningfully  observe an average passenger. If one or both BDOs observe that a  passenger reaches a predetermined point threshold, the BDOs are to  direct the passenger and any traveling companions to the second step of  the SPOT process—SPOT referral screening. During SPOT referral  screening, BDOs are to engage the passenger in casual conversation—a  voluntary informal interview—in the checkpoint area or a predetermined  operational area in an attempt to determine the reason for the  passenger’s behaviors and either confirm or dispel the observed  behaviors. SPOT referral screening also involves a physical search of  the passenger and his or her belongings. According to TSA, an average  SPOT referral takes 13 minutes to complete. If the BDOs concur that a  passenger’s behavior escalates further during the referral screening or if  other events occur, such as the discovery of fraudulent identification  documents or suspected serious prohibited or illegal items, they are to  call a LEO to conduct additional screening—known as a LEO referral— who then may allow the passenger to proceed on the flight, or may  question, detain, or arrest the passenger. The federal security director  or designee, regardless of whether a LEO responds, is responsible for  reviewing the circumstances surrounding a LEO referral and making the  determination about whether the passenger can proceed into the sterile  area of the airport.
section_title: Overview of SPOT Program Funding
paragraphs: The costs of the SPOT program are not broken out as a single line item in  the budget. Rather, SPOT program costs are funded through three  separate program, project, activity (PPA)-level accounts: (1) BDO payroll  costs are funded through the Screener Personnel Compensation and  Benefits (PC&B) PPA, (2) the operating expenses of the BDOs and the  program are funded through the Screener Training and Other PPA, and  (3) the program management payroll costs are funded through the Airport  Management and Support PPA. From fiscal year 2007—when the SPOT  program began deployment nationwide—through fiscal year 2012, about  $900 million has been expended on the program, as shown in figure 2.
The majority of the funding (approximately 79 percent) for the SPOT  program covers workforce costs and is provided under the Screener  Personnel Compensation and Benefits PPA. This PPA—for which TSA  requested about $3 billion for fiscal year 2014—funds, among other TSA  screening activities, BDOs and TSO screening of passengers and their  property. The workforce of about 3,000 BDOs is broken into four separate  pay bands. The F Band, or Master BDO, and the G Band, or Expert BDO,  constitute the primary BDO workforce that screens passengers using  behavior detection. The H and I bands are supervisory-level BDOs,  responsible for overseeing SPOT operations at the airport level.  According to TSA figures, in fiscal year 2012, the average salaries and  benefits of an F Band BDO full-time equivalent (FTE) was $66,310; a G  Band BDO was $78,162, and the average FTE cost of H and I Band BDO  supervisors was $97,392.
section_title: Overview of the Validation Study
paragraphs: In 2007, S&T began research to assess the validity of the SPOT program.  The contracted study, issued in April 2011, was to examine the extent to  which using the SPOT referral report and its indicators, as established in  SPOT procedures, led to correct screening decisions at security  checkpoints. Two primary studies were designed within the broader  validation study:  1.  an indicator study: an analysis of the behavioral and appearance  indicators recorded in SPOT referral reports over an approximate 5- year period and their relationships to outcomes indicating a possible  threat or high-risk passenger, and  2.  a comparison study: an analysis over an 11-month period at 43  airports that compared arrests and other outcomes for passengers  selected using the SPOT referral report with passengers selected and  screened at random, as shown in table 1.
The validation study found, among other things, that some SPOT  indicators appeared to be predictors of outcomes indicating a possible  threat or high-risk passenger, and that SPOT procedures were more  effective than a selection of passengers through a random protocol in  identifying outcomes that represent high-risk passengers.
While the validation study was being finalized, DHS convened a TAC  composed of 12 researchers and law enforcement professionals who met  for 1 day in February 2011 to evaluate the methodology of the SPOT  validation study. According to the TAC report, TAC members received  briefings from the contractor that described the study plans and results,  but because of TSA’s security concerns, TAC members did not receive  detailed information about the contents of the SPOT referral report, the  individual indicators used in the SPOT program, the validation study data,  or the final report containing complete details of the SPOT validation  study results. The TAC report noted that several TAC members felt that  these restrictions hampered their ability to perform their assigned tasks.  According to TSA, TAC members were charged with evaluating the  methodology of the study, not the contents of the SPOT referral report.  Consequently, TSA officials determined that access to this information  was not necessary for the TAC to fulfill its responsibilities. S&T also  contracted with another contractor, a human resources research  organization, to both participate as TAC members and write a report  summarizing the TAC meeting and subsequent discussions among the  TAC members. In June 2011, S&T issued the TAC report, which  contained TAC recommendations on future work as well as an appendix  on TAC dissenting opinions. The findings of the TAC report are discussed  later in this report.
section_title: Available Evidence Does Not Support Whether Behavioral Indicators Can Be Used to Identify Aviation Security Threats
paragraphs: Meta-analyses and other published research studies we reviewed do not  support whether nonverbal behavioral indicators can be used to reliably  identify deception. While the April 2011 SPOT validation study was a  useful initial step and, in part, addressed issues raised in our May 2010  report, it does not demonstrate the effectiveness of the SPOT indicators  because of methodological weaknesses in the study. Further, TSA  program officials and BDOs we interviewed agree that some of the  behavioral indicators used to identify passengers for additional screening  are subjective. TSA has plans to study whether behavioral indicators can  be reliably interpreted, and variation in referral rates raises questions  about the use of the indicators by BDOs.
subsections: section_title: Published Research Does Not Support Whether the Use of Behavioral Indicators by Human Observers Can Identify Deception
subsections: section_title: Studies of Nonverbal Indicators to Identify Deception
paragraphs: Peer-reviewed, published research does not support whether the use of  nonverbal behavioral indicators by human observers can accurately  identify deception. Our review of meta-analyses and other studies  related to detecting deception conducted over the past 60 years, and  interviews with experts in the field, question the use of behavior  observation techniques, that is, human observation unaided by  technology, as a means for reliably detecting deception. The meta- analyses, or reviews that synthesize the findings of other studies, we  reviewed collectively included research from more than 400 separate  studies on detecting deception, and found that the ability of human  observers to accurately identify deceptive behavior based on behavioral  cues or indicators is the same as or slightly better than chance (54  percent). A 2011 meta-analysis showed weak correlations between  most behavioral cues studied and deception. For example, the meta- analysis showed weak correlations for behavioral cues that have been  studied the most, such as fidgeting, postural shifts, and lack of eye  contact. A 2006 meta-analysis reviewed, in part, the ability of both  individuals trained in fields such as law enforcement, as well as those  untrained, and found no difference in their ability to detect deception.  Additionally, a 2007 meta-analysis on nonverbal indicators of deception  states that while there is a general belief that certain nonverbal behaviors  are strongly associated with deception—such as an increase in hand,  foot, and leg movements—these behaviors are diametrically opposed to  observed indicators of deception in experimental studies, which indicate  that movements actually decrease when people are lying.
As part of our analysis, we also reviewed scientific research focused on  detecting passenger deception in an airport environment. We identified a  2010 study–based on a small sample size of passengers–that reviewed a  similar behavior observation program in another country. The first phase  of the study found that passengers who were selected based on  behaviors were more likely to be referred to airport security officials for  further questioning as compared to passengers who had been selected  according to a random selection protocol. However, because the physical  attributes of the passengers were found to be significantly different  between those passengers selected based on behaviors versus those  randomly selected, the researchers undertook a second phase of the  study to control for those differences. The second phase revealed no  differences in initial follow up rate between passengers selected based on  behaviors and those matched for physical attributes. That is, when the  control group was matched by physical attribute to passengers selected  on the basis of behaviors, the follow up rate was the same. The  researchers concluded that the higher number of passengers selected  based on behaviors and referred for further questioning during the first  phase of the study “was more the result of profiling” than the use of  behavior observation techniques.
As mentioned earlier in this report, the goal of the BDA program is to  identify high-risk passengers based on behavioral indicators that may  indicate mal-intent. However, other studies we reviewed found that there  is little available research regarding the use of behavioral indicators to  determine mal-intent, or deception related to an individual’s intentions.  For example, a 2013 RAND report noted that controversy exists regarding  the use of human observation techniques that use behavioral indicators to  identify individuals with intent to deceive security officials. In particular,  the study noted that while behavioral science has identified nonverbal  behaviors associated with emotional and psychological states, these  indicators are subject to certain factors, such as individual variability, that  limit their potential utility in detecting pre-incident indicators of attack.
The RAND report also found that the techniques for measuring the  potential of using behavioral indicators to detect attacks are poorly  developed and worthy of further study.
Moreover, a 2008 study performed for the Department of Defense by the  JASON Program Office reviewed behavior detection programs, including  the methods used by the SPOT program, and found that no compelling  evidence exists to support remote observation of physiological signals  that may indicate fear or nervousness in an operational scenario by  human observers, and no scientific evidence exists to support the use of  these signals in detecting or inferring future behavior or intent. In  particular, the report stated that success in identifying deception and  intent in other studies is post hoc and such studies incorrectly equate  success in identifying terrorists with the identification of drug smugglers,  warrant violators, or others. For example, when describing the  techniques used by BDOs in the SPOT program, the report concluded  that even if a correlation were found between abnormal behaviors and  guilt as a result of some transgression, there is no clear indication that the  guilt caused the abnormal behavior. The report also noted that the  determination that the abnormal behavior was caused by guilt was made  after the fact, rather than being based on established criteria beforehand.
Recent research on behavior detection has identified more promising  results when behavioral indicators are used in combination with certain  interview techniques and automated technologies, which are not used as  part of the SPOT program. For example, several studies we reviewed that  were published in 2012 and 2013 note that specific interviewing  techniques, such as asking unanticipated questions, may assist in  identifying deceptive individuals. Researchers began to develop  automated technologies to detect deception, in part, because humans are  limited in their ability to perceive, detect, and analyze all of the potentially  useful information about an individual, some of which otherwise would not  be noticed by the naked eye. For example, the 2013 RAND report noted  that the link between facial microexpressions—involuntary expressions of  emotion appearing for milliseconds despite best efforts to dampen or hide  them—and deception can be evidenced by coding emotional expressions  from a frame-by-frame analysis of video. However, the study concludes  that the technique is not suitable for use by humans in real time at  checkpoints or other screening areas because of the time lag and hours  of labor required for such analysis. Automated technologies are being  explored by federal agencies in conjunction with academic researchers to  overcome these limitations, as well as human fatigue factors and potential  bias in trying to detect deception. Although in the early stages of  development, the study stated that automated technologies might be  effective at fusing multiple indicators, such as body movement, vocal  stress, and facial microexpression analysis.
section_title: Methodological Issues Limit the Usefulness of DHS’s April 2011 Indicator Validation Study
paragraphs: The usefulness of DHS’s April 2011 validation study is limited, in part  because the data the study used to examine the extent to which the  SPOT behavioral indicators led to correct screening decisions at security  checkpoints were from the SPOT database that we had previously found  in May 2010 to have several weaknesses, and thus were potentially  unreliable. The SPOT indicator study analyzed data collected from 2006  to 2010 to determine the extent to which the indicators could identify high- risk passengers defined as passengers who (1) possessed fraudulent  documents, (2) possessed serious prohibited or illegal items, (3) were  arrested by a LEO, or (4) any combination of the first three measures.  The validation study reported that 14 of the 41 SPOT behavioral  indicators were positively and significantly related to one or more of the  study outcomes. However, in May 2010, we assessed the reliability of  the SPOT database against Standards for Internal Control in the Federal  Government and concluded that the SPOT database lacked controls to  help ensure the completeness and accuracy of the data, such as  computerized edit checks to review the format, existence, and  reasonableness of data. We found, among other things, that BDOs could  not record all behaviors observed in the SPOT database because the  database limited entry to eight behaviors, six signs of deception, and four  types of serious prohibited items per passenger referred for additional  screening. BDOs are trained to identify 94 signs of stress, fear, and  deception, or other related indicators. As a result, we determined that,  as of May 2010, the data were not reliable enough to conduct a statistical  analysis of the association between the indicators and high-risk  passenger outcomes. In May 2010, we recommended that TSA make  changes to ensure the quality of SPOT referral data, and TSA  subsequently made changes to the SPOT database. However, the  validation study used data that were collected from 2006 through 2010,  prior to TSA’s improvements to the SPOT database. Consequently, the  data were not sufficiently reliable for use in conducting a statistical  analysis of the association between the indicators and high-risk  passenger outcomes.
In their report that reviewed the validation study, TAC members  expressed some reservations about the methodology used in analyzing  the SPOT indicators and suggested that the contractor responsible for  completing the study consider not reporting on some of its results and  moving the results to an appendix, rather than including them as a  featured portion of the report. Further, the final validation study report  findings were mixed, that is, they both supported and questioned the use  of these indicators in the airport environment, and the report noted that  the study was an “initial step” toward validating the program. However,  because the study used unreliable data, its conclusions regarding the use  of the SPOT behavioral indicators for passenger screening are  questionable and do not support the conclusion that they can or cannot  be used to identify threats to aviation security. Other aspects of the  validation study are discussed later in this report.
section_title: Subjective Interpretation of Behavioral Indicators and Variation in Referral Rates Raise Questions about the Use of Indicators; TSA Plans to Study Indicators
subsections: section_title: BDO Interpretation of Some Behavioral Indicators Is Subjective; TSA Plans Study
paragraphs: BDA officials at headquarters and BDOs we interviewed in four airports  said that some of the behavioral indicators are subjective, and TSA has  not demonstrated that BDOs can consistently interpret behavioral  indicators, though the agency has efforts under way to reduce subjectivity  in the interpretation by BDOs. For example, BDA officials at headquarters  stated that the definition of some behaviors in SPOT standard operating  procedures is subjective. Further, 21 of 25 BDOs we interviewed said that  certain behaviors can be interpreted differently by different BDOs. SPOT  procedures state that the behaviors should deviate from the  environmental baseline. As a result, BDOs’ application of the definition of  the behavioral indicators may change over time, or in response to  external factors.
Four of the 25 BDOs we spoke with said that newer BDOs might be more  sensitive in applying the definition of certain behaviors. Our analysis of  TSA’s SPOT referral data, discussed further below, shows that there is a  statistically significant correlation between the length of time that an  individual has been a BDO, and the number of SPOT referrals the  individual makes per 160 hours worked, or about four 40-hour work  weeks. This suggests that different levels of experience may be one  reason why BDOs apply the behavioral indicators differently.
BDA officials agree that some of the SPOT indicators are subjective, and  the agency is working to better define the behavioral indicators currently  used by BDOs. In December 2012, TSA initiated a new contract to review  the indicators in an effort to reduce the number of behavioral and  appearance indicators used and to reduce subjectivity in the interpretation  by BDOs. In June 2013, the contractor produced a document that  summarizes information on the SPOT behavioral indicators from the  validation study analysis, such as how frequently the indicator was  observed, that it says will be used in the indicator review process.  According to TSA’s November 2012 performance metrics plan, in 2014,  the agency also intends to complete an inter-rater reliability study. This  study could help TSA determine whether BDOs can reliably interpret the  behavioral indicators, which is a critical component of validating the  SPOT program’s results and ensuring that the program is implemented  consistently.
Our analysis of SPOT referral data from fiscal years 2011 and 2012  indicates that SPOT and LEO referral rates vary significantly across  BDOs at some airports, which raises questions about the use of  behavioral indicators by BDOs. Specifically, we found that variation  exists in the SPOT referral rates among 2,199 nonmanager BDOs and  across the 49 airports in our review, after standardizing the referral data  to take account of the differences in the amount of time each BDO spent  observing passengers, as shown in figure 3.
The SPOT referral rates of BDOs ranged from 0 to 26 referrals per 160  hours worked during the 2-year period we reviewed. Similarly, LEO  referral rates of BDOs ranged from 0 to 8 per 160 hours worked.  Further, at least 153 of the 2,199 nonmanager BDOs were never  identified as the primary BDO responsible for a referral. Of these, at least  76 were not associated with a referral during the 2-year period we  reviewed.
To better understand the variation in referral rates, we analyzed whether  certain variables affected SPOT referral rates and LEO referral rates,  including the airport at which the referral occurred, and BDO  characteristics, such as their annual performance scores, years of  experience, as well as demographic information, including age and  gender. The variables we identified as having a statistically significant  relationship to the referral rates are shown in table 2.
We found that overall, the greatest amount of the variation in SPOT  referral rates by BDOs was explained by the airport in which the referral  occurred. That is, a BDO’s SPOT referral rate was associated with the  airport at which he or she was conducting SPOT activities. However,  separate analyses we conducted indicate that these differences across  airports were not fully accounted for by another variable that is directly  related to individual airports. That variable accounted for less than half of  the variation in SPOT referral rates accounted for by airports. Combined,  the remaining variables–including BDO performance score, age, years of  BDO experience, years of TSA experience, race, and educational level– accounted for little of the variation in SPOT referral rates. In commenting  on this issue, TSA officials noted that variation in referral rates across  airports could be the result of differences in passenger composition, the  airport’s market type, the responsiveness of LEOs to BDO referrals, and  the number and type of airlines at the airports, among other things.  However, because TSA could not provide additional supporting data on  these variables with comparable time frames, we were not able to include  these variables in our analysis. See appendix IV for a more detailed  discussion of the findings from our multivariate analysis of referral rates.
According to TSA, having clearly defined and consistently implemented  standard operating procedures for BDOs in the field at the 176 SPOT  airports is key to the success of the program. In May 2010, we found that  TSA established standardization teams designed to help ensure  consistent implementation of the SPOT standard operating procedures.  We followed up on TSA’s use of standardization teams and found that  from 2012 to 2013, TSA made standardization team visits to 9 airports. In  May 2012, officials changed their approach and data collection  requirements and changed the name of the teams to program compliance  assessment teams. From December 2012 through March 2013, TSA  conducted pilot site visits to 3 airports to test and refine new compliance  team protocols for data collection, which, among other things, involve  more quantitative analysis of BDO performance. The pilot process was  designed to help ensure that the program compliance assessment teams  conduct standardized, on-site evaluations of BDOs’ compliance with the  SPOT standard operating procedures in a way that is based on current  policy and procedures. As of June 2013, TSA had visited and collected  data at 6 additional airports and was refining data input and reporting  processes. According to BDA officials, TSA deployed the new compliance  teams nationally in August 2013 and anticipates visiting an additional 13  airports by the end of fiscal year 2013. However, the compliance teams  are not generally designed to help ensure BDOs’ ability to consistently  interpret the SPOT indicators, and the agency has not developed other  mechanisms to measure inter-rater reliability. TSA does not have  reasonable assurance that BDOs are reliably interpreting passengers’  behaviors within or among airports, in part because of the subjective  interpretation of some SPOT behavioral indicators by BDOs and the  limited scope of the compliance teams. This, coupled with the  inconsistency in referral rates across different airports, raises questions  about the use of behavioral indicators to identify potential threats to  aviation.
section_title: TSA Has Limited Information to Evaluate SPOT Program Effectiveness but Plans to Collect Additional Performance Data
paragraphs: TSA has limited information to evaluate SPOT program effectiveness  because the findings from the April 2011 validation comparison study are  inconclusive because of methodological weaknesses in the study’s  overall design and data collection. However, TSA plans to collect  additional performance data to help it evaluate the effectiveness of its  behavior detection activities.
subsections: section_title: Methodological Issues Affect the Results of DHS’s Study Comparing SPOT with Random Selection of Passengers
subsections: section_title: Design Limitations
paragraphs: DHS’s 2011 validation study compared the effectiveness of SPOT with a  random selection of passengers and found that SPOT was between 4  and 52 times more likely to correctly identify a high-risk passenger than  random selection, depending on which of the study’s outcome measures  was used to define persons knowingly and intentionally trying to defeat  the security process. However, BDOs used various methods to  randomly select passengers during data collection periods of differing  length at the study airports. Initially, the contractor proposed that TSA use  random selection methods at a sample of 143 SPOT airports, based on  factors such as the number of airport passengers. If properly  implemented, the proposed sample would have helped ensure that the  validation study findings could be generalized to all SPOT airports.  However, according to the study and interviews with the contractor, TSA  selected a nonprobability sample of 43 airports based on input from local  TSA airport officials who decided to participate in the study. TSA allowed  the managers of these airports to decide which checkpoints would use  random procedures and when they would do so during airport operating  hours. According to the validation study and a contractor official, the  airports included in the study were not randomly selected because of the  increased time and effort it would take to collect study data at the 143  airports proposed by the contractor. Therefore, the study’s results may  provide insights about the implementation of the SPOT program at the 43  airports where the study was carried out, but they are not generalizable to  all 176 SPOT airports.
Additionally, TSA collected the validation study data unevenly and  experienced challenges in collecting an adequate sample size for the  randomly selected passengers, facts that might have further affected the  representativeness of the findings. According to established evaluation  design practices, data collection should be sufficiently free of bias or other  significant errors that could lead to inaccurate conclusions. Specifically,  in December 2009, TSA initially began collecting data from 24 airports  whose participation in the study was determined by the local TSA  officials. More than 7 months later, TSA added another 18 airports to the  study when it determined that enough data were not being collected on  the randomly selected passengers at participating airports to reach the  study’s required sample size. The addition of the airports coincided with  a substantial increase in referrals for additional screening and an uneven  collection of data, as shown in figure 4.
As a result of this uneven data collection, study data on 61 percent of  randomly selected passengers were collected during the 3-month period  from July through September 2010. By comparison, 33 percent of the  data on passengers selected by the SPOT program were collected during  the same time. Because commercial aviation activity and the  demographics of the traveling public are not constant throughout the year,  this uneven data collection may have conflated the effect of random  versus SPOT selection methods with differences in the rates of high-risk  passengers when TSA used either method.
In addition, the April 2011 validation study noted that BDOs were aware  of whether the passengers they were screening were selected as a result  of the random selection protocol or SPOT procedures, which had the  potential to introduce bias in the assessment. According to established  practices for evaluation design, when feasible, many scientific studies use  “blind” designs, in which study participants do not know which procedures  are being evaluated. This helps avoid potential bias due to the tendency  of participants to behave or search for evidence in a manner that supports  the effects they expect each procedure to have. In contrast, in the SPOT  comparison study, BDOs knew whether each passenger they screened  was selected through SPOT or random methods. This may have biased  BDOs’ screening for high-risk passengers, because BDOs could have  expected randomly selected passengers to be lower risk and thus made  less effort to screen passengers. In interviews, the contractor and four of  the eight members of the TAC we interviewed agreed that this may be a  design weakness. One TAC member told us that the comparison study  would have been more robust if the passengers had been randomly  selected by people without any prior knowledge of SPOT indicators to  decrease the possibility of bias. To reduce the possibility of bias in the  study, another TAC member suggested that instead of using the same  BDOs to select and screen passengers, some BDOs could have been  responsible for selecting passengers and other BDOs for screening the  passengers, regardless of whether they were selected randomly or by  SPOT procedures. According to validation study training materials, BDOs  were used to select both groups of passengers in an effort to maintain  normal security coverage during the study. Another TAC member stated  that controls were needed to ensure that BDOs gave the same level of  scrutiny to randomly selected passengers as those referred because of  their behaviors. The contractor officials reported that they were aware of  the potential bias, and tried to mitigate its potential effects by training  BDOs who participated in the validation study to screen passengers  identically, regardless of how they were selected. However, the contractor  stated that they could not fully control these selections because BDOs  were expected to conduct their regular SPOT duties concurrently during  the study’s data collection on random passenger screening. The  validation study discussed several limitations that had the potential to  introduce bias, but concluded that they did not affect the results of the  study.
Our analysis of the validation study data regarding one of the primary  high-risk outcome measures—LEO arrests—suggests that the screening  process was different for passengers depending on whether they were  selected using SPOT procedures or the random selection protocol.  Therefore, the study’s finding that SPOT was much more likely to identify  high-risk passengers who were ultimately arrested by a LEO may be  considerably inflated. Specifically, a necessary condition influencing the  rate of the arrest outcome measure—exposure to a LEO through a LEO  referral—was not equal in the two groups. The difference between the  groups occurred because randomly selected passengers were likely to  begin the SPOT referral process with zero points or very few points,  whereas passengers selected on the basis of SPOT began the process at  the higher, established point threshold required for BDOs to make a  SPOT referral. However, because the point threshold for a LEO referral  was the same for both groups, the likelihood that passengers selected  using SPOT would escalate to the next point threshold, resulting in a LEO  referral and possible LEO arrest, was greater than for passengers  selected randomly. Our analysis showed that because of the discrepancy  in the points accrued prior to the start of the referral process, passengers  who were selected on the basis of SPOT behavioral indicators were more  likely to be referred to a LEO than randomly selected passengers. Our  analysis indicates that the validation study design could have been  improved by treating each group similarly, regardless of the passengers’  accumulated points. For example, as a possible approach, both groups  could have been referred to LEOs only in the cases where BDOs  discovered a serious prohibited or illegal item. Established study design  practices state that identifying key factors known to influence desired  evaluation outcomes will aid in forming treatment and comparison groups  that are as similar as possible, thus strengthening the analyses’  conclusions.
Additionally, once referred to a LEO, passengers selected at random  were arrested for different reasons than those selected on the basis of  SPOT indicators, which suggests that the two groups of passengers were  subjected to different types of screening. All randomly selected  passengers who were identified as high risk, referred to a LEO, and  ultimately arrested possessed fraudulent documents or serious prohibited  or illegal items. In contrast, most of the passengers arrested after having  been referred on the basis of SPOT behavior indicators were arrested for  reasons other than fraudulent documents or serious prohibited or illegal  items. These reasons for arrest included outstanding warrants by law  enforcement agencies, public intoxication, suspected illegal entry into the  United States, and disorderly conduct.
Such differences in the reasons for arrest suggest that referral screening  methods may have varied according to the method of selection for  screening, consistent with the concerns of the TAC members and the  contractor. Thus, because randomly selected passengers were assigned  points differently during screening and consequently referred to LEOs far  less than those referred by SPOT, and because being referred to a LEO  is a necessary condition for an arrest, the results related to the LEO  arrest metric are questionable and cannot be relied upon to demonstrate  SPOT program effectiveness.
To help ensure that all of the BDOs carried out the comparison study as  intended, protocols for randomly selecting passengers were established  that would help ensure that the methods would be the same across  airports. The contractor emphasized that deviating from the prescribed  protocol could increase the likelihood of introducing systematic  differences across airports in the methods of random screening, which  could bias the results. To ensure that airports and BDOs followed the  study protocols, the contractor conducted monitoring visits at 17 of the 43,  or 40 percent, of participating airports. The first monitoring visits occurred  6 months after data collection began, and 9 of the 17 airports were not  visited until the last 2 months of the study, as shown in figure 5.  Consequently, for 9 of these airports, the contractor could not have  addressed the deviations from the protocols that were identified during  the data-monitoring visits until the last weeks of data collection.
In the April 2011 report of all 17 monitoring visits that were conducted, the  most crucial issue the contractor identified was that BDOs deviated from  the random selection protocol in ways that did not meet the criteria for  systematic random selection. For example, the contractor found that  across airports, local TSA officials had independently decided to exclude  certain types of passengers from the study because the airport officials  felt it was unreasonable to subject these types of passengers to referral  screening. At 1 airport visited less than 4 weeks before data collection  ended, BDOs misunderstood the protocols and incorrectly excluded a  certain type of passenger. As a result, certain groups of potentially  lower-risk passengers were systematically excluded from the population  eligible for random selection. In addition, the contractor found that some  BDOs used their own methods to select passengers, rather than the  random selection protocol that was specified. The contractor reported that  if left uncorrected, this deviation from the protocols could increase the  likelihood of introducing systematic bias into the study. For example, at  one airport visited less than 6 weeks before data collection ended, BDOs  selected passengers by attempting to generate numbers they thought  were random by calling out numbers spontaneously, such as “seven,” and  using the numbers to select the seventh passenger, instead of following  the random selection protocol. At another airport visited less than 6  weeks before data collection ended, contrary to random selection  protocols, BDOs, rather than the data collection coordinator, selected  passengers to undergo referral screening. Although deviations from the  protocol may not have produced a biased sample, any deviation from the  selection protocol suggests that BDOs’ judgment may have affected the  random selection and screening processes in the comparison study.
In addition to the limitations cited above, the April 2011 validation study  noted other limitations such as the limited data useful for measuring high- risk passenger outcomes, the lack of information on the specific location  within the airport where each SPOT indicator was first observed, and  difficulties in differentiating whether passengers were referred because of  observed behaviors related to elevated indicators of stress, fear, and  deception, or for other reasons. The validation study concluded that  further research to fully validate and evaluate the SPOT program was  warranted. Similarly, the TAC report cited TAC members’ concerns that  the validation study results “could be easily misinterpreted given the  limited scope of the study and the caveats to the data,” and that the  “results should be presented as a first step in a broader evaluation  process.” Thus, limitations in the study’s design and in monitoring how it  was implemented at airports could have affected the accuracy of the  study’s conclusions, and limited their usefulness in determining the  effectiveness of the SPOT program. As a result, the incidence of high-risk  passengers in the normal passenger population remains unknown, and  the incidence of high-risk passengers identified by random selection  cannot be compared with the incidence of those identified using SPOT  methods.
section_title: TSA Plans to Collect and Analyze Needed Performance Data
paragraphs: TSA plans to collect and analyze additional performance data needed to  assess the effectiveness of its behavior detection activities. In response  to recommendations we made in May 2010 to conduct a cost-benefit  analysis and a risk assessment, TSA completed two analyses of the BDA  program in December 2012, but needs to complete additional analysis to  fully address our recommendations. Specifically, TSA completed a  return-on-investment analysis and a risk-based allocation analysis, both  of which were designed in part to inform the future direction of the  agency’s behavior detection activities, including the SPOT program. The  return-on-investment analysis assessed the additional value that BDOs  add to TSA’s checkpoint screening system, and concluded that BDOs  provide an integral value to the checkpoint screening process. However,  the report did not fully support its assumptions related to the threat  frequency or the direct and indirect consequence of a successful attack,  as is recommended by best practices. For example, TSA officials told us  that the threat and consequence assumptions in the analysis were  designed to be consistent with the 2013 Transportation Security System  Risk Assessment (TSSRA), but the analysis did not explain why a  catastrophic event was the only relevant threat scenario considered when  determining consequence. Additionally, the analysis relied on  assumptions regarding the effectiveness of BDOs and other  countermeasures that were based on questionable information. For  example, the analysis relied on results reported in the April 2011  validation study—which, as discussed earlier, had several methodological  limitations—as evidence of the effectiveness of BDOs. Further, a May  2013 DHS OIG report found that TSA could not accurately assess the  effectiveness or evaluate the progress of the SPOT program because it  had not developed a system of performance measures at the time of the  OIG review. In response, TSA provided the OIG with a draft version of  its performance metrics plan. This plan has since been finalized and is  discussed further below.
TSA’s risk-based allocation analysis found that an additional 584 BDO  FTEs should be allocated to smaller airports in an effort to cover existing  gaps in physical screening coverage and performance, an action that, if  implemented, would result in an annual budgetary increase of  approximately $42 million. One of the primary assumptions in the risk- based allocation analysis is related to the effectiveness of BDOs. For  example, this analysis suggests that BDOs may be effective in identifying  threats to aviation security where gaps exist in physical screening  coverage and performance, including the use of walk-through metal  detectors and advanced imaging technology machines. However, TSA  has not evaluated the effectiveness of BDOs in comparison with these  other screening methods.
In response to an additional recommendation in our May 2010 report to  develop a plan for outcome-based performance measures, TSA  completed a performance metrics plan in November 2012, which details  the performance measures required for TSA to determine whether the  agency’s behavior detection activities are effective, and identifies the  gaps that exist in its current data collection efforts. The plan defined an  ideal set of 40 metrics within three major categories that BDA needs to  collect to be able to understand and measure the performance of its  behavior detection activities. TSA then identified the gaps in its current  data collection efforts, such as, under the human factors subcategory,  data on BDO fatigue levels and what staffing changes would need to be  made to reduce the negative impact on BDO performance resulting from  fatigue, as shown in figure 6.
As of June 2013, TSA had collected some information for 18 of 40 metrics  the plan identified. Once collected, the data identified by the plan may  help support the completion of a more substantive return-on-investment  analysis and risk-based allocation analysis, but according to TSA’s  November 2012 plan, TSA is currently collecting little to none of the data  required to assess the performance and security effectiveness of BDA or  the SPOT program. For example, TSA does not currently collect data on  the percentage of time a BDO is present at a checkpoint or other areas in  the airport while it is open. Without this information, the assumptions  contained in TSA’s risk-based allocation analysis cannot be validated.  This analysis identified the existing BDO coverage level at the airports  where SPOT was deployed in 2011, and based its recommendations for  an additional 584 BDOs on this coverage level.
In May 2013, TSA began to implement a new data collection system,  BDO Efficiency and Accountability Metrics (BEAM), designed to track and  analyze BDO daily operational data, including BDO locations and time  spent performing different activities. According to BDA officials, this data  will allow the agency to gain insight on how BDOs are utilized, and  improve analysis of the SPOT program. The performance metrics plan  may also provide other useful information in support of some of the other  assumptions in TSA’s risk-based allocation analysis and return-on- investment analysis. For example, both analyses assumed that a BDO  can meaningfully assess 450 passengers per hour, and that fatigue would  degrade this rate over the course of a day. However, according to the  performance metrics plan, TSA does not currently collect any of the  information required to assess the number of passengers meaningfully  assessed by BDOs, BDOs’ level of fatigue, or the impact that fatigue has  on their performance. To address these and other deficiencies, the  performance metrics plan identifies 22 initiatives that are under way or  planned as of November 2012, including efforts discussed earlier in this  report, such as the indicator study and efforts to improve the SPOT  compliance teams, among others. For additional information about the  metrics that will result from these initiatives, see appendix V.
These data could help TSA assess the performance and security  effectiveness of BDA and the SPOT program, and find ways to become  more efficient with fewer resources in order to meet the federal  government’s long-term fiscal challenges, as recommended by federal  government efficiency initiatives. In lieu of these data, TSA uses arrest  and LEO referral statistics to help track the program’s activities. Of the  approximately 61,000 referrals made over the 2-year period at the 49  airports we analyzed, approximately 8,700 (14 percent) resulted in a  referral to a LEO. Of these LEO referrals, 365 (4 percent) resulted in an  arrest. The proportion of LEO referrals that resulted in an arrest (arrest  ratio) could be an indicator of the potential relationship between the  SPOT behavioral indicators and an arrest. As shown in figure 7, 99.4  percent of the passengers that were selected for referral screening—that  is further questioning and inspection by a BDO—were not arrested. The  percentage of passengers referred to LEOs that were arrested was about  4 percent; the other 96 percent of passengers referred to LEOs were not  arrested. The SPOT database identifies 6 reasons for arrest, including (1)  fraudulent documents, (2) illegal alien, (3) other, (4) outstanding warrants,  (5) suspected drugs, and (6) undeclared currency.
In February 2013, BDA officials said between 50 and 60 SPOT referrals  were forwarded by the Federal Air Marshal Service to other law  enforcement agencies for further investigation to identify potential ties to  terrorism. For example, TSA provided documentation of three  suspicious incident reports from 2011 of passengers who were referred  by BDOs to LEOs based on behavioral indicators, and who were later  found to be in possession of large sums of U.S. currency. According to a  FAMS report on these incident reports, the identification of large amounts  of currency leaving the United States could be the first step in the  disruption of funding for terrorist organizations or other form of criminal  enterprise that may or may not be related to terrorism. TSA officials said it  is difficult to identify the terrorism-related nexus in these referrals because  they are rarely, if ever, informed on the outcomes of the investigations  conducted by other law enforcement agencies, and thus have no way of  knowing if these SPOT referrals were ultimately connected to terrorism- related activities or investigations.
Standards for Internal Control in the Federal Government calls for  agencies to report on the performance and effectiveness of their  programs. However, according to the performance metrics plan, TSA  will require at least an additional 3 years and additional resources before  it can begin to report on the performance and security effectiveness of  BDA or the SPOT program. Given the scope of the proposed activities  and some of the challenges that TSA has faced in its earlier efforts to  assess the SPOT program at the national level, to complete the activities  in the time frames outlined in the plan would be difficult. In particular, the  plan notes it is unrealistic that TSA will be able to evaluate the BDO  security effectiveness contribution at each airport within the 3-year  timeframe. According to best practices for program management of  acquisitions, technologies should be demonstrated to work reliably in their  intended environment prior to program deployment. Further, according  to OMB guidance accompanying the fiscal year 2014 budget, it is  incumbent upon agencies to use resources on programs that have been  rigorously evaluated and determined to be effective, and to fix or  eliminate those programs that have not demonstrated results. TSA has  taken a positive step toward determining the effectiveness of BDA’s  behavior detection activities by developing the performance metrics plan,  as we recommended in May 2010. However, 10 years after the  development of the SPOT program, TSA cannot demonstrate the  effectiveness of its behavior detection activities. Until TSA can provide  scientifically validated evidence demonstrating that behavioral indicators  can be used to identify passengers who may pose a threat to aviation  security, the agency risks funding activities that have not been  determined to be effective.
section_title: Conclusions
paragraphs: TSA has taken several positive steps to validate the scientific basis and  strengthen program management of BDA and the SPOT program, which  has been in place for over 6 years at a total cost of approximately $900  million since 2007. Nevertheless, TSA has not demonstrated that BDOs  can consistently interpret the SPOT behavioral indicators, a fact that may  contribute to varying passenger referral rates for additional screening.  The subjectivity of the SPOT behavioral indicators and variation in BDO  referral rates raise questions about the continued use of behavior  indicators for detecting passengers who might pose a risk to aviation  security. Furthermore, decades of peer-reviewed, published research on  the complexities associated with detecting deception through human  observation also draw into question the scientific underpinnings of TSA’s  behavior detection activities. While DHS commissioned a 2011 study to  help demonstrate the validity of its approach, the study’s findings cannot  be used to demonstrate the effectiveness of SPOT because of  methodological limitations in the study’s design and data collection.
While TSA has several efforts under way to assess the behavioral  indicators and expand its collection of data to develop performance  metrics for its behavioral detection activities, these efforts are not  expected to be completed for several years, and TSA has indicated that  additional resources are needed to complete them. Consequently, after  10 years of implementing and testing the SPOT program, TSA cannot  demonstrate that the agency’s behavior detection activities can reliably  and effectively identify high-risk passengers who may pose a threat to the  U.S. aviation system.
section_title: Matter for Congressional Consideration
paragraphs: To help ensure that security-related funding is directed to programs that  have demonstrated their effectiveness, Congress should consider the  findings in this report regarding the absence of scientifically validated  evidence for using behavioral indicators to identify aviation security  threats when assessing the potential benefits of behavior detection  activities relative to their cost when making future funding decisions  related to aviation security.
section_title: Recommendation for Executive Action
paragraphs: To help ensure that security-related funding is directed to programs that  have demonstrated their effectiveness, we recommend that the Secretary  of Homeland Security direct the TSA Administrator to limit future funding  support for the agency’s behavior detection activities until TSA can  provide scientifically validated evidence that demonstrates that behavioral  indicators can be used to identify passengers who may pose a threat to  aviation security.
section_title: Agency and Third- Party Comments and Our Evaluation
paragraphs: We provided a draft of this report to DHS and the Department of Justice  (DOJ) for review and comment. We also provided excerpts of this report  to subject matter experts for their review to ensure that the information in  the report was current, correct, and factual. DOJ did not have any  comments, and we incorporated technical comments from subject matter  experts as appropriate. DHS provided written comments, which are  printed in full in appendix VI, and technical comments, which we  incorporated as appropriate.
DHS did not concur with the recommendation to the Secretary of  Homeland Security that directed the TSA Administrator to limit future  funding support for the agency’s behavior detection activities until TSA  can provide scientifically validated evidence that demonstrates that  behavioral indicators can be used to identify passengers who may pose a  threat to aviation security. Citing concerns with the findings and  conclusions, DHS identified two main areas where it disagreed with  information presented in the report: (1) the findings related to the SPOT  validation study and (2) the findings related to the research literature.  Further, DHS provided information on its investigation of profiling  allegations. We disagree with the statements DHS made in its letter, as  discussed in more detail below.
With regard to the findings related to the SPOT validation study, DHS  stated in its letter that we used different statistical techniques when we  replicated the analysis of SPOT indicators as presented in the DHS April  2011 validation study, a course of action that introduced error into our  analysis and resulted in “misleading” conclusions. We disagree with this  statement. As described in the report, we obtained the validation study  dataset from the DHS contractor and replicated the analyses using the  same techniques that the contractor used to conduct its analyses of  SPOT indicators. As an extra step, in addition to replicating the  approach (split-samples) used by the contractors, as described in  appendixes II and III of this report, we extended those analyses using the  full sample of referral data to increase our ability to detect significant  associations. In both the replication of the study analyses and the  extended analyses we conducted, we found essentially the same result in  one aspect as the validation study—that some SPOT behavioral  indicators were positively and significantly related to one or more of the  outcome measures. Specifically, the validation study reported that 14 of  the 41 SPOT behavioral indicators were positively and significantly  related, and we found that 18 of the 41 behavioral indicators were  positively and significantly related. However, the findings regarding  negatively and significantly related SPOT indicators were not consistent  between the analyses we conducted and the validation study.  Specifically, we found that 20 of the 41 behavioral indicators were  negatively and significantly related to one or more of the study outcomes  (see app. II). That is, we identified 20 SPOT behavioral indicators that  were more commonly associated with passengers who were not identified  as high-risk passengers than with passengers who were identified as  high-risk passengers. In other words, some of the SPOT indicators that  behavior detection officers are trained to detect are associated with  passengers who were defined by DHS as low risk. Our results were not  consistent with the validation study, because the study did not report any  indicators that were negatively and significantly correlated with one or  more of the outcome measures. Further, because of limitations with the  SPOT referral data that we reported in May 2010 and again in this report,  the data the validation study used to examine behavioral indicators were  not sufficiently reliable for use in conducting a statistical analysis of the  association between the indicators and high-risk passenger outcomes.  We did use these data in order to replicate the validation study findings.
Further, DHS stated in its letter that the TAC agreed with the study’s  conclusion that SPOT was substantially better at identifying high-risk  passengers than a random screening protocol. However, we disagree  with this statement. While the TAC report stated that TAC members had  few methodological concerns with the way the contractor carried out its  research, the members did not receive detailed information on the study,  including the validation study data and the final report containing the  SPOT validation study results. Specifically, as discussed in our report and  cited in the TAC report, multiple TAC members had concerns about some  of the conclusions in the validation study and suggested that the  contractor responsible for completing the study consider not reporting on  some of its results and moving the results to an appendix, rather than  including them as a featured portion of the report.
Moreover, since the TAC did not receive detailed information about the  contents of the SPOT referral report, the individual indicators used in the  SPOT program, the validation study data, or the final report containing  complete details of the SPOT validation study results, the TAC did not  have access to all of the information that we used in our analysis. As  discussed in our report, the TAC report noted that several TAC members  felt that this lack of information hampered their ability to perform their  assigned tasks. Thus, we continue to believe that our conclusion related  to the validation study results is valid, and contrary to DHS’s statement,  we do not believe that the study provides useful data in understanding  behavior detection.
With regard to the findings related to the research literature, DHS stated  in its letter that we did not consider all the research that was available and  that S&T had conducted research—while not published in academic  circles for peer review because of various security concerns—that  supported the use of behavior detection. DHS also stated that research  cited in the report “lacked ecological and external validity,” because it did  not relate to the use of behavior detection in an airport security  environment. We disagree. Specifically, as described in the report, we  reviewed several documents on behavior detection research that S&T  and TSA officials provided to us, including an unclassified and a classified  literature review that S&T had commissioned. Further, after meetings in  June and July 2013, S&T officials provided additional studies, which we  reviewed and included in the report as applicable. We also included  research in the report on the use of behavioral indicators that correspond  closely to indicators identified in SPOT procedures as indicative of stress,  fear, or deception. These studies, many of which were included in the  meta-analyses we reviewed, were conducted in a variety of settings— including high-stakes situations where the consequences are great, such  as a police interview with an accused murderer—and with different types  of individuals—including law enforcement personnel. The meta-analyses  we reviewed—which collectively included research from over 400  separate studies related to detecting deception conducted over the past  60 years—found that the ability of human observers to accurately identify  deceptive behavior based on behavioral cues or indicators is the same as  or slightly better than chance (54 percent).
Further, in its letter, DHS cited a 2013 RAND report, which concluded that  there is current value and unrealized potential for using behavioral  indicators as part of a system to detect attacks. We acknowledge that  behavior detection holds promise for use in certain circumstances and in  conjunction with certain other technologies. However, the RAND report  DHS cited in its letter refers to behavioral indicators that are defined and  used significantly more broadly than those in the SPOT program. The  indicators reviewed in the RAND report are neither used in the SPOT  program, nor could be used in real time in an airport environment.  Further, the RAND report findings cannot be used to support TSA’s use of  behavior detection activities because the study stated that it could not  make a determination of SPOT’s effectiveness because information on  the program was not in the public domain.
DHS also stated in its letter that it has several efforts under way to  improve its behavior detection program and the methodologies used to  evaluate it, including the optimization of its behavior detection procedures  and plans to begin testing by the third quarter of fiscal year 2014 using  robust test and evaluation methods similar to the operational testing  conducted in support of technology acquisitions as part of its 3-year  performance metrics plan. We are encouraged by TSA’s plans in this  area. However, TSA did not provide supporting documentation  accompanying these plans describing how it will incorporate robust data  collection and authentication protocols, as discussed in DHS’s letter.  Such documentation is to be completed prior to beginning any operational  testing. These documents might include a test and evaluation master plan  that would describe, among other things, the tests that needed to be  conducted to determine system technical performance, operational  effectiveness or suitability, and any limitations.
Additionally, in its letter, DHS stated that the omission of research related  to verbal indicators of deception was misleading because a large part of  BDOs’ work is interacting with passengers and assessing whether  passengers’ statements match their behaviors, or if the passengers’ trip  stories are in agreement with their travel documents and accessible  property. While BDOs’ interactions with passengers may elicit useful  information, SPOT procedures indicate that casual conversation— voluntary informal interviews conducted by BDOs with passengers  referred for additional screening—is conducted after the passengers have  been selected for a SPOT referral, not as a basis for selecting the  passengers for referral. Further, since these interviews are voluntary,  passengers are under no obligation to respond to the BDOs questions,  and thus information on passengers may not be systematically collected.  As noted in our report, promising research on behavioral indicators cited  in the RAND report and other literature is focused on using indicators in  combination with automated technologies and certain interview  techniques, such as asking unanticipated questions. However, when  interviewing referred passengers for additional screening, BDOs do not  currently have access to the automated technologies discussed in the  RAND report.
Further, DHS stated that the goal of the SPOT program is to identify  individuals exhibiting behavior indicative of simple emotions such as fear  or stress and reroute them to a higher level of screening, and does not  attempt to specifically identify persons engaging in lying or terrorist acts.  However, DHS also stated in its response that “SPOT uses a broader  array of indicators, including stress and fear detection as they relate to  high-stakes situations where the consequences are great, for example,  suicide attack missions.” As noted in the report, TSA’s program and  budget documents associated with behavior detection activities identify  that the purpose of these activities is to identify high-risk passengers  based on behavioral indicators that indicate mal-intent. For example, the  strategic plan notes that in concert with other security measures, behavior  detection activities “must be dedicated to finding individuals with the intent  to do harm, as well as individuals with connections to terrorist networks  that may be involved in criminal activity supporting terrorism.” The  conclusions, which were confirmed in discussions with subject matter  experts and an independent review of studies, indicate that scientifically  validated evidence does not support whether the use of behavioral  indicators by unaided human observers can be used to identify  passengers who may pose a threat to aviation security.   DHS also cited the National Research Council’s 2008 report to support its  use of SPOT. The National Research Council report, which we  reviewed as part of our 2010 review of the SPOT program, noted that  behavior and appearance monitoring might be able to play a useful role in  counterterrorism efforts but also stated that a scientific consensus does  not exist regarding whether any behavioral surveillance or physiological  monitoring techniques are ready for use in the counterterrorist context,  given the present state of the science. According to the National  Research Council report, an information-based program, such as a  behavior detection program, should first determine if a scientific  foundation exists and use scientifically valid criteria to evaluate its  effectiveness before going forward. The report also stated that programs  should have a sound experimental basis, and documentation on the  program’s effectiveness should be reviewed by an independent entity  capable of evaluating the supporting scientific evidence.
With regard to information provided related to profiling, DHS stated that  DHS’s OIG completed an investigation at the request of TSA into  allegations that surfaced at Boston Logan Airport and concluded that  these allegations could not be substantiated. However, while the OIG’s  July 2013 report of investigation on behavior detection officers in Boston  concluded that “there was no indication that BDOs racially profiled  passengers in order to meet production quotas,” the OIG’s report also  stated that there was evidence of “appearance profiling.”
In stating its nonconcurrence with the recommendation to limit future  funding in support of its behavior detection activities, DHS stated that  TSA’s overall security program is composed of interrelated parts, and to  disrupt one piece of the multilayered approach may have an adverse  impact on other pieces. Further, DHS stated that the behavior detection  program should continue to be funded at current levels to allow BDOs to  screen passengers while the optimization process proceeds. We  disagree. As noted in the report, TSA has not developed the performance  measures that would allow it to assess the effectiveness of its behavior  detection activities compared with other screening methods, such as  physical screening. As a result, the impact of behavior detection activities  on TSA’s overall security program is unknown. Further, not all screening  methods are present at every airport, and TSA has modified the  screening procedures and equipment used at airports over time. These  modifications have included the discontinuance of screening equipment  that was determined to be unneeded or ineffective.
Therefore, we continue to believe that providing scientifically validated  evidence that demonstrates that behavioral indicators can be used to  identify passengers who may pose a threat to aviation security is critical  to the implementation of TSA’s behavior detection activities. Further,  OMB guidance highlights the importance of using resources on programs  that have been rigorously evaluated and determined to be effective, and  best practices for program management of acquisitions state that  technologies should be demonstrated to work reliably in their intended  environment prior to program deployment. Consequently, we have  added a matter for congressional consideration to this report to help  ensure that TSA provides information, including scientifically validated  evidence, which supports the continued use of its behavior detection  activities in identifying threats to aviation security.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 5 days from the  report date. We are sending copies of this report to the Secretary of  Homeland Security; the TSA Administrator; the United States’ Attorney  General; and interested congressional committees as appropriate. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-4379 or lords@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are acknowledged in  appendix VII.
section_title: Appendix I: Information on Recent Allegations of Passenger Profiling and TSA’s Actions to Address Such Allegations
paragraphs: According to the Screening of Passengers by Observation Techniques  (SPOT) program’s standard operating procedures, behavior detection  officers (BDO) must apply the SPOT behavioral indicators to passengers  without regard to race, color, religion, national origin, ethnicity, sexual  orientation, or disability.
Since 2010, the Transportation Security Administration (TSA) and the  Department of Homeland Security’s (DHS) Office of Inspector General  (OIG) have examined allegations of the use of profiling related to the  race, ethnicity, or nationality of passengers by behavior detection officers  (BDO) at three airports—Newark Liberty International Airport (Newark),  Honolulu International Airport (Honolulu), and Boston Logan International  Airport (Boston)—and TSA has taken action to address these allegations.  Specifically, in January 2010, TSA concluded an internal investigation at  Newark of allegations that BDOs used specific criteria related to the race,  ethnicity, or nationality of passengers in order to select and search those  passengers more extensively than would have occurred without the use  of these criteria. The investigation was conducted by a team of two BDO  managers from Boston to determine whether two BDO managers at  Newark had established quotas for SPOT referrals to evaluate the  performance of their subordinate BDOs. The investigation also sought to  determine whether these managers at Newark encouraged profiling of  passengers in order to meet quotas that they had established. The  investigating team concluded that no evidence existed to support the  allegation of a quota system, but noted widespread BDO perception that  higher referral rates led to promotion, and that the “overwhelming majority  of BDOs” expressed concern that the BDO managers’ “focus was solely  on increasing the number of referrals and LEO calls.” The investigating  team said the information collected regarding the allegation of profiling  resulted in a reasonable conclusion that that such activity was both  directed and affected on a limited basis at Newark, based on one  manager’s inappropriate direction to BDOs regarding profiling of  passengers, racial comments, and the misuse of information intended for  situational awareness purposes only. According to TSA officials,  disciplinary action taken against this manager resulted in the manager’s  firing.
Additionally, in 2011, TSA’s Office of Inspection (OOI) conducted an  investigation of racial profiling allegations against BDOs at Honolulu. The  investigation consisted of a review of Equal Employment Opportunity  (EEO) complaints, and OOI did not find evidence to support the profiling  allegations in the SPOT program.
In July 2012, OOI conducted a compliance inspection at Boston, during  which allegations of profiling by BDOs surfaced. Specifically, during  interviews with inspectors, allegations surfaced that BDOs were profiling  passengers for the purpose of raising the number of law enforcement  referrals. These accusations included written complaints from BDOs who  claimed other BDOs were selecting passengers for referral screening  based on their ethnic or racial appearance, rather than on the basis of the  SPOT behavioral indicators and were reported in a September 2012 OOI  memorandum. These allegations were referred to the OIG, and in August  2012, the OIG opened an investigation into these profiling allegations in  Boston. According to OIG officials, its investigation was completed and its  final report was provided to TSA in August 2013.
In August 2012, the Secretary of Homeland Security issued a  memorandum directing TSA to take a number of actions in response to  allegations of racial profiling by BDOs. These actions include (1) a  revision of the SPOT standard operating procedures to, among other  things, clarify that passengers who are unwilling or uncomfortable with  participating in an interactive discussion and responding to questions will  not be pressured by BDOs to do so; (2) refresher training for all BDOs  that reinforces antidiscrimination requirements; and (3) TSA  communication with BDO supervisors that performance appraisals should  not depend on achieving either a high number of referrals or on the arrest  rate coming from those referrals, but rather from demonstrated vigilance  and skill in applying the SPOT procedures. As of June 2013, TSA,  together with the DHS Acting Officer for Civil Rights and Civil Liberties  and Counsel to the Secretary of Homeland Security, had completed  several of these action items and others were under way. For example,  the Secretary of Homeland Security sent a memo to all DHS component  heads in April 2013 stating that it is DHS’s policy to prohibit the  consideration of race or ethnicity in DHS’s investigation, screening, and  enforcement activities in all but the most exceptional instances.
During our visits to four airports, we asked a random sample of 25 BDOs  at the airports to what extent they had seen BDOs in their airport referring  passengers based on race, national origin, or appearance rather than  behaviors. These responses are not generalizable to the entire BDO  population at SPOT airports. Of the 25 randomly selected BDOs we  interviewed, 20 said they had not witnessed profiling, and 5 BDOs  (including at least 1 from each of the four airports we visited) said that  profiling was occurring at their airports, according to their personal  observations. Also, 7 additional BDOs contacted us over the course of  our review to express concern about the profiling of passengers that they  had witnessed. We did not substantiate these specific claims.
In an effort to further assess the race, sex, and national origin of  passengers who were referred by BDOs for additional screening, we  analyzed the available information in the SPOT referral database and the  Federal Air Marshal Service’s (FAMS) Transportation Information Sharing  System (TISS) database. However, we found that the SPOT referral  database does not allow for the recording of information such as race or  gender. Without recording these data for every referral, it is difficult to  disprove or substantiate such accusations. Since program-wide data on  race were not available in the SPOT database, we analyzed a subset of  available arrest data that were entered into the TISS database, which  allows for race to be recorded. However, because there is not a unique  identifier to link referrals from the SPOT database to information entered  into TISS, we experienced obstacles when we attempted to match the  two databases. For the SPOT referrals we were able to match, we found  that data on race were inconsistently recorded in TISS. The limitations  associated with matching the two databases and the incompleteness of  the race data in TISS made analyzing trends or anomalies in the data  impractical.
In March 2013, BDA officials stated that they had initiated a feasibility  study to determine the efficacy of collecting data on the race and national  origin of passengers referred by BDOs. A pilot is to be conducted at  approximately five airports, which have not yet been selected, to collect  data and examine whether this type of data collection is feasible and if the  data can be used to identify any airport-specific or system-wide trends in  referrals. According to BDA officials, the purpose of this study is to  examine whether disparities exist in the referral trends, and if so, whether  these differences suggest discrimination or bias in the referral process.  This pilot is to also include an analysis of the broader demographics of  the flying public—not just those referred by BDOs for additional  screening—which is information that TSA had not previously collected.  Having additional information on the characteristics of the flying public  that may be used to compare to the characteristics of those passengers  referred by the SPOT program—if TSA determines these data can  feasibly be collected—could help enable TSA to reach reasonable  conclusions about whether allegations of passenger profiling can be  substantiated.
section_title: Appendix II: Our Analysis of Validation Study Data on SPOT Behavioral Indicators
paragraphs: The validation study reported that 14 of the 41 SPOT behavioral  indicators were positively and significantly related to one or more of the  study outcomes, but did not report that any of the indicators were  negatively and significantly related to the outcome measures. That is,  passengers exhibiting the SPOT behaviors that were positively and  significantly related were more likely to be arrested, to possess fraudulent  documents, or possess prohibited or illegal items. Conversely,  passengers exhibiting the behaviors that were negatively and significantly  related were less likely to be arrested, to possess fraudulent documents,  or possess serious prohibited or illegal items than those who did not  exhibit the behavior. While recognizing that the SPOT referral data used  in this analysis were potentially unreliable, we replicated the SPOT  indicator analysis with the full set of SPOT referral cases from January 1,  2006, to October 31, 2010, and found, consistent with the validation  study, that 18 of the 41 behavioral indicators were positively and  significantly related to one or more of the outcome measures. We also  found, however, that 20 of the 41 behavioral indicators were negatively  and significantly related to one or more of the study outcomes. That is,  we identified 20 SPOT behavioral indicators that were more commonly  associated with passengers who were not identified as high-risk  passengers, than with passengers who were identified as high-risk  passengers. Of the 41 behavioral indicators in the analysis, almost half of  the passengers referred by BDOs for referral screening exhibited one  indicator.
section_title: Appendix III: Objectives, Scope, and Methodology
subsections: section_title: Objectives
paragraphs: This report addresses the following questions:  1.  To what extent does available evidence support the use of behavioral  indicators to identify aviation security threats?  2.  To what extent does TSA have data necessary to assess the  effectiveness of the SPOT program in identifying threats to aviation  security?
In addition, this report provides information on TSA’s response to recent  allegations of racial profiling in the SPOT program, which can be found in  appendix I.
section_title: Overview of Our Scope and Methodology
paragraphs: To obtain background information and identify changes in the SPOT  program since our May 2010 report, we conducted a literature search to  identify relevant reports, studies, and articles on passenger screening and  deceptive behavior detection. We reviewed program documents in place  during the period October 2010 through June 2013, including SPOT  standard operating procedures, behavior detection officer performance  standards and guidance, a strategic plan, and a performance metrics  plan. We met with headquarters TSA and Behavior Detection and  Analysis (BDA) program officials to determine the extent to which TSA  had implemented recommendations in our May 2010 report and obtain an  update on the SPOT program. In addition, we met with officials from U.S.  Customs and Border Protection and the Federal Bureau of Investigation  (FBI) Behavioral Science Unit to determine the extent to which they use  behavior detection techniques. We also interviewed officials in DHS’s  OIG, who were working on a related audit.
We analyzed data for fiscal years 2011 and 2012 from TSA’s SPOT  referral database, which is to record all incidents in which BDOs refer  passengers for additional screening, including the airport, time and date  of the referral, the names of the BDOs involved in the referral, BDOs’  observation of the passengers’ behaviors, and any actions taken by law  enforcement officers, if applicable. We also analyzed data for fiscal years  2011 and 2012 from the FAMS Transportation Information Sharing  System (TISS) database, which is a law enforcement database designed  to retrieve, assess, and disseminate intelligence information regarding  transportation security to FAMS and other federal, state, and local law  enforcement agencies. We reviewed available documentation on these  databases, such as user guides, data audit reports, and training  materials, and interviewed individuals responsible for maintaining these  systems. In addition, we analyzed data on BDOs working at airports  during this 2-year period, such as date started at TSA, date started as  BDO, race, gender, and performance rating scores from TSA’s Office of  Human Capital, and data on the number of hours worked by these BDOs  provided by TSA’s Office of Security Operations officials and drawn from  the U.S. Department of Agriculture’s National Finance Center database,  which handles payroll and personnel data for TSA and other federal  agencies. Further, we analyzed financial data from fiscal years 2007  through 2012 provided by BDA to determine the expenditures associated  with the SPOT program. Additional information about steps we took to  assess the reliability of these data is discussed below. We interviewed  BDA officials in the Office of Security Capabilities and the Office of  Human Capital on the extent to which they collect and analyze these  data.
We conducted visits to four airports—Orlando International in Orlando,  Florida; Detroit Metropolitan Wayne County in Detroit, Michigan; Logan  International in Boston, Massachusetts; and John F. Kennedy  International in New York City, New York. We selected this nonprobability  sample based on the airports’ size and participation in behavior detection  programs. As part of our visits, we interviewed a total of 25 BDOs using  a semi-structured questionnaire, and their responses are not  generalizable to the entire BDO population at SPOT airports. These  BDOs were randomly selected from a list of BDOs on duty at the time of  our visit. We interviewed BDO managers and TSA airport managers, such  as federal security directors, who oversee the SPOT program at the  airports. In addition, to obtain law enforcement officials’ perspectives on  the SPOT program and their experiences in responding to SPOT  referrals, we interviewed officials from the local airport law enforcement  agency with jurisdiction at the four airports we visited (Orlando Police  Department, Wayne County Airport Authority, Massachusetts State  Police, and Port Authority of New York and New Jersey) and federal law  enforcement officials assigned to the airports, including U.S. Customs  and Border Protection, the FBI, and U.S. Immigration and Customs  Enforcement. In nonprobability sampling, a sample is selected from  knowledge of the population’s characteristics or from a subset of a  population where some units in the population have no chance, or an  unknown chance, of being selected. A nonprobability sample may be  appropriate to provide illustrative examples, or to provide some  information on a specific group within a population, but it cannot be used  to make inferences about a population or generalize about the population  from which the sample is taken. The results of our visits and interviews  provided perspectives about the effectiveness of the SPOT program from  local airport officials and opportunities to independently observe TSA’s  behavior detection activities at airports, among other things.
subsections: section_title: Validation Study
paragraphs: To assess the soundness of the methodology and conclusions in the DHS  April 2011 validation study, we reviewed the validation study and  Technical Advisory Committee (TAC) final reports and appendixes, and  other documents, such as the contractor’s proposed study designs,  contracts to conduct the study, data collection training materials, and  interim reports on data monitoring visits and study results. We assessed  these efforts with established practices in designing evaluations and  generally accepted statistical principles. We obtained the validation study  datasets from the contractor and replicated several of the analyses,  based on the methodology described in the final report. Generally, we  replicated the study’s split-sample analyses, and as an extra step,  extended those analyses using the full sample of SPOT referral data, as  discussed below and in appendix II. In addition, we interviewed  headquarters TSA, BDA, and Science and Technology Directorate (S&T)  officials responsible for the validation study, representatives from the  contractor who conducted the study, and 8 of the 12 members of the TAC  who commented on and evaluated the adequacy of the validation study  and issued a separate report in June 2011.
section_title: Data Reliability
paragraphs: To assess the reliability of the SPOT referral data, we reviewed relevant  documentation, including privacy impact assessments and a 2012 data  audit of the SPOT database, and interviewed TSA and BDA headquarters  and field officials about the controls in place to maintain the integrity of  the data. To determine the extent to which the SPOT database is  accurate and complete, we reviewed the data in accordance with  established procedures for assessing data reliability and conducted tests,  such as electronic tests to determine if there were anomalies in the  dataset (such as out-of-range dates and missing data) and reviewed a  sample of certain coded data fields and compared them with narrative  information in the open text fields. We determined that the data for fiscal  years 2011 and 2012 across the 49 airports in our scope were sufficiently  reliable for us to use to reflect the total number of SPOT referrals and  arrests made, and to standardize the referral and arrest data, based on  the number of hours each BDO spent performing operational SPOT  activities.
In October 2012, TSA completed an audit of the data contained in the  SPOT referral database in which it identified common errors, such as  missing data fields and incorrect point totals. According to the 2012 audit,  for the time period of March 1, 2010, through August 31, 2012, covering  more than 108,000 referrals, the SPOT referral database had an overall  error rate of 7.96 percent, which represented more than 8,600 known  errors and more than 14,000 potential errors. According to TSA, the  agency has begun taking steps to reduce this error rate, including visits to  airports with significant data integrity issues and the development of a  new SPOT referral database that is designed to prevent the most  common errors from occurring. BDA officials told us that they have begun  steps toward a nationwide rollout of their new system in May 2013, which  includes pilots and developing procedures to mandate airports’ use of the  system. On the basis of our review of the types of errors identified by the  data audit, we determined that the SPOT referral data were sufficiently  reliable for us to analyze BDO referral rates. However, the audit identifies  problems with arrest data, which is one of the three categories of  “potential errors.” The audit does not report on the magnitude of this error  category, because identifying these errors requires a manual audit of the  data at the airport level. As a result, we determined that the arrest data  were not reliable enough for us to report on details about the arrests.
section_title: Use of Behavioral Indicators
paragraphs: To determine the extent to which available evidence exists to support the  use of behavioral indicators to identify security threats, we analyzed  research on behavioral indicators, reviewed the validation study findings  on behavioral indicators, and analyzed SPOT referral data.
subsections: section_title: Research on Behavioral Indicators
paragraphs: Working from a literature review of articles from 2003 to 2013 that were  identified using search terms such as “behavior detection deception,” and  discussions with researchers who had published articles in this area, we  contacted other researchers to interview and academic and government  research to review. While the results of our interviews cannot be used to  generalize about all research on behavior deception detection, they  represent a mix of researchers and views by virtue of their affiliation with  various academic institutions and governments, authorship of meta- analyses on these issues, and subject matter expertise in particular  research areas.
We also reviewed more than 40 articles and books on behavior-based  deception detection dating from 1999 to 2013. These articles, books, and  reports were identified by our literature search of databases, such as  ArticleFirst, ECO, WorldCat, ProQuest, and Academic One File and  recommendations by TSA and the experts we interviewed. Through our  discussions and research, we identified four meta-analyses, which used  an approach for statistically cumulating the results of several studies to  answer questions about program impacts. These meta-analyses analyzed  “effect sizes” across several studies—the measure of the difference in  outcome between a treatment group and a comparison group. For  example, these meta-analyses measured the accuracy of an individual’s  deception judgments when assessing another individual’s credibility in  terms of the percentage that lies and truths were correctly classified and  the impact of various factors on the accuracy of deception judgments,  such as the liar’s motivation or expertise of the individual making the  judgment. We reviewed the methodologies of 4 meta-analyses covering  over 400 separate studies on detection deception over a 60-year period,  including whether an appropriate evaluation approach was selected for  each meta-analysis, and whether the data were collected and analyzed in  ways that allowed valid conclusions to be drawn, in accordance with  established practices in evaluation design. In addition, we interviewed  two authors of these meta-analyses to ensure that the analyses were  sound and we determined that the analyses were sufficiently reliable for  describing what evidence existed to support the use of behavioral  indicators to identify security threats. We determined that the research we  identified was sufficiently reliable for describing the evidence that existed  regarding the use of behavioral indicators to identify security threats.
Further, we reviewed documents developed by TSA and other foreign  countries as part of an international study group to assess TSA’s efforts  to identify best practices on the use of behavioral detection in an airport  environment.
section_title: Validation Study Results on SPOT Indicators
paragraphs: To assess the soundness of the methodology and conclusions in the April  2011 validation study finding that 14 of the 41 SPOT indicators were  related to outcomes that indicate a possible threat, we reviewed evidence  supporting our May 2010 conclusions that the SPOT referral database  lacked controls to help ensure the completeness and accuracy of the  data. We interviewed TSA officials and obtained documentation, such as  a data audit report and a functional requirements document, to determine  the extent to which problems in the SPOT database were being  addressed. We also reviewed the June 2011 TAC final report and  interviewed contractor officials regarding analysis limitations because of  data sparseness, or low frequency of occurrences of indicators in the  SPOT database.
We also obtained the dataset used in the study—SPOT referral data from  January 2006 through October 2010—and replicated the SPOT indicator  analyses described in the study. Although we found that the data were  not sufficiently reliable for use in conducting a statistical analysis of the  association between the indicators and high-risk passenger outcomes, we  used the data to assess the study’s methodology and conclusions. The  dataset included a total of 247,630 SPOT referrals from 175 airports. As  described in the validation study, we calculated whether the odds on each  of the four study outcome measures—LEO arrest, possession of  fraudulent documents, possession of a serious prohibited or illegal item,  or the combination of all three measures—were associated with the 41  SPOT indicators. These odd ratios were derived from four sets of 41  separate cross-tabulations—2 x 2 tables—in which each of the four  outcomes is cross-classified by each of the 41 individual indicators. Odds  ratios greater than 1.0 indicate positive associations, that is, passengers  exhibiting the behavior were more likely to be arrested, to possess  fraudulent documents, or to possess serious prohibited or illegal items.  On the other hand, odds ratios of less than 1.0 indicate negative  associations, that is, passengers exhibiting the behavior were less likely  to be arrested, to possess fraudulent documents, or to possess serious  prohibited or illegal items than those who do not exhibit the behavior. The  number of positive and significant associations we detected was slightly  larger than the number reported in the validation study mainly because  we reported results from an analysis of the full sample of SPOT  referrals—a total of 247,630 SPOT passenger referrals. In contrast, the  validation study stated that a split-sample approach was used, in which  each years’ dataset was split into two stratified random subsets across  the years and analyses were conducted independently on each  aggregated subset. The validation study stated that this approach allowed  an examination of the extent to which results may vary across each  subset and to address possible random associations in the data. The  validation study further stated that this was important because changes in  the SPOT program, such as fewer airports and BDOs involved in the  earlier years and small changes to the SPOT instrument in March 2009,  could have affected the analyses. However, after replicating the split- sample approach, we determined that it was not the most appropriate one  to use because it substantially diminished the power to detect significant  associations in light of how infrequently referrals occurred. We report the  results of our analyses of the full sample of SPOT referrals that indicate  behavioral indicators that are positively and significantly related, as well  as negatively and significantly related, in the behavioral indicator section  of the report and in appendix II.
section_title: SPOT Referral Data
paragraphs: To determine the extent to which SPOT referrals varied by BDOs across  airports for fiscal years 2011 and 2012, we initially selected the 50  airports identified by TSA’s May 2012 Current Airports Threat  Assessment report as having the highest probability of threat from  terrorist attacks. We chose to limit the scope of our review to the top 50  airports because the majority of the BDOs are deployed to these airports;  and they account for 68 percent of the passenger throughput, and 75  percent of SPOT referrals. To standardize the referral rates across  airports, we calculated the number of SPOT referrals by individual BDOs  and matched these BDOs by the number of hours that particular BDOs  spent performing SPOT activities. San Francisco International Airport  was in the initial selection of 50 airports; however, we excluded San  Francisco International because the hourly data provided to us for San  Francisco BDOs, who are managed by a screening contractor, were not  comparable with the hourly data provided to us for TSA-managed  BDOs. The scope of our analysis was then 49 SPOT airports.
To calculate BDO hours spent performing SPOT activities, we analyzed  BDO time and attendance data provided by TSA for fiscal years 2011 and  2012 from the U.S. Department of Agriculture’s National Finance Center.  We limited our analysis to the hours BDOs spent performing SPOT  activities because it is primarily during these times that BDOs make  SPOT referrals. Thus, BDO hours charged to activities such as leave,  baggage screening, or cargo inspection activities were excluded. For  example, we found that BDOs had charged time to cargo inspection  activities that were unrelated to the SPOT program. These inspections  are carried out under TSA’s Compliance Division in the Office of Security  Operations, and are designed to ensure compliance with transportation  security regulations. We also limited our analysis to nonmanager BDOs,  as managers are not regularly engaged in making referrals. Finally, about  55 BDOs, or about 2 percent of the approximately 2,400 BDOs (including  both managers and nonmanagers), were not included in our analysis  because we could not reconcile their names with time and attendance  data after several attempts with TSA officials. We calculated average  referral rates per 160 hours worked, or about 4 40-hour weeks, across  2,199 BDOs working at 49 airports, and a referral rate for each airport.
To better understand the variation in referral rates, we conducted a  multivariate analysis to determine whether certain variables affected  SPOT referral rates and LEO referral rates, including airports at which  BDOs worked during fiscal years 2011 and 2012; BDO annual  performance scores for 2011 and 2012; years of experience with TSA  and as a BDO; and demographic information on BDOs, such as age,  gender, race, and highest educational level attained at the time of  employment. Although multivariate methods do not allow us to establish  that referral rates are causally related to the BDO characteristics we had  information about, they allowed us to examine the associations between  referral rates and the different specific BDOs while controlling for other  BDO characteristics, including the airports in which the BDOs worked.
Moreover, the methods we employed allowed us to determine whether  the observed differences in the sample data were different more than by  merely chance fluctuations. Our statistical models and estimates are  sensitive to our choice of variables; thus, researchers testing different  variables may find different results. See appendix IV for additional  information on the results of our analyses.
section_title: Data to Assess Effectiveness of SPOT
paragraphs: To determine the extent to which TSA has data necessary to assess the  effectiveness of the SPOT program in identifying threats to aviation  security, we reviewed the validation study’s findings comparing  passengers selected by SPOT with randomly selected passengers,  analyzed TSA plans and analyses designed to measure SPOT’s  effectiveness, and analyzed data on SPOT referrals and LEO arrests.
subsections: section_title: Validation Study Results Comparing Passengers Selected by SPOT with Randomly Selected Passengers
paragraphs: To assess the soundness of the methodology and conclusions in the April  2011 validation study findings that SPOT was more likely to identify high- risk passengers than a random selection of passengers, we assessed the  study design and implementation against established practices for  designing evaluations and generally accepted statistical principles. These  practices include, for example, probability sample methods, data  collection and monitoring procedures, and quasi-experimental design.  We obtained the validation study datasets and replicated the study  findings, based on the methodology described in the final report. Further,  we analyzed the validation study data from December 1, 2009, to October  31, 2010, on passengers who were referred to a LEO and who were  ultimately arrested. To the extent possible, we reviewed SPOT data to  determine the reasons for the arrest and if there were differences  between arrested passengers who were referred by SPOT and arrested  passengers who were randomly selected.
section_title: TSA Performance Data
paragraphs: To determine the extent to which TSA has plans to collect and analyze  performance data to assess SPOT’s overall effectiveness, we reviewed  TSA’s efforts to inform the future direction of BDA and the SPOT  program, such as a return-on-investment and risk-based allocation  analyses. We evaluated TSA’s efforts against DHS, GAO, and other  guidance regarding these analyses. For example, we reviewed TSA’s  return-on-investment analysis against the analytical standards in the  Office of Management and Budget’s Circular A-94, which provides  guidance on conducting benefit-cost and cost-effectiveness analyses.  We also reviewed documentation associated with program oversight,  including a 2012 performance metrics plan, and evaluated TSA’s efforts  to collect and analyze data to provide oversight of BDA and the SPOT  program against criteria in Office of Management and Budget guidance  and Standards for Internal Control in the Federal Government. Further,  we reviewed performance work statements in TSA contracts to determine  the extent to which the contractor’s work is to fulfill the tasks in TSA’s  performance metrics plan. Also, we reviewed FAMS law enforcement  reports, TISS incident reports, and the SPOT referral database to  determine the extent to which information from BDO referrals was used  for further investigation to identify potential ties to terrorist investigations.  We also analyzed SPOT referral data that TSA uses to track SPOT  program activities, including the number of passengers who were referred  to a LEO and ultimately arrested for fiscal years 2011 and 2012.
section_title: Profiling Allegations
paragraphs: To provide information about how TSA and DHS’s OIG have examined  allegations of racial and other types of profiling of passengers by BDOs,  we reviewed documentation from 2010 to 2013, such as investigation  reports, privacy impact assessments, BDO training materials, and TSA  memos. To explore the extent to which we could determine the race,  gender, and national origin of passengers who were referred by BDOs for  additional screening, we analyzed information in the SPOT referral  database and the TISS database for fiscal years 2011 and 2012. We  reviewed a September 2012 TSA contract that will, among other things,  study whether any evidence exists for racial or ethnic profiling in the  SPOT program. We also reviewed interim reports produced by the  contractor as of June 2013. Because racial profiling allegations in Boston  were made during the course of our review, we asked the random sample  of 25 BDOs at the four airports we visited to what extent they had seen  BDOs in their airport referring passengers based on race, national origin,  or appearance rather than behaviors. These responses are not  generalizable to the entire BDO population at SPOT airports. Further, 7  additional BDOs contacted us over the course of our review to express  concern about the profiling of passengers that they had witnessed. We  did not substantiate these specific claims. We also interviewed TSA  headquarters and field officials, such as federal security directors and  BDO managers, as well as DHS OIG officials.
We conducted this performance audit from April 2012 to November 2013  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe the  evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix IV: Characteristics and Referral Rates of Behavior Detection Officers at 49 SPOT Airports
paragraphs: To better understand the variation in referral rates, we analyzed whether  certain variables affected SPOT referral rates and LEO referral rates,  including BDO characteristics, such as average performance scores for  fiscal years 2011 and 2012, years of TSA and BDO experience, age,  gender, educational level, years employed at TSA and as a BDO, and  race, as well as the airport in which the BDOs worked. As described  earlier, these analyses standardized SPOT referral data for 2,199 BDOs  across 49 airports for fiscal years 2011 and 2012.
subsections: section_title: BDO Characteristics and Referral Rates
paragraphs: The characteristics of the 2,199 BDOs in our analyses varied across  different categories, as shown in table 3. About 51 percent of the BDOs  were under 40 years of age, and slightly more than 25 percent were 50  years or older. Nearly 64 percent of the BDOs joined TSA before the end  of 2005, but the majority, or more than 85 percent, became BDOs after  the beginning of 2008. Nearly 65 percent of the BDOs were male. Fifty  percent were white, about 26 percent were African-American, and about  18 percent were Hispanic. About 65 percent of the BDOs had a high  school education or less. The BDOs were distributed unevenly across  airports, with the largest numbers in Logan International (Boston), Dallas- Fort Worth International, John F. Kennedy International (New York), Los  Angeles International, and O’Hare International (Chicago). Each BDO  worked primarily in one airport during the 2-year period. For example, 80  of the 2,199 BDOs, or about 4 percent, worked in multiple airports and the  remaining 2,119 BDOs, or 96 percent, worked at one airport during the 2- year time period.
Overall, BDOs averaged about 1.57 SPOT referrals and 0.22 LEO  referrals per 160 hours worked. These rates vary across the different  BDO categories. However, these differences should be considered  cautiously, as differences that appear to exist across categories for one  characteristic may be confounded with differences across others. For  example, the apparent difference in referral rates between younger and  older BDOs may be the result of younger BDOs working  disproportionately in airports with higher referral rates.
section_title: Multivariate Analysis of SPOT and LEO Referral Rates
paragraphs: To better understand the effects of BDO characteristics, including the  airports they worked in, on SPOT referral and LEO referral rates, we  conducted simple regression analyses. Overall, the greatest amount of  the variation in BDO SPOT referral rates was explained by the airport at  which the referral occurred. That is, the BDO’s referral rate was  associated substantially with the airport at which he or she was  conducting SPOT activities.
These analyses show the size and significance of regression coefficients, from ordinary  least-squares regression models, which reflect the estimated differences in the average  number of SPOT referrals and LEO referrals across categories of BDO, and across  airports.
BDOs in a few airports averaging significantly higher rates of referrals  than BDOs in the referent category, and BDOs in most of the other  airports averaging significantly lower LEO referral rates. Because they  were less common, LEO referrals may have been more difficult to predict  that SPOT referrals. Differences in the other BDO characteristics— multivariate model 1—collectively accounted for a small percentage of the  variation in average LEO referral rates, while differences across airports  accounted for a larger percentage.
section_title: Airport Throughput Analysis
paragraphs: Separate analyses we conducted revealed that the sizeable and highly  significant differences in SPOT referral rates and LEO referral rates  across airports were not fully accounted for by differences in the number  of passengers who pass through airport checkpoints.
section_title: Appendix V: TSA’s Performance Metrics for Behavior Detection and Analysis
paragraphs: Table 4 shows TSA’s proposed performance metrics as detailed in  appendix G in its Behavior Detection and Analysis performance metrics  plan dated November 2012.
Table 5 shows the validity, reliability, and frequency score TSA  determined for each metric and the overall score for each metric  subcategory, as detailed in appendix C of its performance metrics plan,  dated November 2012. TSA’s performance metrics plan defines validity  as the ability of the metric to measure BDO performance, reliability as the  level of certainty that data are collected precisely with minimal possibility  for subjectivity or gaming the system, and frequency as the level of  difficulty in collecting the metric and whether the metric is collected at the  ideal number of scheduled recurrences.
section_title: Appendix VI: Comments from the Department of Homeland Security
section_title: Appendix VII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, David M. Bruno (Assistant  Director); Charles W. Bausell, Jr.; Andrew M. Curry; Nancy K. Kawahara;  Elizabeth B. Kowalewski; Susanna R. Kuebler; Thomas F. Lombardi;  Grant M. Mallie; Amanda K. Miller; Linda S. Miller; Lara R. Miklozek;  Douglas M. Sloane; and Jeff M. Tessin made key contributions to this  report.