Aviation Security: TSA Should Limit Future Funding for Behavior Detection Activities
Released: 2013-11-13T00:00:00

Letter

Background

BDA and the SPOT Program

The Aviation and Transportation Security Act established TSA as the federal agency with primary responsibility for securing the nation’s civil aviation system, which includes the screening of all passengers and property transported by commercial passenger aircraft. At the more than 450 TSA-regulated airports in the United States, all passengers, their accessible property, and their checked baggage are screened prior to boarding an aircraft or entering the sterile area of an airport pursuant to statutory and regulatory requirements and TSA-established standard operating procedures. BDA, and more specifically, the SPOT program, constitutes one of multiple layers of security implemented within TSA- regulated airports. According to TSA’s strategic plan and other program guidance for the BDA program released in December 2012, the goal of the agency’s behavior detection activities, including the SPOT program, is to identify high-risk passengers based on behavioral indicators that indicate “mal-intent.” For example, the strategic plan notes that in concert with other security measures, behavior detection activities “must be dedicated to finding individuals with the intent to do harm, as well as individuals with connections to terrorist networks that may be involved in criminal activity supporting terrorism.”

TSA developed its primary behavior detection activity, the SPOT program, in 2003 as an added layer of security to identify potentially high- risk passengers through behavior observation and analysis techniques. The SPOT program’s standard operating procedures state that BDOs are to observe and visually assess passengers, primarily at passenger screening checkpoints, and identify those who display clusters of behaviors indicative of stress, fear, or deception. The SPOT procedures list a point system BDOs are to use to identify potentially high-risk passengers on the basis of behavioral and appearance indicators, as compared with baseline conditions where SPOT is being conducted. A team of two BDOs is to observe passengers as they proceed through the screening process. This process is depicted in figure 1.

According to TSA, it takes a BDO less than 30 seconds to meaningfully observe an average passenger. If one or both BDOs observe that a passenger reaches a predetermined point threshold, the BDOs are to direct the passenger and any traveling companions to the second step of the SPOT process—SPOT referral screening. During SPOT referral screening, BDOs are to engage the passenger in casual conversation—a voluntary informal interview—in the checkpoint area or a predetermined operational area in an attempt to determine the reason for the passenger’s behaviors and either confirm or dispel the observed behaviors. SPOT referral screening also involves a physical search of the passenger and his or her belongings. According to TSA, an average SPOT referral takes 13 minutes to complete. If the BDOs concur that a passenger’s behavior escalates further during the referral screening or if other events occur, such as the discovery of fraudulent identification documents or suspected serious prohibited or illegal items, they are to call a LEO to conduct additional screening—known as a LEO referral— who then may allow the passenger to proceed on the flight, or may question, detain, or arrest the passenger. The federal security director or designee, regardless of whether a LEO responds, is responsible for reviewing the circumstances surrounding a LEO referral and making the determination about whether the passenger can proceed into the sterile area of the airport.

Overview of SPOT Program Funding

The costs of the SPOT program are not broken out as a single line item in the budget. Rather, SPOT program costs are funded through three separate program, project, activity (PPA)-level accounts: (1) BDO payroll costs are funded through the Screener Personnel Compensation and Benefits (PC&B) PPA, (2) the operating expenses of the BDOs and the program are funded through the Screener Training and Other PPA, and (3) the program management payroll costs are funded through the Airport Management and Support PPA. From fiscal year 2007—when the SPOT program began deployment nationwide—through fiscal year 2012, about $900 million has been expended on the program, as shown in figure 2.

The majority of the funding (approximately 79 percent) for the SPOT program covers workforce costs and is provided under the Screener Personnel Compensation and Benefits PPA. This PPA—for which TSA requested about $3 billion for fiscal year 2014—funds, among other TSA screening activities, BDOs and TSO screening of passengers and their property. The workforce of about 3,000 BDOs is broken into four separate pay bands. The F Band, or Master BDO, and the G Band, or Expert BDO, constitute the primary BDO workforce that screens passengers using behavior detection. The H and I bands are supervisory-level BDOs, responsible for overseeing SPOT operations at the airport level. According to TSA figures, in fiscal year 2012, the average salaries and benefits of an F Band BDO full-time equivalent (FTE) was $66,310; a G Band BDO was $78,162, and the average FTE cost of H and I Band BDO supervisors was $97,392.

Overview of the Validation Study

In 2007, S&T began research to assess the validity of the SPOT program. The contracted study, issued in April 2011, was to examine the extent to which using the SPOT referral report and its indicators, as established in SPOT procedures, led to correct screening decisions at security checkpoints. Two primary studies were designed within the broader validation study: 1. an indicator study: an analysis of the behavioral and appearance indicators recorded in SPOT referral reports over an approximate 5- year period and their relationships to outcomes indicating a possible threat or high-risk passenger, and 2. a comparison study: an analysis over an 11-month period at 43 airports that compared arrests and other outcomes for passengers selected using the SPOT referral report with passengers selected and screened at random, as shown in table 1.

The validation study found, among other things, that some SPOT indicators appeared to be predictors of outcomes indicating a possible threat or high-risk passenger, and that SPOT procedures were more effective than a selection of passengers through a random protocol in identifying outcomes that represent high-risk passengers.

While the validation study was being finalized, DHS convened a TAC composed of 12 researchers and law enforcement professionals who met for 1 day in February 2011 to evaluate the methodology of the SPOT validation study. According to the TAC report, TAC members received briefings from the contractor that described the study plans and results, but because of TSA’s security concerns, TAC members did not receive detailed information about the contents of the SPOT referral report, the individual indicators used in the SPOT program, the validation study data, or the final report containing complete details of the SPOT validation study results. The TAC report noted that several TAC members felt that these restrictions hampered their ability to perform their assigned tasks. According to TSA, TAC members were charged with evaluating the methodology of the study, not the contents of the SPOT referral report. Consequently, TSA officials determined that access to this information was not necessary for the TAC to fulfill its responsibilities. S&T also contracted with another contractor, a human resources research organization, to both participate as TAC members and write a report summarizing the TAC meeting and subsequent discussions among the TAC members. In June 2011, S&T issued the TAC report, which contained TAC recommendations on future work as well as an appendix on TAC dissenting opinions. The findings of the TAC report are discussed later in this report.

Available Evidence Does Not Support Whether Behavioral Indicators Can Be Used to Identify Aviation Security Threats

Meta-analyses and other published research studies we reviewed do not support whether nonverbal behavioral indicators can be used to reliably identify deception. While the April 2011 SPOT validation study was a useful initial step and, in part, addressed issues raised in our May 2010 report, it does not demonstrate the effectiveness of the SPOT indicators because of methodological weaknesses in the study. Further, TSA program officials and BDOs we interviewed agree that some of the behavioral indicators used to identify passengers for additional screening are subjective. TSA has plans to study whether behavioral indicators can be reliably interpreted, and variation in referral rates raises questions about the use of the indicators by BDOs.

Published Research Does Not Support Whether the Use of Behavioral Indicators by Human Observers Can Identify Deception

Studies of Nonverbal Indicators to Identify Deception

Peer-reviewed, published research does not support whether the use of nonverbal behavioral indicators by human observers can accurately identify deception. Our review of meta-analyses and other studies related to detecting deception conducted over the past 60 years, and interviews with experts in the field, question the use of behavior observation techniques, that is, human observation unaided by technology, as a means for reliably detecting deception. The meta- analyses, or reviews that synthesize the findings of other studies, we reviewed collectively included research from more than 400 separate studies on detecting deception, and found that the ability of human observers to accurately identify deceptive behavior based on behavioral cues or indicators is the same as or slightly better than chance (54 percent). A 2011 meta-analysis showed weak correlations between most behavioral cues studied and deception. For example, the meta- analysis showed weak correlations for behavioral cues that have been studied the most, such as fidgeting, postural shifts, and lack of eye contact. A 2006 meta-analysis reviewed, in part, the ability of both individuals trained in fields such as law enforcement, as well as those untrained, and found no difference in their ability to detect deception. Additionally, a 2007 meta-analysis on nonverbal indicators of deception states that while there is a general belief that certain nonverbal behaviors are strongly associated with deception—such as an increase in hand, foot, and leg movements—these behaviors are diametrically opposed to observed indicators of deception in experimental studies, which indicate that movements actually decrease when people are lying.

As part of our analysis, we also reviewed scientific research focused on detecting passenger deception in an airport environment. We identified a 2010 study–based on a small sample size of passengers–that reviewed a similar behavior observation program in another country. The first phase of the study found that passengers who were selected based on behaviors were more likely to be referred to airport security officials for further questioning as compared to passengers who had been selected according to a random selection protocol. However, because the physical attributes of the passengers were found to be significantly different between those passengers selected based on behaviors versus those randomly selected, the researchers undertook a second phase of the study to control for those differences. The second phase revealed no differences in initial follow up rate between passengers selected based on behaviors and those matched for physical attributes. That is, when the control group was matched by physical attribute to passengers selected on the basis of behaviors, the follow up rate was the same. The researchers concluded that the higher number of passengers selected based on behaviors and referred for further questioning during the first phase of the study “was more the result of profiling” than the use of behavior observation techniques.

As mentioned earlier in this report, the goal of the BDA program is to identify high-risk passengers based on behavioral indicators that may indicate mal-intent. However, other studies we reviewed found that there is little available research regarding the use of behavioral indicators to determine mal-intent, or deception related to an individual’s intentions. For example, a 2013 RAND report noted that controversy exists regarding the use of human observation techniques that use behavioral indicators to identify individuals with intent to deceive security officials. In particular, the study noted that while behavioral science has identified nonverbal behaviors associated with emotional and psychological states, these indicators are subject to certain factors, such as individual variability, that limit their potential utility in detecting pre-incident indicators of attack.

The RAND report also found that the techniques for measuring the potential of using behavioral indicators to detect attacks are poorly developed and worthy of further study.

Moreover, a 2008 study performed for the Department of Defense by the JASON Program Office reviewed behavior detection programs, including the methods used by the SPOT program, and found that no compelling evidence exists to support remote observation of physiological signals that may indicate fear or nervousness in an operational scenario by human observers, and no scientific evidence exists to support the use of these signals in detecting or inferring future behavior or intent. In particular, the report stated that success in identifying deception and intent in other studies is post hoc and such studies incorrectly equate success in identifying terrorists with the identification of drug smugglers, warrant violators, or others. For example, when describing the techniques used by BDOs in the SPOT program, the report concluded that even if a correlation were found between abnormal behaviors and guilt as a result of some transgression, there is no clear indication that the guilt caused the abnormal behavior. The report also noted that the determination that the abnormal behavior was caused by guilt was made after the fact, rather than being based on established criteria beforehand.

Recent research on behavior detection has identified more promising results when behavioral indicators are used in combination with certain interview techniques and automated technologies, which are not used as part of the SPOT program. For example, several studies we reviewed that were published in 2012 and 2013 note that specific interviewing techniques, such as asking unanticipated questions, may assist in identifying deceptive individuals. Researchers began to develop automated technologies to detect deception, in part, because humans are limited in their ability to perceive, detect, and analyze all of the potentially useful information about an individual, some of which otherwise would not be noticed by the naked eye. For example, the 2013 RAND report noted that the link between facial microexpressions—involuntary expressions of emotion appearing for milliseconds despite best efforts to dampen or hide them—and deception can be evidenced by coding emotional expressions from a frame-by-frame analysis of video. However, the study concludes that the technique is not suitable for use by humans in real time at checkpoints or other screening areas because of the time lag and hours of labor required for such analysis. Automated technologies are being explored by federal agencies in conjunction with academic researchers to overcome these limitations, as well as human fatigue factors and potential bias in trying to detect deception. Although in the early stages of development, the study stated that automated technologies might be effective at fusing multiple indicators, such as body movement, vocal stress, and facial microexpression analysis.

Methodological Issues Limit the Usefulness of DHS’s April 2011 Indicator Validation Study

The usefulness of DHS’s April 2011 validation study is limited, in part because the data the study used to examine the extent to which the SPOT behavioral indicators led to correct screening decisions at security checkpoints were from the SPOT database that we had previously found in May 2010 to have several weaknesses, and thus were potentially unreliable. The SPOT indicator study analyzed data collected from 2006 to 2010 to determine the extent to which the indicators could identify high- risk passengers defined as passengers who (1) possessed fraudulent documents, (2) possessed serious prohibited or illegal items, (3) were arrested by a LEO, or (4) any combination of the first three measures. The validation study reported that 14 of the 41 SPOT behavioral indicators were positively and significantly related to one or more of the study outcomes. However, in May 2010, we assessed the reliability of the SPOT database against Standards for Internal Control in the Federal Government and concluded that the SPOT database lacked controls to help ensure the completeness and accuracy of the data, such as computerized edit checks to review the format, existence, and reasonableness of data. We found, among other things, that BDOs could not record all behaviors observed in the SPOT database because the database limited entry to eight behaviors, six signs of deception, and four types of serious prohibited items per passenger referred for additional screening. BDOs are trained to identify 94 signs of stress, fear, and deception, or other related indicators. As a result, we determined that, as of May 2010, the data were not reliable enough to conduct a statistical analysis of the association between the indicators and high-risk passenger outcomes. In May 2010, we recommended that TSA make changes to ensure the quality of SPOT referral data, and TSA subsequently made changes to the SPOT database. However, the validation study used data that were collected from 2006 through 2010, prior to TSA’s improvements to the SPOT database. Consequently, the data were not sufficiently reliable for use in conducting a statistical analysis of the association between the indicators and high-risk passenger outcomes.

In their report that reviewed the validation study, TAC members expressed some reservations about the methodology used in analyzing the SPOT indicators and suggested that the contractor responsible for completing the study consider not reporting on some of its results and moving the results to an appendix, rather than including them as a featured portion of the report. Further, the final validation study report findings were mixed, that is, they both supported and questioned the use of these indicators in the airport environment, and the report noted that the study was an “initial step” toward validating the program. However, because the study used unreliable data, its conclusions regarding the use of the SPOT behavioral indicators for passenger screening are questionable and do not support the conclusion that they can or cannot be used to identify threats to aviation security. Other aspects of the validation study are discussed later in this report.

Subjective Interpretation of Behavioral Indicators and Variation in Referral Rates Raise Questions about the Use of Indicators; TSA Plans to Study Indicators

BDO Interpretation of Some Behavioral Indicators Is Subjective; TSA Plans Study

BDA officials at headquarters and BDOs we interviewed in four airports said that some of the behavioral indicators are subjective, and TSA has not demonstrated that BDOs can consistently interpret behavioral indicators, though the agency has efforts under way to reduce subjectivity in the interpretation by BDOs. For example, BDA officials at headquarters stated that the definition of some behaviors in SPOT standard operating procedures is subjective. Further, 21 of 25 BDOs we interviewed said that certain behaviors can be interpreted differently by different BDOs. SPOT procedures state that the behaviors should deviate from the environmental baseline. As a result, BDOs’ application of the definition of the behavioral indicators may change over time, or in response to external factors.

Four of the 25 BDOs we spoke with said that newer BDOs might be more sensitive in applying the definition of certain behaviors. Our analysis of TSA’s SPOT referral data, discussed further below, shows that there is a statistically significant correlation between the length of time that an individual has been a BDO, and the number of SPOT referrals the individual makes per 160 hours worked, or about four 40-hour work weeks. This suggests that different levels of experience may be one reason why BDOs apply the behavioral indicators differently.

BDA officials agree that some of the SPOT indicators are subjective, and the agency is working to better define the behavioral indicators currently used by BDOs. In December 2012, TSA initiated a new contract to review the indicators in an effort to reduce the number of behavioral and appearance indicators used and to reduce subjectivity in the interpretation by BDOs. In June 2013, the contractor produced a document that summarizes information on the SPOT behavioral indicators from the validation study analysis, such as how frequently the indicator was observed, that it says will be used in the indicator review process. According to TSA’s November 2012 performance metrics plan, in 2014, the agency also intends to complete an inter-rater reliability study. This study could help TSA determine whether BDOs can reliably interpret the behavioral indicators, which is a critical component of validating the SPOT program’s results and ensuring that the program is implemented consistently.

Our analysis of SPOT referral data from fiscal years 2011 and 2012 indicates that SPOT and LEO referral rates vary significantly across BDOs at some airports, which raises questions about the use of behavioral indicators by BDOs. Specifically, we found that variation exists in the SPOT referral rates among 2,199 nonmanager BDOs and across the 49 airports in our review, after standardizing the referral data to take account of the differences in the amount of time each BDO spent observing passengers, as shown in figure 3.

The SPOT referral rates of BDOs ranged from 0 to 26 referrals per 160 hours worked during the 2-year period we reviewed. Similarly, LEO referral rates of BDOs ranged from 0 to 8 per 160 hours worked. Further, at least 153 of the 2,199 nonmanager BDOs were never identified as the primary BDO responsible for a referral. Of these, at least 76 were not associated with a referral during the 2-year period we reviewed.

To better understand the variation in referral rates, we analyzed whether certain variables affected SPOT referral rates and LEO referral rates, including the airport at which the referral occurred, and BDO characteristics, such as their annual performance scores, years of experience, as well as demographic information, including age and gender. The variables we identified as having a statistically significant relationship to the referral rates are shown in table 2.

We found that overall, the greatest amount of the variation in SPOT referral rates by BDOs was explained by the airport in which the referral occurred. That is, a BDO’s SPOT referral rate was associated with the airport at which he or she was conducting SPOT activities. However, separate analyses we conducted indicate that these differences across airports were not fully accounted for by another variable that is directly related to individual airports. That variable accounted for less than half of the variation in SPOT referral rates accounted for by airports. Combined, the remaining variables–including BDO performance score, age, years of BDO experience, years of TSA experience, race, and educational level– accounted for little of the variation in SPOT referral rates. In commenting on this issue, TSA officials noted that variation in referral rates across airports could be the result of differences in passenger composition, the airport’s market type, the responsiveness of LEOs to BDO referrals, and the number and type of airlines at the airports, among other things. However, because TSA could not provide additional supporting data on these variables with comparable time frames, we were not able to include these variables in our analysis. See appendix IV for a more detailed discussion of the findings from our multivariate analysis of referral rates.

According to TSA, having clearly defined and consistently implemented standard operating procedures for BDOs in the field at the 176 SPOT airports is key to the success of the program. In May 2010, we found that TSA established standardization teams designed to help ensure consistent implementation of the SPOT standard operating procedures. We followed up on TSA’s use of standardization teams and found that from 2012 to 2013, TSA made standardization team visits to 9 airports. In May 2012, officials changed their approach and data collection requirements and changed the name of the teams to program compliance assessment teams. From December 2012 through March 2013, TSA conducted pilot site visits to 3 airports to test and refine new compliance team protocols for data collection, which, among other things, involve more quantitative analysis of BDO performance. The pilot process was designed to help ensure that the program compliance assessment teams conduct standardized, on-site evaluations of BDOs’ compliance with the SPOT standard operating procedures in a way that is based on current policy and procedures. As of June 2013, TSA had visited and collected data at 6 additional airports and was refining data input and reporting processes. According to BDA officials, TSA deployed the new compliance teams nationally in August 2013 and anticipates visiting an additional 13 airports by the end of fiscal year 2013. However, the compliance teams are not generally designed to help ensure BDOs’ ability to consistently interpret the SPOT indicators, and the agency has not developed other mechanisms to measure inter-rater reliability. TSA does not have reasonable assurance that BDOs are reliably interpreting passengers’ behaviors within or among airports, in part because of the subjective interpretation of some SPOT behavioral indicators by BDOs and the limited scope of the compliance teams. This, coupled with the inconsistency in referral rates across different airports, raises questions about the use of behavioral indicators to identify potential threats to aviation.

TSA Has Limited Information to Evaluate SPOT Program Effectiveness but Plans to Collect Additional Performance Data

TSA has limited information to evaluate SPOT program effectiveness because the findings from the April 2011 validation comparison study are inconclusive because of methodological weaknesses in the study’s overall design and data collection. However, TSA plans to collect additional performance data to help it evaluate the effectiveness of its behavior detection activities.

Methodological Issues Affect the Results of DHS’s Study Comparing SPOT with Random Selection of Passengers

Design Limitations

DHS’s 2011 validation study compared the effectiveness of SPOT with a random selection of passengers and found that SPOT was between 4 and 52 times more likely to correctly identify a high-risk passenger than random selection, depending on which of the study’s outcome measures was used to define persons knowingly and intentionally trying to defeat the security process. However, BDOs used various methods to randomly select passengers during data collection periods of differing length at the study airports. Initially, the contractor proposed that TSA use random selection methods at a sample of 143 SPOT airports, based on factors such as the number of airport passengers. If properly implemented, the proposed sample would have helped ensure that the validation study findings could be generalized to all SPOT airports. However, according to the study and interviews with the contractor, TSA selected a nonprobability sample of 43 airports based on input from local TSA airport officials who decided to participate in the study. TSA allowed the managers of these airports to decide which checkpoints would use random procedures and when they would do so during airport operating hours. According to the validation study and a contractor official, the airports included in the study were not randomly selected because of the increased time and effort it would take to collect study data at the 143 airports proposed by the contractor. Therefore, the study’s results may provide insights about the implementation of the SPOT program at the 43 airports where the study was carried out, but they are not generalizable to all 176 SPOT airports.

Additionally, TSA collected the validation study data unevenly and experienced challenges in collecting an adequate sample size for the randomly selected passengers, facts that might have further affected the representativeness of the findings. According to established evaluation design practices, data collection should be sufficiently free of bias or other significant errors that could lead to inaccurate conclusions. Specifically, in December 2009, TSA initially began collecting data from 24 airports whose participation in the study was determined by the local TSA officials. More than 7 months later, TSA added another 18 airports to the study when it determined that enough data were not being collected on the randomly selected passengers at participating airports to reach the study’s required sample size. The addition of the airports coincided with a substantial increase in referrals for additional screening and an uneven collection of data, as shown in figure 4.

As a result of this uneven data collection, study data on 61 percent of randomly selected passengers were collected during the 3-month period from July through September 2010. By comparison, 33 percent of the data on passengers selected by the SPOT program were collected during the same time. Because commercial aviation activity and the demographics of the traveling public are not constant throughout the year, this uneven data collection may have conflated the effect of random versus SPOT selection methods with differences in the rates of high-risk passengers when TSA used either method.

In addition, the April 2011 validation study noted that BDOs were aware of whether the passengers they were screening were selected as a result of the random selection protocol or SPOT procedures, which had the potential to introduce bias in the assessment. According to established practices for evaluation design, when feasible, many scientific studies use “blind” designs, in which study participants do not know which procedures are being evaluated. This helps avoid potential bias due to the tendency of participants to behave or search for evidence in a manner that supports the effects they expect each procedure to have. In contrast, in the SPOT comparison study, BDOs knew whether each passenger they screened was selected through SPOT or random methods. This may have biased BDOs’ screening for high-risk passengers, because BDOs could have expected randomly selected passengers to be lower risk and thus made less effort to screen passengers. In interviews, the contractor and four of the eight members of the TAC we interviewed agreed that this may be a design weakness. One TAC member told us that the comparison study would have been more robust if the passengers had been randomly selected by people without any prior knowledge of SPOT indicators to decrease the possibility of bias. To reduce the possibility of bias in the study, another TAC member suggested that instead of using the same BDOs to select and screen passengers, some BDOs could have been responsible for selecting passengers and other BDOs for screening the passengers, regardless of whether they were selected randomly or by SPOT procedures. According to validation study training materials, BDOs were used to select both groups of passengers in an effort to maintain normal security coverage during the study. Another TAC member stated that controls were needed to ensure that BDOs gave the same level of scrutiny to randomly selected passengers as those referred because of their behaviors. The contractor officials reported that they were aware of the potential bias, and tried to mitigate its potential effects by training BDOs who participated in the validation study to screen passengers identically, regardless of how they were selected. However, the contractor stated that they could not fully control these selections because BDOs were expected to conduct their regular SPOT duties concurrently during the study’s data collection on random passenger screening. The validation study discussed several limitations that had the potential to introduce bias, but concluded that they did not affect the results of the study.

Our analysis of the validation study data regarding one of the primary high-risk outcome measures—LEO arrests—suggests that the screening process was different for passengers depending on whether they were selected using SPOT procedures or the random selection protocol. Therefore, the study’s finding that SPOT was much more likely to identify high-risk passengers who were ultimately arrested by a LEO may be considerably inflated. Specifically, a necessary condition influencing the rate of the arrest outcome measure—exposure to a LEO through a LEO referral—was not equal in the two groups. The difference between the groups occurred because randomly selected passengers were likely to begin the SPOT referral process with zero points or very few points, whereas passengers selected on the basis of SPOT began the process at the higher, established point threshold required for BDOs to make a SPOT referral. However, because the point threshold for a LEO referral was the same for both groups, the likelihood that passengers selected using SPOT would escalate to the next point threshold, resulting in a LEO referral and possible LEO arrest, was greater than for passengers selected randomly. Our analysis showed that because of the discrepancy in the points accrued prior to the start of the referral process, passengers who were selected on the basis of SPOT behavioral indicators were more likely to be referred to a LEO than randomly selected passengers. Our analysis indicates that the validation study design could have been improved by treating each group similarly, regardless of the passengers’ accumulated points. For example, as a possible approach, both groups could have been referred to LEOs only in the cases where BDOs discovered a serious prohibited or illegal item. Established study design practices state that identifying key factors known to influence desired evaluation outcomes will aid in forming treatment and comparison groups that are as similar as possible, thus strengthening the analyses’ conclusions.

Additionally, once referred to a LEO, passengers selected at random were arrested for different reasons than those selected on the basis of SPOT indicators, which suggests that the two groups of passengers were subjected to different types of screening. All randomly selected passengers who were identified as high risk, referred to a LEO, and ultimately arrested possessed fraudulent documents or serious prohibited or illegal items. In contrast, most of the passengers arrested after having been referred on the basis of SPOT behavior indicators were arrested for reasons other than fraudulent documents or serious prohibited or illegal items. These reasons for arrest included outstanding warrants by law enforcement agencies, public intoxication, suspected illegal entry into the United States, and disorderly conduct.

Such differences in the reasons for arrest suggest that referral screening methods may have varied according to the method of selection for screening, consistent with the concerns of the TAC members and the contractor. Thus, because randomly selected passengers were assigned points differently during screening and consequently referred to LEOs far less than those referred by SPOT, and because being referred to a LEO is a necessary condition for an arrest, the results related to the LEO arrest metric are questionable and cannot be relied upon to demonstrate SPOT program effectiveness.

To help ensure that all of the BDOs carried out the comparison study as intended, protocols for randomly selecting passengers were established that would help ensure that the methods would be the same across airports. The contractor emphasized that deviating from the prescribed protocol could increase the likelihood of introducing systematic differences across airports in the methods of random screening, which could bias the results. To ensure that airports and BDOs followed the study protocols, the contractor conducted monitoring visits at 17 of the 43, or 40 percent, of participating airports. The first monitoring visits occurred 6 months after data collection began, and 9 of the 17 airports were not visited until the last 2 months of the study, as shown in figure 5. Consequently, for 9 of these airports, the contractor could not have addressed the deviations from the protocols that were identified during the data-monitoring visits until the last weeks of data collection.

In the April 2011 report of all 17 monitoring visits that were conducted, the most crucial issue the contractor identified was that BDOs deviated from the random selection protocol in ways that did not meet the criteria for systematic random selection. For example, the contractor found that across airports, local TSA officials had independently decided to exclude certain types of passengers from the study because the airport officials felt it was unreasonable to subject these types of passengers to referral screening. At 1 airport visited less than 4 weeks before data collection ended, BDOs misunderstood the protocols and incorrectly excluded a certain type of passenger. As a result, certain groups of potentially lower-risk passengers were systematically excluded from the population eligible for random selection. In addition, the contractor found that some BDOs used their own methods to select passengers, rather than the random selection protocol that was specified. The contractor reported that if left uncorrected, this deviation from the protocols could increase the likelihood of introducing systematic bias into the study. For example, at one airport visited less than 6 weeks before data collection ended, BDOs selected passengers by attempting to generate numbers they thought were random by calling out numbers spontaneously, such as “seven,” and using the numbers to select the seventh passenger, instead of following the random selection protocol. At another airport visited less than 6 weeks before data collection ended, contrary to random selection protocols, BDOs, rather than the data collection coordinator, selected passengers to undergo referral screening. Although deviations from the protocol may not have produced a biased sample, any deviation from the selection protocol suggests that BDOs’ judgment may have affected the random selection and screening processes in the comparison study.

In addition to the limitations cited above, the April 2011 validation study noted other limitations such as the limited data useful for measuring high- risk passenger outcomes, the lack of information on the specific location within the airport where each SPOT indicator was first observed, and difficulties in differentiating whether passengers were referred because of observed behaviors related to elevated indicators of stress, fear, and deception, or for other reasons. The validation study concluded that further research to fully validate and evaluate the SPOT program was warranted. Similarly, the TAC report cited TAC members’ concerns that the validation study results “could be easily misinterpreted given the limited scope of the study and the caveats to the data,” and that the “results should be presented as a first step in a broader evaluation process.” Thus, limitations in the study’s design and in monitoring how it was implemented at airports could have affected the accuracy of the study’s conclusions, and limited their usefulness in determining the effectiveness of the SPOT program. As a result, the incidence of high-risk passengers in the normal passenger population remains unknown, and the incidence of high-risk passengers identified by random selection cannot be compared with the incidence of those identified using SPOT methods.

TSA Plans to Collect and Analyze Needed Performance Data

TSA plans to collect and analyze additional performance data needed to assess the effectiveness of its behavior detection activities. In response to recommendations we made in May 2010 to conduct a cost-benefit analysis and a risk assessment, TSA completed two analyses of the BDA program in December 2012, but needs to complete additional analysis to fully address our recommendations. Specifically, TSA completed a return-on-investment analysis and a risk-based allocation analysis, both of which were designed in part to inform the future direction of the agency’s behavior detection activities, including the SPOT program. The return-on-investment analysis assessed the additional value that BDOs add to TSA’s checkpoint screening system, and concluded that BDOs provide an integral value to the checkpoint screening process. However, the report did not fully support its assumptions related to the threat frequency or the direct and indirect consequence of a successful attack, as is recommended by best practices. For example, TSA officials told us that the threat and consequence assumptions in the analysis were designed to be consistent with the 2013 Transportation Security System Risk Assessment (TSSRA), but the analysis did not explain why a catastrophic event was the only relevant threat scenario considered when determining consequence. Additionally, the analysis relied on assumptions regarding the effectiveness of BDOs and other countermeasures that were based on questionable information. For example, the analysis relied on results reported in the April 2011 validation study—which, as discussed earlier, had several methodological limitations—as evidence of the effectiveness of BDOs. Further, a May 2013 DHS OIG report found that TSA could not accurately assess the effectiveness or evaluate the progress of the SPOT program because it had not developed a system of performance measures at the time of the OIG review. In response, TSA provided the OIG with a draft version of its performance metrics plan. This plan has since been finalized and is discussed further below.

TSA’s risk-based allocation analysis found that an additional 584 BDO FTEs should be allocated to smaller airports in an effort to cover existing gaps in physical screening coverage and performance, an action that, if implemented, would result in an annual budgetary increase of approximately $42 million. One of the primary assumptions in the risk- based allocation analysis is related to the effectiveness of BDOs. For example, this analysis suggests that BDOs may be effective in identifying threats to aviation security where gaps exist in physical screening coverage and performance, including the use of walk-through metal detectors and advanced imaging technology machines. However, TSA has not evaluated the effectiveness of BDOs in comparison with these other screening methods.

In response to an additional recommendation in our May 2010 report to develop a plan for outcome-based performance measures, TSA completed a performance metrics plan in November 2012, which details the performance measures required for TSA to determine whether the agency’s behavior detection activities are effective, and identifies the gaps that exist in its current data collection efforts. The plan defined an ideal set of 40 metrics within three major categories that BDA needs to collect to be able to understand and measure the performance of its behavior detection activities. TSA then identified the gaps in its current data collection efforts, such as, under the human factors subcategory, data on BDO fatigue levels and what staffing changes would need to be made to reduce the negative impact on BDO performance resulting from fatigue, as shown in figure 6.

As of June 2013, TSA had collected some information for 18 of 40 metrics the plan identified. Once collected, the data identified by the plan may help support the completion of a more substantive return-on-investment analysis and risk-based allocation analysis, but according to TSA’s November 2012 plan, TSA is currently collecting little to none of the data required to assess the performance and security effectiveness of BDA or the SPOT program. For example, TSA does not currently collect data on the percentage of time a BDO is present at a checkpoint or other areas in the airport while it is open. Without this information, the assumptions contained in TSA’s risk-based allocation analysis cannot be validated. This analysis identified the existing BDO coverage level at the airports where SPOT was deployed in 2011, and based its recommendations for an additional 584 BDOs on this coverage level.

In May 2013, TSA began to implement a new data collection system, BDO Efficiency and Accountability Metrics (BEAM), designed to track and analyze BDO daily operational data, including BDO locations and time spent performing different activities. According to BDA officials, this data will allow the agency to gain insight on how BDOs are utilized, and improve analysis of the SPOT program. The performance metrics plan may also provide other useful information in support of some of the other assumptions in TSA’s risk-based allocation analysis and return-on- investment analysis. For example, both analyses assumed that a BDO can meaningfully assess 450 passengers per hour, and that fatigue would degrade this rate over the course of a day. However, according to the performance metrics plan, TSA does not currently collect any of the information required to assess the number of passengers meaningfully assessed by BDOs, BDOs’ level of fatigue, or the impact that fatigue has on their performance. To address these and other deficiencies, the performance metrics plan identifies 22 initiatives that are under way or planned as of November 2012, including efforts discussed earlier in this report, such as the indicator study and efforts to improve the SPOT compliance teams, among others. For additional information about the metrics that will result from these initiatives, see appendix V.

These data could help TSA assess the performance and security effectiveness of BDA and the SPOT program, and find ways to become more efficient with fewer resources in order to meet the federal government’s long-term fiscal challenges, as recommended by federal government efficiency initiatives. In lieu of these data, TSA uses arrest and LEO referral statistics to help track the program’s activities. Of the approximately 61,000 referrals made over the 2-year period at the 49 airports we analyzed, approximately 8,700 (14 percent) resulted in a referral to a LEO. Of these LEO referrals, 365 (4 percent) resulted in an arrest. The proportion of LEO referrals that resulted in an arrest (arrest ratio) could be an indicator of the potential relationship between the SPOT behavioral indicators and an arrest. As shown in figure 7, 99.4 percent of the passengers that were selected for referral screening—that is further questioning and inspection by a BDO—were not arrested. The percentage of passengers referred to LEOs that were arrested was about 4 percent; the other 96 percent of passengers referred to LEOs were not arrested. The SPOT database identifies 6 reasons for arrest, including (1) fraudulent documents, (2) illegal alien, (3) other, (4) outstanding warrants, (5) suspected drugs, and (6) undeclared currency.

In February 2013, BDA officials said between 50 and 60 SPOT referrals were forwarded by the Federal Air Marshal Service to other law enforcement agencies for further investigation to identify potential ties to terrorism. For example, TSA provided documentation of three suspicious incident reports from 2011 of passengers who were referred by BDOs to LEOs based on behavioral indicators, and who were later found to be in possession of large sums of U.S. currency. According to a FAMS report on these incident reports, the identification of large amounts of currency leaving the United States could be the first step in the disruption of funding for terrorist organizations or other form of criminal enterprise that may or may not be related to terrorism. TSA officials said it is difficult to identify the terrorism-related nexus in these referrals because they are rarely, if ever, informed on the outcomes of the investigations conducted by other law enforcement agencies, and thus have no way of knowing if these SPOT referrals were ultimately connected to terrorism- related activities or investigations.

Standards for Internal Control in the Federal Government calls for agencies to report on the performance and effectiveness of their programs. However, according to the performance metrics plan, TSA will require at least an additional 3 years and additional resources before it can begin to report on the performance and security effectiveness of BDA or the SPOT program. Given the scope of the proposed activities and some of the challenges that TSA has faced in its earlier efforts to assess the SPOT program at the national level, to complete the activities in the time frames outlined in the plan would be difficult. In particular, the plan notes it is unrealistic that TSA will be able to evaluate the BDO security effectiveness contribution at each airport within the 3-year timeframe. According to best practices for program management of acquisitions, technologies should be demonstrated to work reliably in their intended environment prior to program deployment. Further, according to OMB guidance accompanying the fiscal year 2014 budget, it is incumbent upon agencies to use resources on programs that have been rigorously evaluated and determined to be effective, and to fix or eliminate those programs that have not demonstrated results. TSA has taken a positive step toward determining the effectiveness of BDA’s behavior detection activities by developing the performance metrics plan, as we recommended in May 2010. However, 10 years after the development of the SPOT program, TSA cannot demonstrate the effectiveness of its behavior detection activities. Until TSA can provide scientifically validated evidence demonstrating that behavioral indicators can be used to identify passengers who may pose a threat to aviation security, the agency risks funding activities that have not been determined to be effective.

Conclusions

TSA has taken several positive steps to validate the scientific basis and strengthen program management of BDA and the SPOT program, which has been in place for over 6 years at a total cost of approximately $900 million since 2007. Nevertheless, TSA has not demonstrated that BDOs can consistently interpret the SPOT behavioral indicators, a fact that may contribute to varying passenger referral rates for additional screening. The subjectivity of the SPOT behavioral indicators and variation in BDO referral rates raise questions about the continued use of behavior indicators for detecting passengers who might pose a risk to aviation security. Furthermore, decades of peer-reviewed, published research on the complexities associated with detecting deception through human observation also draw into question the scientific underpinnings of TSA’s behavior detection activities. While DHS commissioned a 2011 study to help demonstrate the validity of its approach, the study’s findings cannot be used to demonstrate the effectiveness of SPOT because of methodological limitations in the study’s design and data collection.

While TSA has several efforts under way to assess the behavioral indicators and expand its collection of data to develop performance metrics for its behavioral detection activities, these efforts are not expected to be completed for several years, and TSA has indicated that additional resources are needed to complete them. Consequently, after 10 years of implementing and testing the SPOT program, TSA cannot demonstrate that the agency’s behavior detection activities can reliably and effectively identify high-risk passengers who may pose a threat to the U.S. aviation system.

Matter for Congressional Consideration

To help ensure that security-related funding is directed to programs that have demonstrated their effectiveness, Congress should consider the findings in this report regarding the absence of scientifically validated evidence for using behavioral indicators to identify aviation security threats when assessing the potential benefits of behavior detection activities relative to their cost when making future funding decisions related to aviation security.

Recommendation for Executive Action

To help ensure that security-related funding is directed to programs that have demonstrated their effectiveness, we recommend that the Secretary of Homeland Security direct the TSA Administrator to limit future funding support for the agency’s behavior detection activities until TSA can provide scientifically validated evidence that demonstrates that behavioral indicators can be used to identify passengers who may pose a threat to aviation security.

Agency and Third- Party Comments and Our Evaluation

We provided a draft of this report to DHS and the Department of Justice (DOJ) for review and comment. We also provided excerpts of this report to subject matter experts for their review to ensure that the information in the report was current, correct, and factual. DOJ did not have any comments, and we incorporated technical comments from subject matter experts as appropriate. DHS provided written comments, which are printed in full in appendix VI, and technical comments, which we incorporated as appropriate.

DHS did not concur with the recommendation to the Secretary of Homeland Security that directed the TSA Administrator to limit future funding support for the agency’s behavior detection activities until TSA can provide scientifically validated evidence that demonstrates that behavioral indicators can be used to identify passengers who may pose a threat to aviation security. Citing concerns with the findings and conclusions, DHS identified two main areas where it disagreed with information presented in the report: (1) the findings related to the SPOT validation study and (2) the findings related to the research literature. Further, DHS provided information on its investigation of profiling allegations. We disagree with the statements DHS made in its letter, as discussed in more detail below.

With regard to the findings related to the SPOT validation study, DHS stated in its letter that we used different statistical techniques when we replicated the analysis of SPOT indicators as presented in the DHS April 2011 validation study, a course of action that introduced error into our analysis and resulted in “misleading” conclusions. We disagree with this statement. As described in the report, we obtained the validation study dataset from the DHS contractor and replicated the analyses using the same techniques that the contractor used to conduct its analyses of SPOT indicators. As an extra step, in addition to replicating the approach (split-samples) used by the contractors, as described in appendixes II and III of this report, we extended those analyses using the full sample of referral data to increase our ability to detect significant associations. In both the replication of the study analyses and the extended analyses we conducted, we found essentially the same result in one aspect as the validation study—that some SPOT behavioral indicators were positively and significantly related to one or more of the outcome measures. Specifically, the validation study reported that 14 of the 41 SPOT behavioral indicators were positively and significantly related, and we found that 18 of the 41 behavioral indicators were positively and significantly related. However, the findings regarding negatively and significantly related SPOT indicators were not consistent between the analyses we conducted and the validation study. Specifically, we found that 20 of the 41 behavioral indicators were negatively and significantly related to one or more of the study outcomes (see app. II). That is, we identified 20 SPOT behavioral indicators that were more commonly associated with passengers who were not identified as high-risk passengers than with passengers who were identified as high-risk passengers. In other words, some of the SPOT indicators that behavior detection officers are trained to detect are associated with passengers who were defined by DHS as low risk. Our results were not consistent with the validation study, because the study did not report any indicators that were negatively and significantly correlated with one or more of the outcome measures. Further, because of limitations with the SPOT referral data that we reported in May 2010 and again in this report, the data the validation study used to examine behavioral indicators were not sufficiently reliable for use in conducting a statistical analysis of the association between the indicators and high-risk passenger outcomes. We did use these data in order to replicate the validation study findings.

Further, DHS stated in its letter that the TAC agreed with the study’s conclusion that SPOT was substantially better at identifying high-risk passengers than a random screening protocol. However, we disagree with this statement. While the TAC report stated that TAC members had few methodological concerns with the way the contractor carried out its research, the members did not receive detailed information on the study, including the validation study data and the final report containing the SPOT validation study results. Specifically, as discussed in our report and cited in the TAC report, multiple TAC members had concerns about some of the conclusions in the validation study and suggested that the contractor responsible for completing the study consider not reporting on some of its results and moving the results to an appendix, rather than including them as a featured portion of the report.

Moreover, since the TAC did not receive detailed information about the contents of the SPOT referral report, the individual indicators used in the SPOT program, the validation study data, or the final report containing complete details of the SPOT validation study results, the TAC did not have access to all of the information that we used in our analysis. As discussed in our report, the TAC report noted that several TAC members felt that this lack of information hampered their ability to perform their assigned tasks. Thus, we continue to believe that our conclusion related to the validation study results is valid, and contrary to DHS’s statement, we do not believe that the study provides useful data in understanding behavior detection.

With regard to the findings related to the research literature, DHS stated in its letter that we did not consider all the research that was available and that S&T had conducted research—while not published in academic circles for peer review because of various security concerns—that supported the use of behavior detection. DHS also stated that research cited in the report “lacked ecological and external validity,” because it did not relate to the use of behavior detection in an airport security environment. We disagree. Specifically, as described in the report, we reviewed several documents on behavior detection research that S&T and TSA officials provided to us, including an unclassified and a classified literature review that S&T had commissioned. Further, after meetings in June and July 2013, S&T officials provided additional studies, which we reviewed and included in the report as applicable. We also included research in the report on the use of behavioral indicators that correspond closely to indicators identified in SPOT procedures as indicative of stress, fear, or deception. These studies, many of which were included in the meta-analyses we reviewed, were conducted in a variety of settings— including high-stakes situations where the consequences are great, such as a police interview with an accused murderer—and with different types of individuals—including law enforcement personnel. The meta-analyses we reviewed—which collectively included research from over 400 separate studies related to detecting deception conducted over the past 60 years—found that the ability of human observers to accurately identify deceptive behavior based on behavioral cues or indicators is the same as or slightly better than chance (54 percent).

Further, in its letter, DHS cited a 2013 RAND report, which concluded that there is current value and unrealized potential for using behavioral indicators as part of a system to detect attacks. We acknowledge that behavior detection holds promise for use in certain circumstances and in conjunction with certain other technologies. However, the RAND report DHS cited in its letter refers to behavioral indicators that are defined and used significantly more broadly than those in the SPOT program. The indicators reviewed in the RAND report are neither used in the SPOT program, nor could be used in real time in an airport environment. Further, the RAND report findings cannot be used to support TSA’s use of behavior detection activities because the study stated that it could not make a determination of SPOT’s effectiveness because information on the program was not in the public domain.

DHS also stated in its letter that it has several efforts under way to improve its behavior detection program and the methodologies used to evaluate it, including the optimization of its behavior detection procedures and plans to begin testing by the third quarter of fiscal year 2014 using robust test and evaluation methods similar to the operational testing conducted in support of technology acquisitions as part of its 3-year performance metrics plan. We are encouraged by TSA’s plans in this area. However, TSA did not provide supporting documentation accompanying these plans describing how it will incorporate robust data collection and authentication protocols, as discussed in DHS’s letter. Such documentation is to be completed prior to beginning any operational testing. These documents might include a test and evaluation master plan that would describe, among other things, the tests that needed to be conducted to determine system technical performance, operational effectiveness or suitability, and any limitations.

Additionally, in its letter, DHS stated that the omission of research related to verbal indicators of deception was misleading because a large part of BDOs’ work is interacting with passengers and assessing whether passengers’ statements match their behaviors, or if the passengers’ trip stories are in agreement with their travel documents and accessible property. While BDOs’ interactions with passengers may elicit useful information, SPOT procedures indicate that casual conversation— voluntary informal interviews conducted by BDOs with passengers referred for additional screening—is conducted after the passengers have been selected for a SPOT referral, not as a basis for selecting the passengers for referral. Further, since these interviews are voluntary, passengers are under no obligation to respond to the BDOs questions, and thus information on passengers may not be systematically collected. As noted in our report, promising research on behavioral indicators cited in the RAND report and other literature is focused on using indicators in combination with automated technologies and certain interview techniques, such as asking unanticipated questions. However, when interviewing referred passengers for additional screening, BDOs do not currently have access to the automated technologies discussed in the RAND report.

Further, DHS stated that the goal of the SPOT program is to identify individuals exhibiting behavior indicative of simple emotions such as fear or stress and reroute them to a higher level of screening, and does not attempt to specifically identify persons engaging in lying or terrorist acts. However, DHS also stated in its response that “SPOT uses a broader array of indicators, including stress and fear detection as they relate to high-stakes situations where the consequences are great, for example, suicide attack missions.” As noted in the report, TSA’s program and budget documents associated with behavior detection activities identify that the purpose of these activities is to identify high-risk passengers based on behavioral indicators that indicate mal-intent. For example, the strategic plan notes that in concert with other security measures, behavior detection activities “must be dedicated to finding individuals with the intent to do harm, as well as individuals with connections to terrorist networks that may be involved in criminal activity supporting terrorism.” The conclusions, which were confirmed in discussions with subject matter experts and an independent review of studies, indicate that scientifically validated evidence does not support whether the use of behavioral indicators by unaided human observers can be used to identify passengers who may pose a threat to aviation security. DHS also cited the National Research Council’s 2008 report to support its use of SPOT. The National Research Council report, which we reviewed as part of our 2010 review of the SPOT program, noted that behavior and appearance monitoring might be able to play a useful role in counterterrorism efforts but also stated that a scientific consensus does not exist regarding whether any behavioral surveillance or physiological monitoring techniques are ready for use in the counterterrorist context, given the present state of the science. According to the National Research Council report, an information-based program, such as a behavior detection program, should first determine if a scientific foundation exists and use scientifically valid criteria to evaluate its effectiveness before going forward. The report also stated that programs should have a sound experimental basis, and documentation on the program’s effectiveness should be reviewed by an independent entity capable of evaluating the supporting scientific evidence.

With regard to information provided related to profiling, DHS stated that DHS’s OIG completed an investigation at the request of TSA into allegations that surfaced at Boston Logan Airport and concluded that these allegations could not be substantiated. However, while the OIG’s July 2013 report of investigation on behavior detection officers in Boston concluded that “there was no indication that BDOs racially profiled passengers in order to meet production quotas,” the OIG’s report also stated that there was evidence of “appearance profiling.”

In stating its nonconcurrence with the recommendation to limit future funding in support of its behavior detection activities, DHS stated that TSA’s overall security program is composed of interrelated parts, and to disrupt one piece of the multilayered approach may have an adverse impact on other pieces. Further, DHS stated that the behavior detection program should continue to be funded at current levels to allow BDOs to screen passengers while the optimization process proceeds. We disagree. As noted in the report, TSA has not developed the performance measures that would allow it to assess the effectiveness of its behavior detection activities compared with other screening methods, such as physical screening. As a result, the impact of behavior detection activities on TSA’s overall security program is unknown. Further, not all screening methods are present at every airport, and TSA has modified the screening procedures and equipment used at airports over time. These modifications have included the discontinuance of screening equipment that was determined to be unneeded or ineffective.

Therefore, we continue to believe that providing scientifically validated evidence that demonstrates that behavioral indicators can be used to identify passengers who may pose a threat to aviation security is critical to the implementation of TSA’s behavior detection activities. Further, OMB guidance highlights the importance of using resources on programs that have been rigorously evaluated and determined to be effective, and best practices for program management of acquisitions state that technologies should be demonstrated to work reliably in their intended environment prior to program deployment. Consequently, we have added a matter for congressional consideration to this report to help ensure that TSA provides information, including scientifically validated evidence, which supports the continued use of its behavior detection activities in identifying threats to aviation security.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 5 days from the report date. We are sending copies of this report to the Secretary of Homeland Security; the TSA Administrator; the United States’ Attorney General; and interested congressional committees as appropriate. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4379 or lords@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are acknowledged in appendix VII.

Appendix I: Information on Recent Allegations of Passenger Profiling and TSA’s Actions to Address Such Allegations

According to the Screening of Passengers by Observation Techniques (SPOT) program’s standard operating procedures, behavior detection officers (BDO) must apply the SPOT behavioral indicators to passengers without regard to race, color, religion, national origin, ethnicity, sexual orientation, or disability.

Since 2010, the Transportation Security Administration (TSA) and the Department of Homeland Security’s (DHS) Office of Inspector General (OIG) have examined allegations of the use of profiling related to the race, ethnicity, or nationality of passengers by behavior detection officers (BDO) at three airports—Newark Liberty International Airport (Newark), Honolulu International Airport (Honolulu), and Boston Logan International Airport (Boston)—and TSA has taken action to address these allegations. Specifically, in January 2010, TSA concluded an internal investigation at Newark of allegations that BDOs used specific criteria related to the race, ethnicity, or nationality of passengers in order to select and search those passengers more extensively than would have occurred without the use of these criteria. The investigation was conducted by a team of two BDO managers from Boston to determine whether two BDO managers at Newark had established quotas for SPOT referrals to evaluate the performance of their subordinate BDOs. The investigation also sought to determine whether these managers at Newark encouraged profiling of passengers in order to meet quotas that they had established. The investigating team concluded that no evidence existed to support the allegation of a quota system, but noted widespread BDO perception that higher referral rates led to promotion, and that the “overwhelming majority of BDOs” expressed concern that the BDO managers’ “focus was solely on increasing the number of referrals and LEO calls.” The investigating team said the information collected regarding the allegation of profiling resulted in a reasonable conclusion that that such activity was both directed and affected on a limited basis at Newark, based on one manager’s inappropriate direction to BDOs regarding profiling of passengers, racial comments, and the misuse of information intended for situational awareness purposes only. According to TSA officials, disciplinary action taken against this manager resulted in the manager’s firing.

Additionally, in 2011, TSA’s Office of Inspection (OOI) conducted an investigation of racial profiling allegations against BDOs at Honolulu. The investigation consisted of a review of Equal Employment Opportunity (EEO) complaints, and OOI did not find evidence to support the profiling allegations in the SPOT program.

In July 2012, OOI conducted a compliance inspection at Boston, during which allegations of profiling by BDOs surfaced. Specifically, during interviews with inspectors, allegations surfaced that BDOs were profiling passengers for the purpose of raising the number of law enforcement referrals. These accusations included written complaints from BDOs who claimed other BDOs were selecting passengers for referral screening based on their ethnic or racial appearance, rather than on the basis of the SPOT behavioral indicators and were reported in a September 2012 OOI memorandum. These allegations were referred to the OIG, and in August 2012, the OIG opened an investigation into these profiling allegations in Boston. According to OIG officials, its investigation was completed and its final report was provided to TSA in August 2013.

In August 2012, the Secretary of Homeland Security issued a memorandum directing TSA to take a number of actions in response to allegations of racial profiling by BDOs. These actions include (1) a revision of the SPOT standard operating procedures to, among other things, clarify that passengers who are unwilling or uncomfortable with participating in an interactive discussion and responding to questions will not be pressured by BDOs to do so; (2) refresher training for all BDOs that reinforces antidiscrimination requirements; and (3) TSA communication with BDO supervisors that performance appraisals should not depend on achieving either a high number of referrals or on the arrest rate coming from those referrals, but rather from demonstrated vigilance and skill in applying the SPOT procedures. As of June 2013, TSA, together with the DHS Acting Officer for Civil Rights and Civil Liberties and Counsel to the Secretary of Homeland Security, had completed several of these action items and others were under way. For example, the Secretary of Homeland Security sent a memo to all DHS component heads in April 2013 stating that it is DHS’s policy to prohibit the consideration of race or ethnicity in DHS’s investigation, screening, and enforcement activities in all but the most exceptional instances.

During our visits to four airports, we asked a random sample of 25 BDOs at the airports to what extent they had seen BDOs in their airport referring passengers based on race, national origin, or appearance rather than behaviors. These responses are not generalizable to the entire BDO population at SPOT airports. Of the 25 randomly selected BDOs we interviewed, 20 said they had not witnessed profiling, and 5 BDOs (including at least 1 from each of the four airports we visited) said that profiling was occurring at their airports, according to their personal observations. Also, 7 additional BDOs contacted us over the course of our review to express concern about the profiling of passengers that they had witnessed. We did not substantiate these specific claims.

In an effort to further assess the race, sex, and national origin of passengers who were referred by BDOs for additional screening, we analyzed the available information in the SPOT referral database and the Federal Air Marshal Service’s (FAMS) Transportation Information Sharing System (TISS) database. However, we found that the SPOT referral database does not allow for the recording of information such as race or gender. Without recording these data for every referral, it is difficult to disprove or substantiate such accusations. Since program-wide data on race were not available in the SPOT database, we analyzed a subset of available arrest data that were entered into the TISS database, which allows for race to be recorded. However, because there is not a unique identifier to link referrals from the SPOT database to information entered into TISS, we experienced obstacles when we attempted to match the two databases. For the SPOT referrals we were able to match, we found that data on race were inconsistently recorded in TISS. The limitations associated with matching the two databases and the incompleteness of the race data in TISS made analyzing trends or anomalies in the data impractical.

In March 2013, BDA officials stated that they had initiated a feasibility study to determine the efficacy of collecting data on the race and national origin of passengers referred by BDOs. A pilot is to be conducted at approximately five airports, which have not yet been selected, to collect data and examine whether this type of data collection is feasible and if the data can be used to identify any airport-specific or system-wide trends in referrals. According to BDA officials, the purpose of this study is to examine whether disparities exist in the referral trends, and if so, whether these differences suggest discrimination or bias in the referral process. This pilot is to also include an analysis of the broader demographics of the flying public—not just those referred by BDOs for additional screening—which is information that TSA had not previously collected. Having additional information on the characteristics of the flying public that may be used to compare to the characteristics of those passengers referred by the SPOT program—if TSA determines these data can feasibly be collected—could help enable TSA to reach reasonable conclusions about whether allegations of passenger profiling can be substantiated.

Appendix II: Our Analysis of Validation Study Data on SPOT Behavioral Indicators

The validation study reported that 14 of the 41 SPOT behavioral indicators were positively and significantly related to one or more of the study outcomes, but did not report that any of the indicators were negatively and significantly related to the outcome measures. That is, passengers exhibiting the SPOT behaviors that were positively and significantly related were more likely to be arrested, to possess fraudulent documents, or possess prohibited or illegal items. Conversely, passengers exhibiting the behaviors that were negatively and significantly related were less likely to be arrested, to possess fraudulent documents, or possess serious prohibited or illegal items than those who did not exhibit the behavior. While recognizing that the SPOT referral data used in this analysis were potentially unreliable, we replicated the SPOT indicator analysis with the full set of SPOT referral cases from January 1, 2006, to October 31, 2010, and found, consistent with the validation study, that 18 of the 41 behavioral indicators were positively and significantly related to one or more of the outcome measures. We also found, however, that 20 of the 41 behavioral indicators were negatively and significantly related to one or more of the study outcomes. That is, we identified 20 SPOT behavioral indicators that were more commonly associated with passengers who were not identified as high-risk passengers, than with passengers who were identified as high-risk passengers. Of the 41 behavioral indicators in the analysis, almost half of the passengers referred by BDOs for referral screening exhibited one indicator.

Appendix III: Objectives, Scope, and Methodology

Objectives

This report addresses the following questions: 1. To what extent does available evidence support the use of behavioral indicators to identify aviation security threats? 2. To what extent does TSA have data necessary to assess the effectiveness of the SPOT program in identifying threats to aviation security?

In addition, this report provides information on TSA’s response to recent allegations of racial profiling in the SPOT program, which can be found in appendix I.

Overview of Our Scope and Methodology

To obtain background information and identify changes in the SPOT program since our May 2010 report, we conducted a literature search to identify relevant reports, studies, and articles on passenger screening and deceptive behavior detection. We reviewed program documents in place during the period October 2010 through June 2013, including SPOT standard operating procedures, behavior detection officer performance standards and guidance, a strategic plan, and a performance metrics plan. We met with headquarters TSA and Behavior Detection and Analysis (BDA) program officials to determine the extent to which TSA had implemented recommendations in our May 2010 report and obtain an update on the SPOT program. In addition, we met with officials from U.S. Customs and Border Protection and the Federal Bureau of Investigation (FBI) Behavioral Science Unit to determine the extent to which they use behavior detection techniques. We also interviewed officials in DHS’s OIG, who were working on a related audit.

We analyzed data for fiscal years 2011 and 2012 from TSA’s SPOT referral database, which is to record all incidents in which BDOs refer passengers for additional screening, including the airport, time and date of the referral, the names of the BDOs involved in the referral, BDOs’ observation of the passengers’ behaviors, and any actions taken by law enforcement officers, if applicable. We also analyzed data for fiscal years 2011 and 2012 from the FAMS Transportation Information Sharing System (TISS) database, which is a law enforcement database designed to retrieve, assess, and disseminate intelligence information regarding transportation security to FAMS and other federal, state, and local law enforcement agencies. We reviewed available documentation on these databases, such as user guides, data audit reports, and training materials, and interviewed individuals responsible for maintaining these systems. In addition, we analyzed data on BDOs working at airports during this 2-year period, such as date started at TSA, date started as BDO, race, gender, and performance rating scores from TSA’s Office of Human Capital, and data on the number of hours worked by these BDOs provided by TSA’s Office of Security Operations officials and drawn from the U.S. Department of Agriculture’s National Finance Center database, which handles payroll and personnel data for TSA and other federal agencies. Further, we analyzed financial data from fiscal years 2007 through 2012 provided by BDA to determine the expenditures associated with the SPOT program. Additional information about steps we took to assess the reliability of these data is discussed below. We interviewed BDA officials in the Office of Security Capabilities and the Office of Human Capital on the extent to which they collect and analyze these data.

We conducted visits to four airports—Orlando International in Orlando, Florida; Detroit Metropolitan Wayne County in Detroit, Michigan; Logan International in Boston, Massachusetts; and John F. Kennedy International in New York City, New York. We selected this nonprobability sample based on the airports’ size and participation in behavior detection programs. As part of our visits, we interviewed a total of 25 BDOs using a semi-structured questionnaire, and their responses are not generalizable to the entire BDO population at SPOT airports. These BDOs were randomly selected from a list of BDOs on duty at the time of our visit. We interviewed BDO managers and TSA airport managers, such as federal security directors, who oversee the SPOT program at the airports. In addition, to obtain law enforcement officials’ perspectives on the SPOT program and their experiences in responding to SPOT referrals, we interviewed officials from the local airport law enforcement agency with jurisdiction at the four airports we visited (Orlando Police Department, Wayne County Airport Authority, Massachusetts State Police, and Port Authority of New York and New Jersey) and federal law enforcement officials assigned to the airports, including U.S. Customs and Border Protection, the FBI, and U.S. Immigration and Customs Enforcement. In nonprobability sampling, a sample is selected from knowledge of the population’s characteristics or from a subset of a population where some units in the population have no chance, or an unknown chance, of being selected. A nonprobability sample may be appropriate to provide illustrative examples, or to provide some information on a specific group within a population, but it cannot be used to make inferences about a population or generalize about the population from which the sample is taken. The results of our visits and interviews provided perspectives about the effectiveness of the SPOT program from local airport officials and opportunities to independently observe TSA’s behavior detection activities at airports, among other things.

Validation Study

To assess the soundness of the methodology and conclusions in the DHS April 2011 validation study, we reviewed the validation study and Technical Advisory Committee (TAC) final reports and appendixes, and other documents, such as the contractor’s proposed study designs, contracts to conduct the study, data collection training materials, and interim reports on data monitoring visits and study results. We assessed these efforts with established practices in designing evaluations and generally accepted statistical principles. We obtained the validation study datasets from the contractor and replicated several of the analyses, based on the methodology described in the final report. Generally, we replicated the study’s split-sample analyses, and as an extra step, extended those analyses using the full sample of SPOT referral data, as discussed below and in appendix II. In addition, we interviewed headquarters TSA, BDA, and Science and Technology Directorate (S&T) officials responsible for the validation study, representatives from the contractor who conducted the study, and 8 of the 12 members of the TAC who commented on and evaluated the adequacy of the validation study and issued a separate report in June 2011.

Data Reliability

To assess the reliability of the SPOT referral data, we reviewed relevant documentation, including privacy impact assessments and a 2012 data audit of the SPOT database, and interviewed TSA and BDA headquarters and field officials about the controls in place to maintain the integrity of the data. To determine the extent to which the SPOT database is accurate and complete, we reviewed the data in accordance with established procedures for assessing data reliability and conducted tests, such as electronic tests to determine if there were anomalies in the dataset (such as out-of-range dates and missing data) and reviewed a sample of certain coded data fields and compared them with narrative information in the open text fields. We determined that the data for fiscal years 2011 and 2012 across the 49 airports in our scope were sufficiently reliable for us to use to reflect the total number of SPOT referrals and arrests made, and to standardize the referral and arrest data, based on the number of hours each BDO spent performing operational SPOT activities.

In October 2012, TSA completed an audit of the data contained in the SPOT referral database in which it identified common errors, such as missing data fields and incorrect point totals. According to the 2012 audit, for the time period of March 1, 2010, through August 31, 2012, covering more than 108,000 referrals, the SPOT referral database had an overall error rate of 7.96 percent, which represented more than 8,600 known errors and more than 14,000 potential errors. According to TSA, the agency has begun taking steps to reduce this error rate, including visits to airports with significant data integrity issues and the development of a new SPOT referral database that is designed to prevent the most common errors from occurring. BDA officials told us that they have begun steps toward a nationwide rollout of their new system in May 2013, which includes pilots and developing procedures to mandate airports’ use of the system. On the basis of our review of the types of errors identified by the data audit, we determined that the SPOT referral data were sufficiently reliable for us to analyze BDO referral rates. However, the audit identifies problems with arrest data, which is one of the three categories of “potential errors.” The audit does not report on the magnitude of this error category, because identifying these errors requires a manual audit of the data at the airport level. As a result, we determined that the arrest data were not reliable enough for us to report on details about the arrests.

Use of Behavioral Indicators

To determine the extent to which available evidence exists to support the use of behavioral indicators to identify security threats, we analyzed research on behavioral indicators, reviewed the validation study findings on behavioral indicators, and analyzed SPOT referral data.

Research on Behavioral Indicators

Working from a literature review of articles from 2003 to 2013 that were identified using search terms such as “behavior detection deception,” and discussions with researchers who had published articles in this area, we contacted other researchers to interview and academic and government research to review. While the results of our interviews cannot be used to generalize about all research on behavior deception detection, they represent a mix of researchers and views by virtue of their affiliation with various academic institutions and governments, authorship of meta- analyses on these issues, and subject matter expertise in particular research areas.

We also reviewed more than 40 articles and books on behavior-based deception detection dating from 1999 to 2013. These articles, books, and reports were identified by our literature search of databases, such as ArticleFirst, ECO, WorldCat, ProQuest, and Academic One File and recommendations by TSA and the experts we interviewed. Through our discussions and research, we identified four meta-analyses, which used an approach for statistically cumulating the results of several studies to answer questions about program impacts. These meta-analyses analyzed “effect sizes” across several studies—the measure of the difference in outcome between a treatment group and a comparison group. For example, these meta-analyses measured the accuracy of an individual’s deception judgments when assessing another individual’s credibility in terms of the percentage that lies and truths were correctly classified and the impact of various factors on the accuracy of deception judgments, such as the liar’s motivation or expertise of the individual making the judgment. We reviewed the methodologies of 4 meta-analyses covering over 400 separate studies on detection deception over a 60-year period, including whether an appropriate evaluation approach was selected for each meta-analysis, and whether the data were collected and analyzed in ways that allowed valid conclusions to be drawn, in accordance with established practices in evaluation design. In addition, we interviewed two authors of these meta-analyses to ensure that the analyses were sound and we determined that the analyses were sufficiently reliable for describing what evidence existed to support the use of behavioral indicators to identify security threats. We determined that the research we identified was sufficiently reliable for describing the evidence that existed regarding the use of behavioral indicators to identify security threats.

Further, we reviewed documents developed by TSA and other foreign countries as part of an international study group to assess TSA’s efforts to identify best practices on the use of behavioral detection in an airport environment.

Validation Study Results on SPOT Indicators

To assess the soundness of the methodology and conclusions in the April 2011 validation study finding that 14 of the 41 SPOT indicators were related to outcomes that indicate a possible threat, we reviewed evidence supporting our May 2010 conclusions that the SPOT referral database lacked controls to help ensure the completeness and accuracy of the data. We interviewed TSA officials and obtained documentation, such as a data audit report and a functional requirements document, to determine the extent to which problems in the SPOT database were being addressed. We also reviewed the June 2011 TAC final report and interviewed contractor officials regarding analysis limitations because of data sparseness, or low frequency of occurrences of indicators in the SPOT database.

We also obtained the dataset used in the study—SPOT referral data from January 2006 through October 2010—and replicated the SPOT indicator analyses described in the study. Although we found that the data were not sufficiently reliable for use in conducting a statistical analysis of the association between the indicators and high-risk passenger outcomes, we used the data to assess the study’s methodology and conclusions. The dataset included a total of 247,630 SPOT referrals from 175 airports. As described in the validation study, we calculated whether the odds on each of the four study outcome measures—LEO arrest, possession of fraudulent documents, possession of a serious prohibited or illegal item, or the combination of all three measures—were associated with the 41 SPOT indicators. These odd ratios were derived from four sets of 41 separate cross-tabulations—2 x 2 tables—in which each of the four outcomes is cross-classified by each of the 41 individual indicators. Odds ratios greater than 1.0 indicate positive associations, that is, passengers exhibiting the behavior were more likely to be arrested, to possess fraudulent documents, or to possess serious prohibited or illegal items. On the other hand, odds ratios of less than 1.0 indicate negative associations, that is, passengers exhibiting the behavior were less likely to be arrested, to possess fraudulent documents, or to possess serious prohibited or illegal items than those who do not exhibit the behavior. The number of positive and significant associations we detected was slightly larger than the number reported in the validation study mainly because we reported results from an analysis of the full sample of SPOT referrals—a total of 247,630 SPOT passenger referrals. In contrast, the validation study stated that a split-sample approach was used, in which each years’ dataset was split into two stratified random subsets across the years and analyses were conducted independently on each aggregated subset. The validation study stated that this approach allowed an examination of the extent to which results may vary across each subset and to address possible random associations in the data. The validation study further stated that this was important because changes in the SPOT program, such as fewer airports and BDOs involved in the earlier years and small changes to the SPOT instrument in March 2009, could have affected the analyses. However, after replicating the split- sample approach, we determined that it was not the most appropriate one to use because it substantially diminished the power to detect significant associations in light of how infrequently referrals occurred. We report the results of our analyses of the full sample of SPOT referrals that indicate behavioral indicators that are positively and significantly related, as well as negatively and significantly related, in the behavioral indicator section of the report and in appendix II.

SPOT Referral Data

To determine the extent to which SPOT referrals varied by BDOs across airports for fiscal years 2011 and 2012, we initially selected the 50 airports identified by TSA’s May 2012 Current Airports Threat Assessment report as having the highest probability of threat from terrorist attacks. We chose to limit the scope of our review to the top 50 airports because the majority of the BDOs are deployed to these airports; and they account for 68 percent of the passenger throughput, and 75 percent of SPOT referrals. To standardize the referral rates across airports, we calculated the number of SPOT referrals by individual BDOs and matched these BDOs by the number of hours that particular BDOs spent performing SPOT activities. San Francisco International Airport was in the initial selection of 50 airports; however, we excluded San Francisco International because the hourly data provided to us for San Francisco BDOs, who are managed by a screening contractor, were not comparable with the hourly data provided to us for TSA-managed BDOs. The scope of our analysis was then 49 SPOT airports.

To calculate BDO hours spent performing SPOT activities, we analyzed BDO time and attendance data provided by TSA for fiscal years 2011 and 2012 from the U.S. Department of Agriculture’s National Finance Center. We limited our analysis to the hours BDOs spent performing SPOT activities because it is primarily during these times that BDOs make SPOT referrals. Thus, BDO hours charged to activities such as leave, baggage screening, or cargo inspection activities were excluded. For example, we found that BDOs had charged time to cargo inspection activities that were unrelated to the SPOT program. These inspections are carried out under TSA’s Compliance Division in the Office of Security Operations, and are designed to ensure compliance with transportation security regulations. We also limited our analysis to nonmanager BDOs, as managers are not regularly engaged in making referrals. Finally, about 55 BDOs, or about 2 percent of the approximately 2,400 BDOs (including both managers and nonmanagers), were not included in our analysis because we could not reconcile their names with time and attendance data after several attempts with TSA officials. We calculated average referral rates per 160 hours worked, or about 4 40-hour weeks, across 2,199 BDOs working at 49 airports, and a referral rate for each airport.

To better understand the variation in referral rates, we conducted a multivariate analysis to determine whether certain variables affected SPOT referral rates and LEO referral rates, including airports at which BDOs worked during fiscal years 2011 and 2012; BDO annual performance scores for 2011 and 2012; years of experience with TSA and as a BDO; and demographic information on BDOs, such as age, gender, race, and highest educational level attained at the time of employment. Although multivariate methods do not allow us to establish that referral rates are causally related to the BDO characteristics we had information about, they allowed us to examine the associations between referral rates and the different specific BDOs while controlling for other BDO characteristics, including the airports in which the BDOs worked.

Moreover, the methods we employed allowed us to determine whether the observed differences in the sample data were different more than by merely chance fluctuations. Our statistical models and estimates are sensitive to our choice of variables; thus, researchers testing different variables may find different results. See appendix IV for additional information on the results of our analyses.

Data to Assess Effectiveness of SPOT

To determine the extent to which TSA has data necessary to assess the effectiveness of the SPOT program in identifying threats to aviation security, we reviewed the validation study’s findings comparing passengers selected by SPOT with randomly selected passengers, analyzed TSA plans and analyses designed to measure SPOT’s effectiveness, and analyzed data on SPOT referrals and LEO arrests.

Validation Study Results Comparing Passengers Selected by SPOT with Randomly Selected Passengers

To assess the soundness of the methodology and conclusions in the April 2011 validation study findings that SPOT was more likely to identify high- risk passengers than a random selection of passengers, we assessed the study design and implementation against established practices for designing evaluations and generally accepted statistical principles. These practices include, for example, probability sample methods, data collection and monitoring procedures, and quasi-experimental design. We obtained the validation study datasets and replicated the study findings, based on the methodology described in the final report. Further, we analyzed the validation study data from December 1, 2009, to October 31, 2010, on passengers who were referred to a LEO and who were ultimately arrested. To the extent possible, we reviewed SPOT data to determine the reasons for the arrest and if there were differences between arrested passengers who were referred by SPOT and arrested passengers who were randomly selected.

TSA Performance Data

To determine the extent to which TSA has plans to collect and analyze performance data to assess SPOT’s overall effectiveness, we reviewed TSA’s efforts to inform the future direction of BDA and the SPOT program, such as a return-on-investment and risk-based allocation analyses. We evaluated TSA’s efforts against DHS, GAO, and other guidance regarding these analyses. For example, we reviewed TSA’s return-on-investment analysis against the analytical standards in the Office of Management and Budget’s Circular A-94, which provides guidance on conducting benefit-cost and cost-effectiveness analyses. We also reviewed documentation associated with program oversight, including a 2012 performance metrics plan, and evaluated TSA’s efforts to collect and analyze data to provide oversight of BDA and the SPOT program against criteria in Office of Management and Budget guidance and Standards for Internal Control in the Federal Government. Further, we reviewed performance work statements in TSA contracts to determine the extent to which the contractor’s work is to fulfill the tasks in TSA’s performance metrics plan. Also, we reviewed FAMS law enforcement reports, TISS incident reports, and the SPOT referral database to determine the extent to which information from BDO referrals was used for further investigation to identify potential ties to terrorist investigations. We also analyzed SPOT referral data that TSA uses to track SPOT program activities, including the number of passengers who were referred to a LEO and ultimately arrested for fiscal years 2011 and 2012.

Profiling Allegations

To provide information about how TSA and DHS’s OIG have examined allegations of racial and other types of profiling of passengers by BDOs, we reviewed documentation from 2010 to 2013, such as investigation reports, privacy impact assessments, BDO training materials, and TSA memos. To explore the extent to which we could determine the race, gender, and national origin of passengers who were referred by BDOs for additional screening, we analyzed information in the SPOT referral database and the TISS database for fiscal years 2011 and 2012. We reviewed a September 2012 TSA contract that will, among other things, study whether any evidence exists for racial or ethnic profiling in the SPOT program. We also reviewed interim reports produced by the contractor as of June 2013. Because racial profiling allegations in Boston were made during the course of our review, we asked the random sample of 25 BDOs at the four airports we visited to what extent they had seen BDOs in their airport referring passengers based on race, national origin, or appearance rather than behaviors. These responses are not generalizable to the entire BDO population at SPOT airports. Further, 7 additional BDOs contacted us over the course of our review to express concern about the profiling of passengers that they had witnessed. We did not substantiate these specific claims. We also interviewed TSA headquarters and field officials, such as federal security directors and BDO managers, as well as DHS OIG officials.

We conducted this performance audit from April 2012 to November 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix IV: Characteristics and Referral Rates of Behavior Detection Officers at 49 SPOT Airports

To better understand the variation in referral rates, we analyzed whether certain variables affected SPOT referral rates and LEO referral rates, including BDO characteristics, such as average performance scores for fiscal years 2011 and 2012, years of TSA and BDO experience, age, gender, educational level, years employed at TSA and as a BDO, and race, as well as the airport in which the BDOs worked. As described earlier, these analyses standardized SPOT referral data for 2,199 BDOs across 49 airports for fiscal years 2011 and 2012.

BDO Characteristics and Referral Rates

The characteristics of the 2,199 BDOs in our analyses varied across different categories, as shown in table 3. About 51 percent of the BDOs were under 40 years of age, and slightly more than 25 percent were 50 years or older. Nearly 64 percent of the BDOs joined TSA before the end of 2005, but the majority, or more than 85 percent, became BDOs after the beginning of 2008. Nearly 65 percent of the BDOs were male. Fifty percent were white, about 26 percent were African-American, and about 18 percent were Hispanic. About 65 percent of the BDOs had a high school education or less. The BDOs were distributed unevenly across airports, with the largest numbers in Logan International (Boston), Dallas- Fort Worth International, John F. Kennedy International (New York), Los Angeles International, and O’Hare International (Chicago). Each BDO worked primarily in one airport during the 2-year period. For example, 80 of the 2,199 BDOs, or about 4 percent, worked in multiple airports and the remaining 2,119 BDOs, or 96 percent, worked at one airport during the 2- year time period.

Overall, BDOs averaged about 1.57 SPOT referrals and 0.22 LEO referrals per 160 hours worked. These rates vary across the different BDO categories. However, these differences should be considered cautiously, as differences that appear to exist across categories for one characteristic may be confounded with differences across others. For example, the apparent difference in referral rates between younger and older BDOs may be the result of younger BDOs working disproportionately in airports with higher referral rates.

Multivariate Analysis of SPOT and LEO Referral Rates

To better understand the effects of BDO characteristics, including the airports they worked in, on SPOT referral and LEO referral rates, we conducted simple regression analyses. Overall, the greatest amount of the variation in BDO SPOT referral rates was explained by the airport at which the referral occurred. That is, the BDO’s referral rate was associated substantially with the airport at which he or she was conducting SPOT activities.

These analyses show the size and significance of regression coefficients, from ordinary least-squares regression models, which reflect the estimated differences in the average number of SPOT referrals and LEO referrals across categories of BDO, and across airports.

BDOs in a few airports averaging significantly higher rates of referrals than BDOs in the referent category, and BDOs in most of the other airports averaging significantly lower LEO referral rates. Because they were less common, LEO referrals may have been more difficult to predict that SPOT referrals. Differences in the other BDO characteristics— multivariate model 1—collectively accounted for a small percentage of the variation in average LEO referral rates, while differences across airports accounted for a larger percentage.

Airport Throughput Analysis

Separate analyses we conducted revealed that the sizeable and highly significant differences in SPOT referral rates and LEO referral rates across airports were not fully accounted for by differences in the number of passengers who pass through airport checkpoints.

Appendix V: TSA’s Performance Metrics for Behavior Detection and Analysis

Table 4 shows TSA’s proposed performance metrics as detailed in appendix G in its Behavior Detection and Analysis performance metrics plan dated November 2012.

Table 5 shows the validity, reliability, and frequency score TSA determined for each metric and the overall score for each metric subcategory, as detailed in appendix C of its performance metrics plan, dated November 2012. TSA’s performance metrics plan defines validity as the ability of the metric to measure BDO performance, reliability as the level of certainty that data are collected precisely with minimal possibility for subjectivity or gaming the system, and frequency as the level of difficulty in collecting the metric and whether the metric is collected at the ideal number of scheduled recurrences.

Appendix VI: Comments from the Department of Homeland Security

Appendix VII: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, David M. Bruno (Assistant Director); Charles W. Bausell, Jr.; Andrew M. Curry; Nancy K. Kawahara; Elizabeth B. Kowalewski; Susanna R. Kuebler; Thomas F. Lombardi; Grant M. Mallie; Amanda K. Miller; Linda S. Miller; Lara R. Miklozek; Douglas M. Sloane; and Jeff M. Tessin made key contributions to this report.