Aviation Security: Efforts to Validate TSA's Passenger Screening Behavior Detection Program Underway, but Opportunities Exist to Strengthen Validation and Address Operational Challenges
Published: 2010-05-20T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: To enhance aviation security, the Transportation Security Administration (TSA) began initial testing in October 2003 of its Screening of Passengers by Observation Techniques (SPOT) program. Behavior Detection Officers (BDO) carry out SPOT's mission to identify persons who pose a risk to aviation security by focusing on behavioral and appearance indicators. GAO was asked to review the SPOT program. GAO analyzed (1) the extent to which TSA validated the SPOT program before deployment, (2) implementation challenges, and (3) the extent to which TSA measures SPOT's effect on aviation security. GAO analyzed TSA documents, such as strategic plans and operating procedures; interviewed agency personnel and subject matter experts; and visited 15 SPOT airports, among other things. Although the results from these visits are not generalizable, they provided insights into SPOT operations.
section_title: What GAO Found
paragraphs: Although the Department of Homeland Security (DHS) is in the process of validating some aspects of the SPOT program, TSA deployed SPOT nationwide without first validating the scientific basis for identifying suspicious passengers in an airport environment. A scientific consensus does not exist on whether behavior detection principles can be reliably used for counterterrorism purposes, according to the National Research Council of the National Academy of Sciences. According to TSA, no other large-scale security screening program based on behavioral indicators has ever been rigorously scientifically validated. DHS plans to review aspects of SPOT, such as whether the program is more effective at identifying threats than random screening. Nonetheless, DHS's current plan to assess SPOT is not designed to fully validate whether behavior detection can be used to reliably identify individuals in an airport environment who pose a security risk. For example, factors such as the length of time BDOs can observe passengers without becoming fatigued are not part of the plan and could provide additional information on the extent to which SPOT can be effectively implemented. Prior GAO work has found that independent expert review panels can provide comprehensive, objective reviews of complex issues. Use of such a panel to review DHS's methodology could help ensure a rigorous, scientific validation of SPOT, helping provide more assurance that SPOT is fulfilling its mission to strengthen aviation security. TSA is experiencing implementation challenges, including not fully utilizing the resources it has available to systematically collect and analyze the information obtained by BDOs on passengers who may pose a threat to the aviation system. TSA's Transportation System Operations Center has the resources to investigate aviation threats but generally does not check all law enforcement and intelligence databases available to it to identify persons referred by BDOs. Utilizing existing resources would enhance TSA's ability to quickly verify passenger identity and could help TSA to more reliably "connect the dots." Further, most BDOs lack a mechanism to input data on suspicious passengers into a database used by TSA analysts and also lack a means to obtain information from the Transportation System Operations Center on a timely basis. TSA states that it is in the process of providing input capabilities, but does not have a time frame for when this will occur at all SPOT airports. Providing BDOs, or other TSA personnel, with these capabilities could help TSA "connect the dots" to identify potential threats. Although TSA has some performance measures related to SPOT, it lacks outcome-oriented measures to evaluate the program's progress toward reaching its goals. Establishing a plan to develop these measures could better position TSA to determine if SPOT is contributing to TSA's strategic goals for aviation security. TSA is planning to enhance its evaluation capabilities in 2010 to more readily assess the program's effectiveness by conducting statistical analysis of data related to SPOT referrals to law enforcement and associated arrests.

section_title: Letter
paragraphs: The terrorist attacks of September 11, 2001, highlighted the need to  improve security within the nation’s civil aviation system to deter persons  seeking to repeat similar attacks on the nation’s critical infrastructure.  In  October 2003, the Transportation Security Administration (TSA) of the  Department of Homeland Security (DHS) conducted an operational test of  the use of  behavior detection techniques to screen passengers in an  airport environment, and subsequently began training certain  Transportation Security Officers (TSO)—TSA employees responsible for  screening passengers and their property—in these techniques.  These  TSOs performed behavior observation as a collateral duty.  Beginning in  fiscal year 2007, TSA created separate Behavior Detection Officer (BDO)  positions as part of the Screening of Passengers by Observation  Techniques (SPOT) program.  According to TSA, the SPOT program is a  derivative of other behavioral analysis programs that have been  successfully employed by law enforcement and security personnel both in  the United States and around the world, particularly that of Israel’s airline,  El Al.
TSA designed SPOT to provide BDOs with a means of identifying persons  who may pose a potential security risk at TSA-regulated airports by  focusing on behaviors and appearances that deviate from an established  baseline, and that may be indicative of stress, fear, or deception.   Passengers in an airport terminal, including those waiting in security  checkpoint lines, are observed by the BDOs to determine if their  behavioral and appearance indicators—which are assigned varying points  by SPOT—have (in combination) exceeded a predetermined numerical  threshold.  In cases where the passenger exceeds the threshold, the  passenger is referred for additional screening by BDOs and a TSO.  During  this referral screening, if the passenger exhibits behaviors that exceed  another numerical threshold, they are to be referred to a law enforcement  officer (LEO) for further investigation.  In addition to observing  passengers at airport checkpoints, BDOs may patrol throughout an airport  terminal, and sometimes participate in other activities, such as TSA’s  Visible Intermodal Prevention and Response team operations.  These  teams are responsible for periodically augmenting security at air and  ground transportation facilities around the country.
As of March 2010, TSA deployed about 3,000 BDOs at an annual cost of  about $212 million; this force increased almost fifteen-fold between March  2007 and July 2009.  BDOs have been selectively deployed to 161 of the 457  TSA-regulated airports in the United States at which passengers and their  property are subject to TSA-mandated screening procedures.  The  conference report accompanying the fiscal year 2010 DHS appropriations  act provided that $211.9 million of aviation security funding was for the  SPOT program.  The administration has requested $232 million for SPOT  for fiscal year 2011, a $20.2 million (9.5 percent) increase over the current  funding level.  This increase would support a workforce increase from  about 3,000 to 3,350 BDOs.  If this funding request is approved and  maintained, SPOT would cost about $1.2 billion over the next 5 years.
You asked us to address SPOT’s development and implementation.  This  report addresses the following questions:   1.  To what extent did TSA determine whether SPOT had a scientifically  validated basis for identifying passengers before deploying it and  utilize recognized best practices during SPOT’s development?   2.  What management challenges, if any, have emerged during the  implementation of SPOT at the nation’s airports?   3.  To what extent has TSA measured SPOT’s effect on aviation security?   4.  To what extent has TSA incorporated the attributes of an effective  training program into the training for SPOT?
This report is a public version of the restricted report (GAO-10--157SU)  that we provided to you on May 14, 2010.  DHS and TSA deemed some of  the information in the restricted report as sensitive security information,  which must be protected from public disclosure. Therefore, this report  omits this information.  Although the information provided in this report is  more limited in scope, it addresses the same questions as the restricted  report. Also, the overall methodology used for both reports is the same.
To determine the extent to which TSA determined whether SPOT had a  scientifically validated basis for identifying passengers who may pose a  risk to aviation security before deploying it, we reviewed literature on  behavior analysis by subject matter experts, and analyzed relevant reports  and books on the topic.  These included a 2008 study by the National  Research Council of the National Academy of Sciences that included a  discussion section on deception and behavioral surveillance, as well as  other issues related to behavioral analysis.  We interviewed seven  recognized experts in the field, and an expert on emergency responses to  terror attacks and mathematical models in operations management.   Although the views of these experts cannot be generalized across all  experts on behavior analysis, because we selected these individuals based  on their publications on behavioral analysis or related topics, their  recognized accomplishments and expertise, and, in some cases, TSA’s use  of their work or expertise to design and review the SPOT program’s  behaviors, they provided us with an understanding of the fundamentals of  behavior analysis, and its use in airports.  We also interviewed cognizant  officials from other U.S. government agencies that utilize behavior analysis  in their work, including U.S. Customs and Border Protection (CBP), the  U.S. Secret Service, the Federal Air Marshall Service (FAMS), and the  Federal Bureau of Investigation (FBI).  To better understand how SPOT  incorporated expertise on behavior analysis for aviation security, we also  interviewed current and retired officials of Israel’s El Al Airlines, whose  security processes TSA cites as providing part of the basis of the SPOT  program.
To determine to what extent TSA utilized best practices during SPOT’s  development—including carrying out a comprehensive risk assessment, a  cost-benefit analysis, and a strategic plan—we interviewed program  officials and reviewed related program documentation, including briefings  used in the course of developing and fielding SPOT, strategic plans, and  standard operating procedures.  We compared these documents to DHS’s  2006 Cost Benefit Analysis Guidebook, Office of Management and  Budget (OMB) guidance, and DHS’s 2006 and 2009 National  Infrastructure Protection Plans (NIPP) , which set forth a risk management  framework to guide security decision making and resource allocation  decisions, and our previous work on the characteristics of an effective  strategic plan.
To identify any challenges that emerged during implementation of the  SPOT program, we conducted field site visits to 15 TSA-regulated airports  with SPOT that represent almost 10 percent of the 161 TSA-regulated  airports with SPOT to observe operations and meet with key program  personnel.  We chose airports with high, medium, and low passenger  volume; airports with BDOs who are TSA (i.e., government) employees  and an airport with BDOs employed by contractors as part of the TSA  Screening Partnership Program; and airports with LEOs who were  identified by TSA as having received some form of behavior detection  training and airports where they were not known to have received such  training.  We also selected airports on the basis of TSA’s assessment of  which ones are at highest risk of attack by terrorists, including the 2 that  ranked the highest, as reported in TSA’s Current Airport Threat  Assessment.  Since the airports we selected range broadly in terms of   passenger volume, physical size and layout, geographic location, and  potential value as a target for terrorism, among other things, the results  from these visits are not generalizable to other airports.  However, these  visits provided helpful insights into the operation of SPOT at airports.  In  addition, to determine whether challenges emerged in implementing  SPOT, we compared TSA’s approach for implementing and managing  SPOT to our Standards for Internal Control in the Federal Government  and to risk management principles we had previously identified.  In  reviewing TSA’s approach to developing and implementing SPOT, we  considered relevant laws, regulations, and other materials, including those  related to privacy, such as TSA’s Privacy Impact Assessments. To obtain  comparative data on how SPOT had been implemented at different  airports across the nation, we conducted a survey of all Federal Security  Directors responsible for security operations at TSA-regulated airports  with SPOT.  (This accounted for all 161 TSA-regulated airports with  SPOT because a single Federal Security Director may be responsible for  several airports.)  We obtained a 100 percent response rate.  This survey  asked, among other things, about the relationship between LEOs and the  airport authority and BDOs.  In addition, to understand the interaction of  BDOs and LEOs, as well as other SPOT implementation issues, at each of  the 15 TSA-regulated airports we visited we spoke with BDO managers,  Federal Security Directors, Assistant Federal Security Directors, 1 or 2  BDOs, and 1 or 2 LEOs.
To determine the extent to which TSA has measured SPOT’s effect on  aviation security, we obtained and analyzed the TSA SPOT referral  database, which aims to record all incidents in which passengers who  have passed through the checkpoint are sent to SPOT referral screening  for additional questioning and screening of property and person.  The  database also maintains records of instances where passengers were  referred by a BDO to a LEO for questioning.  We assessed the reliability of  the SPOT referral data by (1) performing electronic testing of required  data elements, (2) reviewing existing information about the data and the  system that produced them, and (3) interviewing agency officials  knowledgeable about the data.  We found a number of problems related to  how the data were collected and recorded that are discussed later in this  report.  As a result, we were unable to use the SPOT referral data to assess  whether any behavior or combination of SPOT behaviors could be used to  reliably predict the final outcome of an incident involving the use of SPOT.   However, with the stated limitations in mind, and after resolving certain  contradictions and anomalies in the database, we utilized the SPOT  referral data to provide examples of information used by TSA to report on  the program's performance, including a count of arrests and the reasons  for those arrests.  In addition, to determine if individuals who were later  charged with or pleaded guilty to terrorism-related offenses had transited  SPOT airports and whether TSA could obtain information from these  transits to enhance its understanding of terrorist behaviors, we reviewed  CBP and Department of Justice information to (1) identify individuals who  were charged with or pleaded guilty to terrorism-related offenses and (2)  determine if these individuals had, prior to being charged, transited  airports where SPOT had been deployed.  Further, we used our survey of  Federal Security Directors at SPOT airports to determine the extent to  which video surveillance cameras, which could make video recordings of  terrorists transiting airports, are present at checkpoints.
To assess the extent that SPOT training incorporates the attributes of an  effective training program, we had TSA training experts complete a  training assessment tool that we developed using guidance we prepared in  our previous work for assessing training courses and curricula.  To better  understand how other entities train their employees in behavior detection,  and what their curricula include, we conducted site visits to the Secret  Service, CBP, FAMS, and the FBI, and also interviewed nongovernmental  experts on aspects of behavior detection training.  We interviewed BDOs  and BDO managers about the SPOT training.  In addition, we interviewed  El Al officials with regard to how El Al trains and tests its personnel in  behavior recognition and analysis.  Appendix I contains additional details  about our scope and methodology.
We conducted this performance audit from May 2008 through May 2010 in  accordance with generally accepted government auditing standards.   Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives.  We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The Aviation and Transportation Security Act established TSA as the  federal agency with primary responsibility for securing the nation’s civil  aviation system, which includes the screening of all passenger and  property transported by commercial passenger aircraft.  TSA currently  has direct responsibility for, or oversees the performance of, security  operations at approximately 457 TSA-regulated airports in the United  States implementing security requirements in accordance with TSA- approved security programs and other TSA direction.  At TSA-regulated  airports, prior to boarding an aircraft, all passengers, their accessible  property, and their checked baggage are screened pursuant to TSA- established procedures, which include, for example, passengers passing  through security checkpoints where they and their identification  documents are checked by TSOs and Travel Document Checkers, or by  Screening Partnership Program employees.
TSA uses multiple layers of security to deter, detect, and disrupt persons  posing a potential risk to aviation security.   These layers include three  principal types of screening employees at airport checkpoints—Travel  Document Checkers, who examine tickets, passports, and other forms of  identification; TSOs, who examine property, including checked baggage,  and persons using x-ray equipment and magnetometers, as well as other  devices; and BDOs, using SPOT to assess passenger behaviors and  appearance.  BDOs are the only type of TSA screening employees not  deployed to all TSA-regulated airports and all checkpoints within the  airports where it is deployed on a regular basis.  TSA deployed SPOT as an  added layer of security to help deter terrorists attempting to exploit TSA’s  focus on prohibited items and other potential security weaknesses.  Other  security layers cited by TSA include intelligence gathering and analysis;  passenger prescreening; random canine team searches at airports; federal  air marshals; reinforced cockpit doors; federal flight deck officers; the  passengers themselves; as well as other measures both visible and  invisible to the public.  Figure 1 shows TSA’s 20 aviation security layers.
The grey area in figure 1 highlights four layers that apply to passengers  and their property as they seek to board an aircraft.  Airport LEOs, another  layer of security cited by TSA, do not report to TSA and may not maintain  a physical presence at smaller TSA-regulated airports.  According to TSA,  each one of these layers alone is capable of stopping a terrorist attack.  In  combination, TSA states that their security value is multiplied, creating a  much stronger system, and that a terrorist who has to overcome multiple  security layers in order to carry out an attack is more likely to be  preempted, deterred, or to fail during the attempt.
subsections: section_title: SPOT Uses Behavior Detection Techniques to Assess Passenger Behaviors and Appearances
paragraphs: The SPOT program utilizes behavior observation and analysis techniques  to identify potentially high-risk passengers.  Individuals who exhibit  suspicious behaviors, including both physical and appearance indicators,  may be required to undergo additional screening.  Field agents and law  enforcement officers of other federal agencies and entities—such as the  FBI, the Secret Service, CBP, and FAMS—utilize elements of behavior  detection analysis as a part of their work.  In addition, some foreign  entities, such as Israel’s El Al airlines, use behavior detection and analysis  techniques as part of their security efforts.  However, TSA emphasized to  us that the SPOT program is unique among these entities because it uses a  point system to help identify suspicious persons on the basis of their  behavior and appearance and because behavior detection and analysis are  the central focus of SPOT.  Officials from the other agencies stated that  their field personnel incorporate behavior detection as one of many skills  used in their work; in contrast, behavior detection is the primary element  of the BDOs’ work.
SPOT trains BDOs to look for and recognize facial expressions, body  language, and appearance that indicate the possibility that an individual is  engaged in some form of deception and fears discovery.  These behaviors  and appearances are listed on a SPOT score sheet used in SPOT training.
Passenger behavior and appearance are to be compared by the BDOs— who typically work in two-person teams.  BDOs are expected to “walk the  line”—that is, to initiate casual conversations with passengers waiting in  line, particularly if their observations led them to question someone  exhibiting behaviors or appearances on the SPOT checklist.  As the BDOs  walk the line, and the passenger with SPOT indicators is reached, a casual  conversation is used to determine if there is a basis for observed behaviors  or appearances on the checklist.  In most instances, these conversations  provide information to the BDOs that permits them to consider the issue  resolved, and hence not a security concern.  Figure 2 below illustrates the  first step of the three-step SPOT process, the BDO-passenger interaction at  a checkpoint prior to the passenger passing through a magnetometer.
As shown in figure 2, passenger behavior and appearance are observed by  the BDOs as passengers wait in line for screening at a security checkpoint.   Even if the checkpoint is busy, the BDOs must attempt to visually scan all  the passengers waiting in line, as well as persons near the checkpoint, to  determine if any are showing behaviors or appearances on the SPOT  checklist.  According to TSA, on average a BDO has approximately 30  seconds to assess each passenger while the passenger waits in line.  For  passengers exhibiting indicators above baseline conditions, the BDOs are  to (mentally) add up the points assigned to each indicator they observe.   Both BDO team members must agree that observed indicators have  exceeded the predetermined numerical threshold, although they do not  have to identify the same indicators the passenger exhibited.  In instances  when a passenger’s SPOT indicators place them above the numerical  threshold, and the passenger has placed their property on the conveyor  belt for x-raying, and has walked through the magnetometer or equivalent  screening device for passengers, he or she will be directed to the second  step of SPOT, referral screening.  This involves additional questioning and  physical search of their person and property by BDOs and TSOs.  This  referral screening occurs in the checkpoint area.
If the passenger’s behavior escalates further—accumulating more points  based on the SPOT checklist—the BDOs are to refer the passenger to a  LEO.  A referral to a LEO is a potential third step in the SPOT process.   BDOs are not LEOs—they do not conduct criminal investigations, carry  weapons, or make arrests.
After a passenger has been referred by the BDOs to a LEO, the LEO is then  expected to independently determine, through additional investigation,  such as questioning the passenger and, if appropriate, by conducting an  identity verification and background check through the FBI’s National  Crime Information Center (NCIC), whether sufficient grounds exist to take  further action, such as detaining or arresting the passenger.  TSA officials  who are LEOs also have access to NCIC, such as an airport’s Assistant  Federal Security Director for Law Enforcement or federal air marshals.  NCIC is the FBI’s computerized index of criminal justice information (i.e.,  criminal record history information, fugitives, stolen properties, and  missing persons), available to federal, state, and local law enforcement  and other criminal justice agencies at all times.  Similarly, other federal  LEOs also have such access, including CBP, and Drug Enforcement  Agency (DEA) personnel.  However, since both local and federal LEOs  have other responsibilities, and may not be present at each operating  checkpoint, BDOs may have to seek them out to request an NCIC check.   According to TSA, aside from requiring that an airport maintain a law  enforcement presence, it exercises no jurisdiction over the law  enforcement activities of non-TSA officers or entities at an airport; thus, it  cannot require LEOs to conduct an NCIC check or to provide BDOs with  information about the ultimate disposition of cases referred by them to  LEOs.
Once the LEO concludes his or her investigation and determines whether  the passenger will be arrested or detained, TSA officials are to evaluate the  security concerns to determine whether to allow the passenger to proceed  to the boarding gate.  (In some instances, a LEO might choose not to arrest  or detain a passenger; TSA would then decide whether the infraction was  sufficiently serious to necessitate barring the passenger from boarding.)   After a referral incident has been resolved, BDOs are to enter information  about the incident into TSA’s SPOT referral database.  The data entered  are to include time, date, location of the incident, behaviors witnessed,  prohibited items found (if any), and information on the LEO’s response (if  applicable), such as whether the LEO questioned the passenger, arrested  the individual, or released the passenger.  The SPOT referral database  contains no personal identifying information about passengers.
section_title: SPOT Has Been Deployed in Phases
paragraphs: The SPOT program began with pilot tests in 2003 and 2004 at several New  England airports, in which TSA began using uniformed BDOs at airport  checkpoints.  After some initial pilot projects and test deployments, 644  BDOs were deployed to 42 airports in the first phase of the program from  November 2006 through June 2007.  As of March 2010, about 3,000 BDOs  utilizing SPOT were deployed at 161 of 457 TSA-regulated airports.
BDO eligibility is restricted to TSOs with at least 12 months of TSO  experience, or others with related security experience.  Applicants must  apply and be accepted into the BDO training program.  The training  includes 4 days of classroom courses, followed by 3 days of on-the-job  training.  BDOs must memorize all of the behaviors and appearances on  the SPOT checklist, as well as the point value assigned to each, in order to  be able to add these up to determine if a passenger should be sent to SPOT  referral screening.  BDO applicants must also pass a job knowledge test at  the conclusion of the training.  The test includes related multiple choice  questions, true or false statements, and case-based scenarios.
section_title: DHS Is Taking Action to Validate the Scientific Basis of TSA’s SPOT Program but Opportunities Exist to Help Inform Future Program Decisions
paragraphs: Although DHS is in the process of validating the way in which the SPOT  program utilizes the science of behavior detection in an airport  environment, TSA deployed SPOT nationwide before first determining  whether there was a scientifically valid basis for using behavior and  appearance indicators as a means for reliably identifying passengers as  potential threats in airports.  TSA reported that it deployed SPOT before a  scientific validation of the program was completed in response to the need  to address potential threats to the aviation system that would not  necessarily be detected by existing layers of aviation security.  TSA stated  that no other large-scale U.S. or international screening program  incorporating behavior- and appearance-based indicators has ever been  rigorously scientifically validated.  While TSA deployed SPOT on the basis  of some risk-related factors, such as threat information and airport  passenger volume, it did not use a comprehensive risk assessment to guide  its strategy of selectively deploying SPOT to 161 of the nation’s 457 TSA- regulated airports.  TSA also expanded the SPOT program over the last 3  years without the benefit of a cost-benefit analysis of SPOT.  Additionally,  TSA’s strategic plan for SPOT could be improved by the inclusion of  desirable characteristics identified in our prior work, such as risk  assessment information, cost and resources analysis, and a means for  collaboration with other key entities.
subsections: section_title: TSA Is in the Process of Validating the Scientific Basis Used to Identify Passengers with SPOT Behaviors
paragraphs: TSA proceeded with deploying SPOT on a nationwide basis before  determining whether the list of passenger behaviors and appearances  underpinning the SPOT program were scientifically validated, and whether  these techniques could be applied for counterterrorism purposes in an  airport environment.  In 2008, a report issued by the National Research  Council of the National Academy of Sciences noted that behavior and  appearances monitoring might be able to play a useful role in  counterterrorism efforts but stated that a scientific consensus does not  exist regarding whether any behavioral surveillance or physiological  monitoring techniques are ready for use in the counterterrorist context  given the present state of the science.  The report also stated that the  scientific evidence for behavioral monitoring is preliminary in nature.   According to the report, an information-based program, such as a behavior  detection program, should first determine if a scientific foundation exists  and use scientifically valid criteria to evaluate its effectiveness before  going forward.  The report added that programs should have a sound  experimental basis and documentation on the program’s effectiveness  should be reviewed by an independent entity capable of evaluating the  supporting scientific evidence.  The report also stated that often scientists  and other experts can help independently assess the scientific evidence on  the effectiveness of a program.  A contributor to the National Research  Council report also stated that no conclusive research has been conducted  to determine if behavior detection can be reliably used on a larger scale,  such as in an airport setting, to identify persons intending to cause harm to  the aviation system.
While TSA and DHS’s Science and Technology (S&T) Directorate officials  agreed that SPOT was deployed before its scientific underpinnings were  fully validated, they stated that no large-scale U.S. or international  operational screening program incorporating behavior- and appearance- based indicators has been rigorously scientifically validated.  These  officials also questioned the findings of the National Research Council  report and stated that the study lacked sufficient information for its  conclusions because it did not consider recent findings from unpublished  DHS, defense, and intelligence community studies.  However, National  Research Council officials stated that an agency should be cautious about  relying on the results of unpublished research that has not been peer  reviewed, such as that generated by DHS and the defense and intelligence  community, and using unpublished work as a basis for proceeding with a  process, method, or program.  Moreover, we have previously reported  that peer review is widely accepted as an important quality control  mechanism that helps prevent the dissemination of potentially erroneous  information.
In addition to the unpublished research, TSA told us that the SPOT  program was based on operational best practices from law enforcement,  defense, and the intelligence communities.  According to TSA officials, the  agency based its choice of SPOT behavior, appearance, and deception  indicators on existing research and training programs.  For example, TSA  cited research on emotions and their behavior indicators by Dr. Paul  Ekman, interviewing and interrogation by Stan Walters, and nonverbal  indicators by Dr. David Givens and Dr. Mark Frank as support for the  choice of several of the behavior indicators.  According to TSA, its  development of the SPOT program was based on related DHS research  and information from the training curricula of other federal agencies, such  as the Federal Transit Administration and the Bureau of Alcohol, Tobacco,  Firearms, and Explosives.
As with the SPOT behavior indicators, TSA told us that it sought input in  creating the SPOT point scoring system from subject matter experts and  from participants in TSA’s SPOT working group, which consisted of law  enforcement officials from agencies such as FBI, DEA, and local law  enforcement officials.  While TSA officials said that they coordinated  with relevant subject matter experts, such as Dr. Ekman, and based the  SPOT scoring system on existing research and training programs, no  validation of the behavior, appearance, and deception indicators was  conducted prior to the deployment of SPOT in November 2006.  According  to TSA officials, they used professional judgment in developing the SPOT  point system and stated that the purpose of developing the scoring system  was to increase the objectivity of the SPOT process.
Dr. Ekman stated that, in his opinion, and after reviewing the scoring  system and observing the program in operation, it was not clear whether  the SPOT behaviors and appearances, and the related point system, could  be used effectively in an airport environment because no credible  validation research on this issue had been conducted.  He noted, for  example, that research is needed to identify how many BDOs are required  to observe a given number of passengers moving at a given rate per day in  an airport environment, or the length of time that such observation can be  conducted before observation fatigue affects the effectiveness of the  personnel.  He commented that observation fatigue is a well-known  phenomenon among workers whose work involves intense observation,  and that it is essential to determine the duration of effective observation  and to ensure consistency and reliability among the personnel carrying out  the observations.
DHS has recognized the need to conduct additional research to  scientifically validate the use of the SPOT behavioral indicators in an  airport environment.  DHS’s S&T Directorate began research in 2007 to  determine if there is a statistically significant correlation between the  SPOT behaviors exhibited by airport passengers and finding airport  passengers with prohibited items (such as weapons), false documents, and  illegal drugs or who pose a potential risk to aviation security.  According  to S&T, this research is expected to be completed in fiscal year 2011 and is  to include three key elements.  First, the study’s purpose is to assess the  reliability of the SPOT program by analyzing TSA’s SPOT database to  determine patterns of BDO scoring to measure consistency across BDOs,  teams, locations, and other variables.  Second, the study aims to compare  the current implementation of SPOT to random passenger screening.   Specifically, according to S&T officials, 130,000 passengers are to be  randomly selected for additional SPOT referral screening.  The study’s  design states that data collected about these passengers will be compared  to data for passengers screened through the normal SPOT process.  S&T  officials expect that the results of this element of the study will provide a  better understanding of how SPOT compares to random selection, as well  as providing a baseline of each indicator present in the traveling public.   Third, the study also aims to utilize live and video data, as available, to  measure SPOT score ratings by BDOs of behaviors exhibited by  passengers against ratings of the same passengers by subject matter  experts.  This element of the study could help determine whether BDOs  are using, or are continuing to use, the SPOT score sheet correctly as time  passes after their initial training.  According to S&T officials, the study is  to form the basis for BDO performance and training requirements.
The S&T Directorate reported some preliminary findings associated with  this research in February 2008.  The Directorate reported that although  some of the existing literature supported the possibility of using  behavioral and physiological cues, the results are not methodologically  strong enough to support standardized applications in an operational  setting.  The preliminary findings also noted that it is not known whether  behavioral and physiological cues linked to deception in planning a hostile  action will be the same or different as those indicators linked to deception  by an individual after they have already engaged in a hostile action.   However, an S&T program director stated that although early literature  can be characterized as methodologically weak, more recent unpublished  research sponsored by DHS, the Department of Defense, and the  intelligence community is promising in that it has demonstrated some  linkages between behavioral and physiological indicators and deception.
In March 2009, the Under Secretary (Acting) for DHS’s S&T Directorate  testified that the Directorate had performed an initial validation of the  behavior indicators used by BDOs.  The Under Secretary stated that this  analysis provided statistically significant support that persons  demonstrating select behavioral indicators are more likely to possess  prohibited items and that behaviors can distinguish deceptive from  nondeceptive individuals.  According to S&T, this validation was the result  of statistical analyses performed by S&T using operational data from the  SPOT program database.  However, we identified weaknesses in TSA’s  process for maintaining these data.  For example, controls over the SPOT  database to help ensure the completeness and accuracy of the data were  missing.  Specifically, the SPOT database did not have computerized edit  checks built into the system to review the format, existence, and  reasonableness of data.  For example, we found that discrepancies existed  between the number of passengers arrested by local law enforcement at  the screening checkpoints and the number of screened passengers  recorded as arrested.  In another example, we found that the total number  of LEO referrals differed from the number of passenger records with  information on the reasons for LEO referral.  Internal control standards  state that controls should be installed at an application’s interfaces with  other systems to ensure that all inputs are received and are valid and that  outputs are correct and properly distributed.  TSA officials explained  these issues as data anomalies and planned to change instructions to staff  entering data to reduce these problems.  Although TSA is taking steps to  update the SPOT database, which are discussed later in this report, the  data used by S&T to conduct its preliminary validation of related  behaviors lacked such controls.  In addition, BDOs could not input all  behaviors observed in the SPOT database because the database limits  entry to eight behaviors, six signs of deception, and four types of  prohibited items per passenger referred for additional screening.  Because  of these data-related issues, meaningful analyses could not be conducted  to determine if there is an association between certain behaviors and the  likelihood that a person displaying certain behaviors would be referred to  a LEO or whether any behavior or combination of behaviors could be used  to distinguish deceptive from nondeceptive individuals.  As a result, TSA  lacks assurance that the SPOT data can be used effectively to determine  that the person poses a risk to aviation security.  S&T has recognized  weaknesses in the procedures for collecting data on passengers screened  by SPOT and plans to more systematically collect data during its study by,  for example, requiring BDOs to record more complete and accurate  information related to a passenger referral immediately following  resolution.
The S&T study is an important step to determine whether SPOT is more  effective at identifying passengers who may be threats to the aviation  system than random screening.  However, S&T’s current research plan is  not designed to fully validate whether behavior detection and appearances  can be effectively used to reliably identify individuals in an airport  terminal environment who pose a risk to the aviation system.  For  example, research on other issues, such as determining the number of  individuals needed to observe a given number of passengers moving at a  given rate per day in an airport environment or the duration that such  observation can be conducted by BDOs before observation fatigue affects  effectiveness, could provide additional information on the extent to which  SPOT can be effectively implemented in airports.  In another example, Dr.  Ekman told us that additional research could help determine the need for  periodic refresher training since no research has yet determined whether  behavior detection is easily forgotten or can be potentially degraded with  time or lack of use.  While S&T officials agree on the need to validate the  science of behavior detection programs, they told us that some of these  other issues could be examined in the future but are not part of the  current plan due to time and budgetary constraints.  According to S&T,  some additional analysis is underway to inform the current BDO selection  process.  This analysis is intended to provide information on the  knowledge, skills, abilities, and other characteristics of successful BDOs.   Since the analysis is scheduled for completion in May 2010, it remains  unclear to what extent the findings will help to validate the science related  to SPOT.  While we recognize the potential benefits of these efforts, we  believe that an assessment by an independent panel of experts of the  planned methodology of DHS’s study could help DHS assess the costs and  benefits associated with a more comprehensive methodology designed to  fully validate the science related to SPOT.  Our prior work has  recommended the use of such independent panels for comprehensive,  objective reviews of complex issues.  In addition, according to the  National Research Council, an independent panel could provide an  objective assessment of the methodology and findings of DHS’s study to  better ensure that SPOT is based on validated science.  Thus, an  independent panel of experts could help DHS develop a comprehensive  methodology to determine if the SPOT program is based on valid scientific  principles that can be effectively applied in an airport environment for  counterterrorism purposes.
section_title: SPOT Was Deployed Nationwide on Basis of Threat, but Without a Comprehensive Risk Assessment
paragraphs: According to DHS’s National Infrastructure Protection Plan (NIPP), risk  assessments are to be documented, reproducible (so that others can verify  the results), defensible (technically sound and free of significant errors),  and complete.  The NIPP states that comprehensive risk assessments are  necessary for determining which assets or systems face the highest risk,  for prioritizing risk mitigation efforts and the allocation of resources, and  for effectively measuring how security programs reduce risks.  For a risk  assessment to be considered complete, the NIPP states that it must  specifically assess threat, vulnerability, and consequence; after these  three components have been assessed, they are to be combined to  produce a risk estimate.
According to TSA, SPOT was deployed to TSA-regulated airports on the  basis of threat information in TSA’s Current Airport Threat Assessment  list.  TSA deployed SPOT to 161 of 457 TSA-regulated airports.  TSA  officials told us that this selective deployment creates unpredictability for  persons seeking to cause harm to the aviation system because they would  not know which airports had BDO teams and because BDOs are  occasionally sent out to the smaller airports that do not have BDOs on a  permanent basis.  Although TSA’s selective deployment of SPOT was  based on threat information, TSA did not conduct vulnerability and  consequence assessments to inform the deployment of BDOs.  As a result,  it could not combine the results to conduct a comprehensive risk  assessment to inform the deployment of BDOs to those airports with the  highest risks.
TSA officials told us that while they have not completed a comprehensive  risk assessment for airport security, they have prepared and are currently  reviewing a draft of a comprehensive, scenario-based Aviation Modal Risk  Assessment—known as the AMRA—which is to serve as a comprehensive  risk assessment for aviation security.  According to TSA officials, the  AMRA is to address all three elements of risk for domestic commercial  aviation, general aviation, and air cargo.  Although TSA planned to  release the AMRA in February 2008, it now expects to finalize the AMRA in  2010.  According to TSA, the AMRA may help provide information for the  prioritization of BDO deployment within airports, but could not provide  specifics on how it would do so.  Further, TSA officials noted that  information from AMRA would inform BDO deployment in conjunction  with other TSA priorities not related to SPOT.  Since the AMRA is not yet  complete, it is not clear whether it will provide the risk analysis— including assessments of vulnerability and consequence—needed to  inform TSA’s decisions and planning for any revisions or future  deployment of SPOT.  If AMRA lacks information relevant to the  deployment of SPOT and further research determines that SPOT has a  scientifically validated basis for using behavior detection for  counterterrorism purposes in the airport environment, then conducting a  comprehensive risk assessment of airports could strengthen TSA’s ability  to establish priorities and make cost-effective resource decisions  regarding the deployment of BDOs to those airports deemed to have the  highest priority risks.
section_title: TSA Deployed SPOT Nationwide Without Conducting a Cost-Benefit Analysis but Such an Analysis Could Help Inform Program Decisions Moving Forward
paragraphs: DHS and other federal guidance recommend conducting a cost-benefit  analysis before implementing new programs to avoid unnecessary costs  and identify the best way to achieve goals at the lowest costs among  potential alternatives.  Our prior work has also supported the use of cost- benefit analyses during retrospective reviews to validate the agency’s  original assumptions regarding costs and benefits.  In addition, the DHS  February 2006 Cost-Benefit Analysis Guidebook and OMB guidance both  recommend the use of cost-benefit analysis, both in the planning stage for  a program, and when significant milestones or financial options are to be  assessed.  The DHS Guidebook states that a cost-benefit analysis is  designed to identify optimal financial solutions among competing  alternatives.  OMB guidance also identifies cost-benefit analysis as one of  the key principles to be considered when making capital expenditures,  that expected benefits of proposed actions should be explained, and that a  baseline should be identified discussing costs and benefits in comparison  with clearly defined alternatives.  DHS’s 2006 and 2009 NIPPs also state  that priority is to be given to those protective measures that provide the  greatest mitigation of risk for the resources that are available.  The DHS  NIPPs add that effective protective programs seek to use resources  efficiently by focusing on actions that offer the greatest mitigation of risk  for any given expenditure.  In addition, measuring cost effectiveness of  SPOT was a key TSA goal in an October 2005 version of the SPOT strategic  plan.
Although the DHS and OMB guidance recommend that a cost-benefit  analysis be conducted prior to deploying a program nationwide—and  potentially incurring substantial costs—TSA did not conduct such an  analysis of SPOT to inform its pilot testing prior to full-scale nationwide  deployment.  In early 2003, TSA began conducting a pilot test of the SPOT  program at Boston Logan airport to better understand the benefits of the  program.  According to Boston Logan’s Federal Security Director, the  primary purpose of this pilot test was to understand the potential of the  program, not to validate its success.  TSA officials stated that the  program had several benefits, one of which was its “negligible cost.”   However, TSA did not analyze the pilot test results to determine if SPOT  was more cost effective than other alternatives, such as random screening  of passengers.  In October 2004, TSA implemented additional pilot  programs in Providence, Rhode Island and Portland, Maine with the goal  of providing Federal Security Directors with an additional layer of security  to identify high-risk passengers for additional screening using behavior  detection techniques.  TSA concluded that the pilot program was  successful and cited several security benefits of these pilots.  For example,  TSA personnel in Providence identified two individuals in possession of  illegal drugs, who were then arrested.  Law enforcement also arrested  another individual referred to them for providing a fraudulent passport.  In  another example, BDOs in Portland discovered a passenger with multiple  passports and a hidden luggage compartment.  The passenger was  interviewed by LEOs and later released.
TSA determined that these initial pilot tests at three airports were  successful without comparing pilot test data to other possible security  alternatives.  For example, the results of random screening of passengers  at the pilot airports could have provided TSA with objective baseline data.   Specifically, these data could have been compared to data collected during  the SPOT pilots to determine if SPOT was more effective than random  screening in detecting passengers who pose a potential risk to aviation  security.  TSA concluded that the pilot tests were successful because pilot  airports were able to easily incorporate SPOT into their security program,  train personnel in SPOT, and implement procedures for an additional layer  of security according to TSA.
TSA conducted additional pilot tests at the Minneapolis-St. Paul,  Minnesota and Bangor, Maine airports in October 2005.  TSA also deployed  the program to nine additional airports in response to TSA’s holiday  preparedness plan in December 2005 to further operationally test the  program.  Senior SPOT program officials explained that TSA did not  conduct an analysis of the pilot testing because the program was in its  infancy and officials were focused on deploying SPOT to additional  airports.  Since that time, TSA has not conducted a cost-benefit analysis,  which could help the agency establish the value of the program relative to  other layers of aviation security.  Moreover, a cost-benefit analysis could  also be useful considering recent program growth.  For example, from  fiscal year 2007 through fiscal year 2009, TSA allotted about $383 million  for SPOT.  During this period, SPOT’s share of TSA’s total screening  operations budget increased from 1 percent to 5 percent.  The conference  report accompanying the fiscal year 2010 DHS appropriations act  designates $212 million of the appropriated aviation security funding for  the SPOT program.  A cost-benefit analysis could have provided TSA  management with analysis on whether this allocation was a prudent  investment, as well as whether this level of investment in SPOT is  appropriate.  Figure 3 shows the growth in the budget and personnel  numbers for SPOT from fiscal years 2007 through 2010.
section_title: SPOT’s Strategic Plan Could be Strengthened by Addressing Key Characteristics of a Successful Strategy
paragraphs: Our previous work, and the Government Performance and Results Act,  set forth several key elements of a strategic plan.  Such plans can guide  agencies in planning and implementing an effective government program.   Table 1 summarizes the desirable characteristics of an effective strategic  plan, as identified in our prior work.  In April 2009, we reported that these  characteristics are the starting point for developing a strategic plan.
TSA officials at Boston Logan airport told us that they completed the first  strategic plan for SPOT in 2006.  The strategic plan was last updated in  March 2007.  The March 2007 plan includes some of the desirable  characteristics described above, such as an overall purpose.  However,  incorporating additional characteristics of an effective strategic plan could  enhance the plan’s usefulness in program management and resource  allocation decisions to effectively manage the deployment of SPOT if TSA  determines that the program has a scientifically valid basis.  TSA officials  stated that they believed the plan was sufficiently comprehensive to  develop a national program, such as SPOT.  However, these officials told  us that the plan was not updated after TSA expanded the program in 2008  and 2009.  They also stated that the program’s focus remained on  deploying SPOT to additional airports.  Our assessment of the extent to  which the SPOT strategic plan addresses these characteristics is presented  below.
Purpose, scope, and methodology:  The SPOT strategic plan addresses  why the plan was developed (i.e., purpose) and the scope of its coverage.   Specifically, the plan describes a strategy to utilize behavior detection  screening as an additional layer of security.  The plan also notes that the  primary focus is to expand SPOT in the aviation environment while also  developing a capability to deploy BDOs to support security efforts in all  modes of transportation.  However, the plan does not discuss the process  by which it was developed (i.e., methodology).  According to TSA, officials  responsible for developing the plan received input from relevant  stakeholders at Boston Logan airport and TSA headquarters.  We believe  incorporating the methodology into the plan could make the document  more useful to TSA and other organizations, such as local law  enforcement, responsible for implementing the plan.
Problem definition and risk assessment: The plan addresses the  particular threat it is directed towards.  Specifically, the plan describes the  need to implement SPOT to counter terrorist activities, improve security,  and incorporate additional layers of protection within aviation security.   However, the plan does not incorporate risk assessment information to  identify priorities or guide program implementation because TSA has not  conducted a comprehensive risk assessment related to the deployment of  SPOT.  Using available risk assessment information to inform the  development of a strategic plan would help ensure that clear priorities are  established and focused on the areas of greatest need.  Specifically,  incorporating the results of a risk assessment in the program’s strategic  plan could help inform TSA’s decisions such as whether to deploy SPOT to  additional TSA-regulated airports, to shift SPOT teams from one airport to  another, or to remove SPOT at airports where the benefit of addressing the  risk does not outweigh the costs, as well as to identify and communicate  the risks to aviation security if SPOT was not deployed to all TSA- regulated airports.
Goals, subordinate objectives, activities, and performance  measures:  The plan outlines several goals, objectives, and activities for  the SPOT program to achieve.  For example, the plan outlines a goal to  develop multimodal partnerships, including at the local level, to support  SPOT.  An associated objective for this goal includes identifying and  fostering advocates within each mode of transportation by developing  transportation, intelligence, and law enforcement working groups with  relevant officials to share information and foster cooperation.  The plan  also includes a goal to develop and implement performance measures for  SPOT.  However, the plan did not include performance measures for  SPOT.  Incorporating performance measures into the plan could help TSA  officials measure progress in implementing the plan’s goals, objectives,  and activities.
Resources, investments, and risk management: The plan does not  identify the costs and resources needed to achieve program objectives  discussed in the plan.  Incorporating information about cost and resources  would facilitate TSA’s ability to allocate resources across programs  according to priorities and constraints, track costs and performance, and  shift such investments and resources as appropriate.
Organizational roles, responsibilities, and coordination:  The SPOT  program relies on a close partnership with law enforcement officers at  airports.  TSA provides briefings to law enforcement on the SPOT  program, and TSA officials conduct outreach efforts to local law  enforcement as needed.  The SPOT SOP guidance and SPOT training  include guidance about ensuring that LEOs receive complete and accurate  information about each SPOT referral.  However, while the strategic plan  identifies TSA officials and offices as responsible parties for implementing  the strategic plan, it does not provide guidance on how to effectively link  the roles, responsibilities, and capabilities of federal, state, and local  officials providing program support.  Moreover, although SPOT SOP  guidance discusses the need for BDOs to coordinate with other TSA  personnel, such as TSOs and TDCs, TSA does not identify their roles and  responsibilities in regards to the SPOT program in the program’s strategic  plan.  Integrating these elements into the strategic plan could help to  clarify the relationships between these various implementing parties,  which would thereby increase accountability and improve the  effectiveness of implementation.
Integration and implementation: The SPOT strategic plan does not  discuss how its scope complements, expands upon, or overlaps with other  related strategic documents.  For example, TSA’s April 2008 Office of  Security Operations Organizational Business Plan for Fiscal Year 2010  describes how its goals—including those for SPOT—relate to DHS and  TSA strategic goals.  However, TSA does not link goals in the SPOT  strategic plan with other related strategic documents, such as the Aviation  Implementation Plan of DHS’s Transportation Systems Sector-Specific  Plan and the Passenger Checkpoint Screening Program Strategic Plan.   By linking goals in its SPOT strategic plan to other TSA efforts, TSA could  better ensure that the program’s objectives are integrated with other TSA  security programs and that resources are used effectively by minimizing  any unnecessary duplication with these other actions.
section_title: More Fully and Consistently Utilizing Available Information Technology Could Enhance TSA’s Ability to Identify Threats to the Aviation System
paragraphs: Inconsistencies in the use of available information technology to aid in the  collection and recording of data on passengers by BDOs during referrals to  LEOs, lack of guidance on, or a mechanism for, BDOs to request the TSA’s  Transportation Security Operations Center to run the names of passengers  exhibiting suspicious behaviors against law enforcement and intelligence   databases, and the Center’s not checking all of the databases available to  it—have limited TSA’s ability to identify potential terrorist threats to the  aviation system.  Among other information, these databases include  terrorism-related watch lists.
subsections: section_title: Systematic Collection of Data on Passengers Identified Through the SPOT Program Could be Improved to Better Identify Activity Potentially Harmful to the Aviation System
paragraphs: TSA is not fully utilizing the resources it has available to systematically  collect the information obtained by BDOs on passengers whose behaviors  and appearances resulted in either SPOT referral screening, or in a referral  to LEOs, and who thus may pose a risk to the aviation system.  TSA’s July  2008 Privacy Impact Assessment on the TSA Transportation Security  Operations Center, and its August 2008 Privacy Impact Assessment on  SPOT, state that information may be obtained by BDOs to check an  individual’s identity against intelligence, terrorist, and law enforcement  databases and to permit intelligence analysts to conduct trend analysis.
The August 2008 SPOT Privacy Impact Assessment states that information  about a passenger who has exceeded the SPOT behavior threshold,  leading to LEO referral, may be collected and entered into DHS’s  Transportation Information Sharing System.  According to the SPOT  Privacy Impact Assessment, information collected may be submitted to the  Transportation Information Sharing System database for analysis, and,  through it to other linked intelligence databases and the intelligence  analysts who study them, to detect, deter, and defeat a criminal or terrorist  act in the transportation domain before it occurs.  The SPOT Privacy  Impact Assessment notes that terrorist acts that threaten transportation  security are most vulnerable in the planning stages and that the timely  passage of SPOT referral information may assist in identifying such efforts  before they become operational.  A June 2008 Transportation Information  Sharing System Privacy Impact Assessment similarly states that one goal is  to use the system data to find trends and patterns that may indicate  preoperational terrorist or criminal activity—that is, to “connect the dots”  about a planned terrorist attack or criminal enterprise.  Information in  TSA’s Transportation Information Sharing System is primarily activity or  behavioral information but may also contain personal information  regarding the individuals identified by the BDO through SPOT. According  to TSA, BDOs do not analyze the data obtained during referrals; if they  have the appropriate training, they may enter the data by computer into  the Transportation Information Sharing System, where they can be  analyzed by intelligence analysts.  Other appropriately trained and  officially designated TSA officials, such as Federal Security Directors, may  also enter data into the system.
According to TSA, a 2008 pilot program it conducted that involved BDOs  entering data into the Transportation Information Sharing System  database was “invaluable,” in part because over 40 referrals have since  been passed on to other LEO organizations for further investigation, most  of which came from BDO input.  A February 2006 TSA memorandum  describes the Transportation Information Sharing System as “a critical  element in the success of SPOT” because it provides the necessary  platform for the reporting of information obtained as a result of SPOT  referrals.  TSA noted that through the use of the Transportation  Information Sharing System, two different BDO teams had separately  identified and selected the “same extremist” for secondary questioning.   TSA officials also told us about an incident in which an individual sought  to board an aircraft with a handgun on two separate occasions, at two  different airports.  Although the handgun was detected both times, the  individual was released after providing what seemed to be a credible  explanation.  After the second incident, however, intelligence analysts  who reviewed the system information saw that this individual had tried  twice in 2 weeks to bring a weapon onto an aircraft.  A LEO was  dispatched to the person’s home, and an arrest was made.  Without the  data inputted into the system both times, no pattern would have been  detected by the analysts, according to TSA.  Although the pilot program  illustrated the benefits of BDOs entering data into the system, access to  the system was not expanded to all SPOT airports in 2008 or 2009.
Internal control standards call for management to develop policies,  procedures, and techniques to help enforce management directives. TSA  does not provide official guidance on how or when BDOs or other TSA  personnel should enter data into the Transportation Information Sharing  System or which data should be entered.  Official guidance on what data  should be entered into the system on passengers could better position TSA  personnel to be able to consistently collect information to facilitate  synthesis and analysis in “connecting the dots” with regard to persons who  may pose a threat to the aviation system.
On March 18, 2010, TSA officials told us that TSA recognizes the value of  recording SPOT incidents for the purposes of intelligence gathering.  As a  result, TSA decided that certain data would be entered into the  Transportation Information Sharing System, and would, in turn, be  analyzed as a way to potentially “connect the dots” with other  transportation security incidents.
TSA officials said that the Federal Security Director at each SPOT airport  has been given the discretion to decide which personnel should have  access to the Transportation Information Sharing System.  However, TSA  has not developed a plan detailing how many personnel would have access  to the system, or when they would have access at SPOT airports.  TSA  officials said that training is currently being provided to personnel  responsible for using the system at all SPOT airports although they did not  provide information on the number being trained.
Standard practices for defining, designing, and executing programs  include developing a road map, or program plan, to establish an order for  executing specific projects needed to obtain defined programmatic results  within a specified time frame.  However, TSA stated that it has not  developed a schedule or milestones by which database access will be  deployed to SPOT airports, or a date by which access at all SPOT airports  will be completed.  Setting milestones for expanding Transportation  Information Sharing System access to all SPOT airports, and setting a date  by which the expansion will be completed, could better position TSA to  identify threats to the aviation system that may otherwise go undetected  and help TSA track its progress in expanding Transportation Information  Sharing System access as management intended.
section_title: Guidance on and a Mechanism for Running Names of Referred Passengers Through the Databases Available to the Transportation Security Operations Center Could Help Improve SPOT Practices
paragraphs: Internal control standards state that policies, procedures, techniques, and  other mechanisms are essential to help ensure that actions are taken to  address program risks. The current process makes the BDOs dependent  on the LEOs with regard to the timeliness that LEOs respond to BDO calls  for service, as well as with regard to whether the LEOs choose to question  the passengers referred to them or conduct a background check.  Our  analysis of the SPOT referral database found a wide variation in the  percent of times that LEOs responded to calls for service at SPOT  airports.  Moreover, if a local LEO decides to run a background check on  a passenger referred to them, they would be accessing the FBI’s NCIC and  not other intelligence and law enforcement databases.
Although LEOs may not always respond to calls for service, question  passengers, or check passenger names against databases available to TSA,  TSA has not developed a mechanism allowing BDOs to send information  to the Transportation Security Operations Center about passengers whose  behavior indicates that they may be a possible threat to aviation security.     According to TSA’s July 2008 Transportation Security Operations Center  Privacy Impact Assessment, passenger information may be submitted to  the Transportation Security Operations Center to ascertain, as quickly as  possible, the individual’s identity, whether they are already the subject of a  terrorist or criminal investigation, or to analyze suspicious behavior that  may signal some form of preoperational surveillance or activity.
Our survey of Federal Security Directors at SPOT airports found a notable  inconsistency in the rates at which BDOs at different airports contacted  the Transportation Security Operations Center.  Developing additional  guidance in the SPOT operating procedures could help improve  consistency in the extent to which BDOs utilize Transportation Security  Operations Center resources.  Given the range of responses we received  from SPOT airports about whether the BDOs contact the Transportation  Security Operations Center to verify passenger identities and run their  names against terrorist and intelligence databases and the inconsistencies  identified related to LEO responses to BDO requests for service,  developing a standard mechanism and providing BDOs with additional  guidance could help TSA achieve greater consistency in the SPOT process.   Such a mechanism would provide designated TSA officials with a means of  verifying passenger identities and help them determine whether a  passenger was the subject of a terrorist or criminal investigation and thus  posed a risk to aviation security.
Standards for internal control state that effectively using available  resources, including key information databases, is one element of  functioning internal controls. In this connection, it is widely recognized  among intelligence entities and police forces that a capability to “run”  names against databases that contain criminal and other records is a  potentially powerful tool to both identify those with outstanding warrants  and to help discover an ongoing criminal or security-related incident.   Additionally, TSA recommended in an April 2008 Organizational Business  plan for its Office of Security Operations that the SPOT program should  establish a mechanism and policy for allowing real-time checks of federal  records for individuals whose behavior indicates they may be a threat to  security.  The Office of Security Operations plan also states that BDOs  should communicate the data to U.S. intelligence centers, with the  purpose of permitting rapid communication of this information to local  LEOs to take action.  However, TSA officials told us that because of safety  concerns, the Transportation Security Operations Center does not provide  information from database checks directly to BDOs because BDOs are not  LEOs, are unarmed, and do not have the training needed to deal with  potentially violent persons.  If the mechanism discussed in the Office of  Security Operations business plan were implemented, it would allow the  Transportation Security Operations Center to use BDO information to  conduct real-time record checks of passengers and communicate the  results to LEOs for action.  Such a mechanism could increase the chances  to detect ongoing criminal or terror plans.
section_title: TSA’s Transportation Security Operations Center Does Not Use All Database Resources When Contacted
paragraphs: The final report of the National Commission on Terrorist Attacks Upon the  United States (the “9/11 Commission Report”) recommends that in  carrying out its goal of protecting aviation, TSA should utilize the larger  set of information maintained by the federal government, that is, the entire  Terrorist Screening Database—the U.S. government’s consolidated watch  list that contains information on known or suspected international and  domestic terrorists—as well as other government databases, such as  intelligence or law enforcement databases.  However, the Transportation  Security Operations Center is not using all the resources at its disposal to  support BDOs in verifying potential risks to the aviation system.  This  reduces the opportunities to “connect the dots” that would increase the  chances of detecting terrorist attacks in their planning stage, which the  SPOT Privacy Impact Assessment states is when they are the most  vulnerable.
According to TSA, the Transportation Security Operations Center has  access to multiple law enforcement and intelligence databases that can be  used to verify the identity of airline passengers; these include among  others:   1.  the Selectee list, which identifies persons who must undergo enhanced  screening at the checkpoint prior to boarding;   2.  the No-Fly list, which lists persons prohibited from boarding aircraft;  and   3.  the Terrorist Identity Datamark Environment terrorist list.
TSA stated that the Transportation Security Operations Center checks  passenger names submitted to it against these three databases if the  passenger has been referred by a BDO to a LEO, but has not been arrested.   Of the three databases that the Transportation Security Operations Center  is to check in the case of a referral, passengers would have already been  screened against two—the Selectee and No-Fly lists—in accordance with  TSA passenger prescreening procedures when purchasing a ticket.  The  third database checked—the Terrorist Identity Datamark Environment— tracks terrorists but not persons wanted for other crimes.  The FBI’s NCIC  information system would contain names of such persons, but is not  among the three databases checked for nonarrest referrals.  If the  passenger has been arrested, the Transportation Security Operations  Center will run the passenger’s name against the additional law  enforcement and intelligence databases available to it.
In addition, TSA told us that the Operations Center does not have direct  electronic access to the Terrorist Screening Database and must call the  FBI’s Terrorist Screening Center to provide it with a name to verify.  TSA  stated that this is done if a passenger’s identity could not be verified using  the Operations Center databases.  In effect, if a passenger has been  referred to a LEO, but not arrested, the Operations Center is to check the  three databases shown above to verify the passenger’s identity.  If a  passenger has been arrested, but the three databases do not list the  person, the Center can check the additional databases available to it.  If  none of these databases can verify the person’s identity, the Operations  Center can contact the Terrorist Screening Center by telephone to request  further screening. screening database (the watch list) and to provide for the use of watch-list records during  security-related screening processes.  See GAO-08-136T, Aviation Security: TSA Is  Enhancing Its Oversight of Air Carrier Efforts to Screen Passengers against Terrorist  Watch-List Records, but Expects Ultimate Solution to Be Implementation of Secure Flight  (Washington, D.C.: Sept. 9, 2008).
For passengers who have risen to the level of a LEO referral at an airport  checkpoint, having the Transportation Security Operations Center  consistently check their names against all the databases available to it  could potentially help TSA identify threats to the aviation system and aid  in “connecting the dots.”  TSA indicated that there are no obstacles to  rapidly checking all databases rather than the three listed.  We did not  analyze the extent to which the law enforcement and intelligence  databases available to TSA may contain overlapping information.
section_title: TSA Lacks Program Effectiveness Measures for SPOT but Is Taking Steps to Improve Evaluation Capabilities
paragraphs: TSA has established some performance measures by tracking SPOT  referral and arrest data, but lacks the measures needed to evaluate the  effectiveness of the SPOT program and, as a result, has not been able to  fully assess SPOT’s contribution to improving aviation security.  TSA  emphasized the difficulty of developing performance measures for  deterrence-based programs, but stated that it is developing additional  measures to quantify the effectiveness of the program.  The SPOT program  uses teams to assess BDO proficiency, provide individual and team  guidance, and address issues related to the interaction of BDOs with TSA  checkpoint personnel.  However, TSA does not systematically track the  teams’ recommendations or the frequency of the teams’ airport visits.  TSA  states that it is working to address these issues and plans to do so by the  end of fiscal year 2010.
subsections: section_title: TSA Has Taken Action to Collect Data for Some Performance Measures, but Work Remains to Assess Progress Towards Achieving Strategic Goals
paragraphs: TSA agreed that the SPOT program lacked sufficient performance  measures in the past, but stated that it has some performance measures in  place including tracking data on passengers referred for additional  screening and the resolution of this screening, such as if prohibited items  were found or if law enforcement arrested the passenger and the reason  for the arrest.  TSA is also working to improve its evaluation capabilities to  better assess the effectiveness of the program.  DHS’s NIPP, internal  controls standards, and our previous work on program assessment state  that performance metrics and associated program evaluations are needed  to determine if a program works and to identify adjustments that may  improve its results.  Moreover, standard practices in program  management for defining, designing, and executing programs include  developing a road map, or program plan, to establish an order for  executing specific projects needed to obtain defined programmatic results  within a specified time frame.  Congress also needs information on  whether and in what respects a program is working well or poorly to  support its oversight of agencies and their budgets; and agencies’  stakeholders need performance information to accurately judge program  effectiveness.  For example, in the Senate Appropriations Committee  report accompanying the fiscal year 2010 DHS appropriations bill, the  committee noted that while TSA has dramatically increased the size and  scope of SPOT, resources were not tied to specific program goals and  objectives.  In addition, the conference report accompanying the fiscal  year 2010 DHS appropriations act requires TSA to report to Congress,  within 60 days of enactment, on the effectiveness of the program in  meeting its goals and objectives, among other things.  This report was  completed on March 15, 2010.
Although TSA tracks data related to SPOT activities including prohibited  items, law enforcement arrests related to SPOT referrals, and reasons for  the arrests (output measures), it has not yet developed measures to gauge  SPOT’s effectiveness in meeting TSA strategic goals (outcome measures),  such as identifying individuals who may pose a threat to the transportation  system.  OMB encourages the use of outcome measures because they are  more meaningful than output measures, which tend to be more process- oriented or means to an end.  For example, TSA’s Assistant General  Manager for the Office of Operation Process and Performance Metrics  told us that SPOT staffing levels are currently used as one performance  metric.  The official said that since the SPOT is an added layer of security,  additional SPOT staffing would add to security effectiveness.  While  staffing levels may help gauge how fast the program is growing, they do  not measure the effectiveness in meeting strategic goals.
Similarly, TSA also cited the number of prohibited items discovered by  BDOs in SPOT metrics reports as a measure of program success.   However, TSA told us that possession of a prohibited item is often an  oversight and not an intentional act; moreover, other checkpoint screening  layers are intended to find such items, such as the TSOs and the property  screening equipment.  TSA also cited measures of BDO job performance  as some of the existing measures of program effectiveness, but noted that  these are “pass/fail” assessments of individual BDOs, rather than overall  program measures.
TSA notes that one purpose of the SPOT program is to deter terrorists, but  that proving that it has succeeded at deterring terrorists is difficult  because the lack of data has presented challenges for the SPOT program  office when developing performance measures.  We agree that developing  performance measures, especially outcome measures, for programs with a  deterrent focus is difficult.  Nevertheless, such measures are an important  tool to communicate what a program has accomplished and provide  information for budget decisions.  TSA uses proxy measures—indirect  measures or indicators that approximate or represent the direct  measure—to address deterrence, other security goals, or a combination of  both.  For example, TSA tracks the number of prohibited items found and  individuals arrested as a result of SPOT referrals.  According to OMB,  proxy measures are to be correlated to an improved security outcome, and  the program should be able to demonstrate—such as through the use of  modeling—how the proxies tie to the eventual outcome.In using a  variety of proxy measures, failure in any one of the identified measures  could provide an indication on the overall risk to security.  However,  developing a plan that includes objectives, milestones, and time frames to  develop outcome-based performance measures could better position TSA  in assessing the effectiveness of the SPOT program.
With regard to more readily quantifiable output performance measures,  such as the number of referrals by BDOs, or the ratio of arrests to  referrals, TSA was limited in its ability to analyze the data related to these  measures.  The SPOT database includes information on all passengers  referred by BDOs for additional SPOT screening including the behaviors of  the passengers that led to the additional screening, as well as the  resolution of the screening process (e.g., no further action taken, law  enforcement notification, law enforcement investigation, arrested, and  reason for arrest).  However, TSA reported that any analysis of the data  had to be done manually.
In March 2010, TSA migrated the SPOT referral data to its Performance  Management Information System, allowing for more statistical and other  analyses.  According to TSA, migrating the SPOT referral database will  enhance the SPOT program’s analytic capabilities.  For example, TSA  stated that it would be able to conduct trend analyses, better segregate  data, and create specific reports for certain data.  This includes better  tracking of performance data at specific airports, analyzing by categories  of airports (threat or geographic location), and tracking the performance  data of individual BDOs, such as number of referrals, number of arrests,  arrest to referral ratios, and other analyses.  However, since these changes  to the database were not complete at the time of our audit, we could not  assess whether the problems we identified with the database had been  corrected.
subsections: section_title: Over 4 Years, SPOT Resulted in About 1,100 Arrests Out of Almost 14,000 Referrals to Law Enforcement
paragraphs: The SPOT referral database records the total number of SPOT referrals  since May 29, 2004, how many were resolved, how many passengers BDOs  referred to LEOs, the recorded reasons for the referral, and how many  referrals led to arrests, among other things.  As shown in figure 4, we  analyzed the SPOT referral data for the period May 29, 2004, to August 31,  2008.
Figure 4 shows that approximately 2 billion passengers boarded aircraft at  SPOT airports from May 29, 2004, through August 31, 2008.  Of these,  151,943 (less than 1/100th of 1 percent) were sent to SPOT referral  screening, and of these, 14,104 (9.3 percent) were then referred to LEOs.   These LEO referrals resulted in 1,083 arrests, or 7.6 percent of those  referred, and less than 1 percent of all SPOT referrals (0.7 percent of  151,943).
We also analyzed the reasons for arrests resulting from SPOT referrals, for  the May 29, 2004, through August 31, 2008, period.  Table 2 shows, in  descending order, the reasons for the arrests.
While SPOT personnel did not determine a specific reason for arrest for  128 cases categorized as “other” or 16 other cases categorized as “no  reason given,” our analysis of the SPOT database found that a specific  reason for arrest could have been determined for these cases by using the  LEO resolution notes included in the database.  For example, we identified  43 additional arrests related to fraudulent documents, illegal aliens, and  suspect documents, among others.  The remaining 101 arrests originally  characterized as “other” or “no reason given” included arrests for reasons  such as intoxication, unruly behavior, theft, domestic violence, and  possession of prohibited items.  Many of the arrests resulting from BDO  referrals would typically fall under the jurisdiction of various local, state,  and federal agencies and are not directly related to threats to aviation  security.  For example, the 427 individuals arrested as illegal aliens, and  the 166 arrested for possession of fraudulent documents, are subject to the  enforcement responsibilities shared by U.S. Immigration and Customs  Enforcement (ICE) and CBP.  Although outstanding warrants and the  possession of fraudulent or suspect documents could be associated with a  terrorist threat, TSA officials did not identify any direct links to terrorism  or any threat to the aviation system in any of these cases.
According to TSA, anecdotal examples of BDO actions at airports show  the value added by SPOT to securing the aviation system.  However,  because the SPOT program has not been scientifically validated, it cannot  be determined if the anecdotal results cited by TSA were better than if  passengers had been pulled aside at random, rather than as a consequence  of being identified for further screening by BDOs.  Some of the incidents  cited by TSA include the following.
A BDO referred two passengers who were traveling together to referral  screening due to suspicious behavior.  During secondary screening, one  passenger presented fraudulent travel documents.  The other could not  produce any documentation of his citizenship and it was determined he  was in the United States illegally.  ICE responded and interviewed both  passengers.  ICE stated one passenger was also in possession of  $10,000 dollars which alarmed positive for narcotics when swept by a  K-9 team.  ICE arrested one passenger on a federal charge of  possession of fraudulent identification documents and entry without  inspection.  ICE stated charges are still pending for the possession of  $10,000.  The second passenger was charged with a federal charge of  entry without inspection.
A BDO referred a passenger to referral screening for exhibiting  suspicious behavior.  Port Authority of Portland (Oregon) Police  responded and interviewed the passenger who did not give a statement.   LEOs conducted an NCIC check which revealed that there was an  outstanding warrant for the failure to appear for a theft charge.  LEOs  arrested the passenger on a state charge for an outstanding warrant for  the failure to appear for theft.
A BDO referred a passenger for referral screening due to suspicious  behavior.  During the referral, the passenger admitted that he was  unlawfully present in the United States.  The Orlando (Florida) Police  Department and CBP responded and interviewed the passenger who  stated he had $100,000 in his checked baggage, which was confirmed  by CBP.  The passenger was arrested on a federal charge of illegal  entry.
Because these are anecdotal examples, they cannot be used to reliably  generalize about the SPOT program’s overall effectiveness or success rate.   Our analysis of the SPOT referral database found that the referral data do  not indicate if any of the passengers sent to referral screening, or those  arrested by LEOs after being referred to them, intended to harm the  aircraft, its passengers, or other components of the aviation system.   Additionally, SPOT officials told us that it is not known if the SPOT  program has ever resulted in the arrest of anyone who is a terrorist, or  who was planning to engage in terrorist-related activity.
section_title: Reviewing Airport Video Recordings of Individuals Later Arrested or Who Pleaded Guilty for Engaging in Terrorism- Related Activities Could Help TSA Better Identify Terrorist- Related Behaviors
paragraphs: Studying airport video recordings of the behaviors exhibited by persons  waiting in line and moving through airport checkpoints and who were later  charged with or pleaded guilty to terrorism-related offenses could provide  insights about behaviors that may be common among terrorists or could  demonstrate that terrorists do not generally display any identifying  behaviors.  TSA officials agreed that examining video recordings of  individuals who were later charged with or pleaded guilty to terrorism- related offenses, as they used the aviation system to travel to overseas  locations allegedly to receive terrorist training or to execute attacks, may  help inform the SPOT program’s identification of behavioral indicators. In  addition, such images could help determine if BDOs are looking for the  right behaviors or seeing the behaviors they have been trained to observe.
Using CBP and Department of Justice information, we examined the travel  of key individuals allegedly involved in six terrorist plots that have been  uncovered by law enforcement agencies.  We determined that at least 16  of the individuals allegedly involved in these plots moved through 8  different airports where the SPOT program had been implemented.  Six  of the 8 airports were among the 10 highest risk airports, as rated by TSA  in its Current Airport Threat Assessment.  In total, these individuals moved  through SPOT airports on at least 23 different occasions.  For example,  according to Department of Justice documents, in December 2007 an  individual who later pleaded guilty to providing material support to Somali  terrorists boarded a plane at the Minneapolis-Saint Paul International  Airport en route to Somalia to join terrorists there and engage in jihad.   Similarly, in August 2008 an individual who later pleaded guilty to  providing material support to Al-Qaeda boarded a plane at Newark Liberty  International Airport en route to Pakistan to receive terrorist training to  support his efforts to attack the New York subway system.
Our survey of Federal Security Directors at 161 SPOT airports indicated  most checkpoints at SPOT airports have surveillance cameras installed.   As we previously reported, best practices for project management call for  conducting feasibility studies to assess issues related to technical and  economic feasibility, among other things.  In addition, Standards for  Internal Control state that effectively using available resources is one  element of functioning internal controls.  TSA may be able to utilize the  installed video infrastructure at the nation’s airports to study the behavior  of persons who were later charged with or pleaded guilty to terrorism- related offenses, and determine whether BDOs saw the behaviors.  The  Director of Special Operations in TSA’s Office of Inspection told us that  video recordings could be used as a teaching tool to show the BDOs which  behaviors or activities they did or did not observe. In addition, TSA  indicated that although the airports may have cameras at the security  screening checkpoints, the cameras are not owned by TSA, and in many  cases, they are not accessible to TSA.  However, TSA officials lack  information on the scope of these potential limitations because prior to  our work TSA did not have information on the number of checkpoints  equipped with video surveillance.  We obtained this information as part of  our survey of Federal Security Directors at SPOT airports.  While TSA  officials noted several possible limitations of the use of the existing video  surveillance equipment, these images provide TSA a means of acquiring  information about terrorist behaviors in the checkpoint environment that  is not available elsewhere.  If current research determines that the SPOT  program has a scientifically validated basis for using behavior detection  for counterterrorism purposes in the airport environment, then conducting  a study to determine the feasibility of using images captured by video  cameras could better position TSA in identifying behaviors to observe.
section_title: Standardization Teams Assess BDO Proficiency in SPOT Activities and Provide Guidance and Mentoring to BDOs
paragraphs: TSA sends standardization teams to SPOT airports on a periodic basis to  conduct activities related to quality control.  Teams observe SPOT  operations at an airport for several days, working side by side with the  BDOs, on multiple shifts, observing their performance, offering guidance,  and providing training when required.  According to TSA, the purpose of a  standardization team visit is to provide operational support to the BDOs,  which includes additional training, mentoring, and guidance to help  maintain a successful SPOT program.
The standardization teams are comprised of at least two G-Band, or  Expert BDOs who have received an additional week of training on SPOT  behaviors and mentoring skills.  SPOT officials stated that the SPOT  program uses its standardization teams to assess overall BDO proficiency  by observing BDOs, reviewing SPOT score sheet data, and other relevant  data.  Standardization teams may also provide a Behavior Observation and  Analysis review class to refresh BDOs if the team determines that such a  class is needed.  The SPOT program director also said that the  standardization teams aim to monitor the airport’s compliance with the  SPOT program’s Standard Operating Procedures.  As part of this  mentoring approach, the standardization teams provide individual and  team guidance to the BDOs, offer assistance in program management, and  cover issues related to the interaction of BDOs with other TSA checkpoint  personnel.
TSA reported to us that it does not systematically track the  standardization teams’ recommendations or the frequency of the teams’  airport visits.  Standards for Internal Control state that programs should  have controls in place to assess the quality of performance over time and  ensure that the findings of audits and other reviews are promptly resolved.   Managers are to (1) promptly evaluate findings from audits and other  reviews, including those showing deficiencies and recommendations  reported by auditors and others who evaluate agencies’ operations; (2)  determine proper actions in response to findings and recommendations  from audits and reviews; and (3) complete, within established time frames,  all actions that correct or otherwise resolve the matters brought to  management’s attention.  Although the standardization teams may  provide an airport Federal Security Director with recommendations on  how to improve SPOT operations, the SPOT program director stated that  Federal Security Directors are not required to document whether they  have implemented the team recommendations.  TSA officials told us that  standardization teams can follow up on recommendations made during  previous visits.  However, TSA did not track whether corrective actions  were implemented or the frequency of the team’s airport visits to ensure  the implementation of the airport’s SPOT program.  TSA officials stated  that they are currently examining ways to compile data to address this  issue, and expect to have a system in place in fiscal year 2010.
section_title: TSA Developed and Deployed SPOT Training but Further Action Could Enhance Its Effectiveness
paragraphs: Although TSA has taken steps to incorporate all four elements of an  effective training program by planning, designing, implementing, and  evaluating training for BDOs, further action could help enhance the  training’s effectiveness.  TSA initially consulted outside experts for help in  the training’s development, which began as a half-day course and has  grown to include classroom, on-the-job, and advanced training.  TSA also  has efforts underway to improve its training program, such as the  deployment of SPOT recurrent training.  However, TSA evaluations of  SPOT program instructors found mixed quality among them, from 2006  onwards.  Additionally, TSA has ongoing plans to evaluate the SPOT  training for effectiveness, but has not yet developed time frames and  milestones for completing the evaluation.
subsections: section_title: TSA Has Taken Actions to Develop and Deploy SPOT Training
subsections: section_title: TSA’s SPOT Training Evolved Over Time
paragraphs: In 2003, TSA officials at Boston Logan International airport developed the  initial half-day training course for SPOT based on an existing course  developed for the Massachusetts State Police.  Their goal was to take the  behavior detection program designed for law enforcement and apply it to  screeners at airport checkpoints.  According to TSA officials at Boston  Logan, after they recognized that the lecture-style course they originally  designed was not effective, they tasked an instructional system designer  from TSA’s Workplace Performance and Training (the former name of  TSA’s Operational and Technical Training Division) and an industrial  psychologist from the Office of Human Capital to redesign and expand the  course, which was piloted in 2005.  The 2007 SPOT strategic plan included  training objectives for the SPOT program as follows:   reviewing existing behavior observation training providers,    establishing and prioritizing multimodal training and assistance efforts  based on threat assessments and critical infrastructure,    establishing a Center of Excellence for Behavior Detection Program  training that would continually enhance the quantity and quality of  training to selected candidates, and    developing a recurrent training program designed to refresh and hone  skills needed for an effective Behavior Detection Program.
Since that time, the SPOT program implemented, or is in the process of  implementing, some of these objectives.  For example, in 2008, as part of  its effort towards establishing a center for excellence in behavior  detection training (third objective), the SPOT program participated in a  meeting with behavior detection training officials from various DHS  components facilitated by DHS’s Screening Coordination Office to  promote the sharing of information about behavior detection training and  foster future collaboration.  Additionally, the SPOT program worked with  TSA’s Operational and Technical Training Division to create a recurrent  training component for BDOs (fourth objective).  For example, in 2008, the  SPOT program office added a course on detecting microfacial expressions  called Additional Behavior Detection Techniques.  This 3-day course  builds on the behavior detection skills taught in basic training, by teaching  BDOs how to detect microfacial expressions.  After pilot testing, the  course began implementation nationwide in January 2009.
section_title: TSA Consulted with Some Experts on Developing SPOT Training
paragraphs: In developing an effective training program, we previously reported that  consultation with subject matter experts and expert entities is a core  characteristic of the strategic training and development process.  TSA  SPOT program staff told us that they consulted with experts on behavior  detection and observed existing behavior detection courses before  deploying the SPOT training program.  According to SPOT program  officials, a TSA staff member from Boston Logan International Airport  attended other training programs offered by other federal agencies and  private training organizations to inform the design of SPOT training.   TSA officials told us that information from the training courses was used  to help develop the list of behaviors or “stress elevators” for the program,  and that the point system used to identify passengers for referral screening  was based in part on consultations with several subject-matter experts.
TSA documentation also notes that a SPOT working group created in  February 2004 consulted with the FBI’s Behavioral Science Unit.  The  Behavioral Science Unit specializes in developing and facilitating training,  research, and consultation in the behavioral sciences for the FBI, law  enforcement, intelligence, and military communities.  While TSA officials  from Boston Logan told us that the FBI was included in this initial SPOT  working group, these officials agree that coordination with the FBI lapsed  until June 2009 when the SPOT Program Office reengaged with the  Behavioral Science Unit, and held a meeting with the unit at the FBI  Academy in Quantico, Virginia.  Since that meeting, a subject matter  expert from the SPOT Program Office has been invited to be a member of  the Terrorism Research and Analysis Project, which is an ongoing working  group sponsored by the unit.
In July 2008, DHS’s Screening Coordination Office facilitated a  collaborative discussion on behavior detection that included TSA, CBP,  and Secret Service officials to better ensure that components within DHS  share information regarding their efforts in behavior detection and provide  a forum for components to have an informed and collaborative discussion  on current capabilities, best practices, and lessons learned.  According to  TSA, no further contact has occurred between the DHS Behavior  Detection Working Group and the SPOT program.  Thus, the extent to  which the working group’s expertise will be used to refine or augment  SPOT training in the future is not yet clear.
section_title: SPOT Program Office Recently Deployed Recurrent Training
paragraphs: Along with basic and remedial training required by the Aviation and  Transportation Security Act, TSA policy requires its screening force to  regularly complete recurrent (refresher) training.  TSA recognized that  ongoing training of screeners on a frequent basis and effective supervisory  training are critical to maintaining and enhancing skills learned during  basic training.  According to agency officials, TSA is currently working  with DHS S&T to determine the necessary frequency for refresher training  for each training course within the SPOT program.  Furthermore, TSA  plans to place BDOs under TSA’s Performance and Accountability  Standards System (PASS) beginning in fiscal year 2010.  This will include a  recertification module.
In 2008, the SPOT program office began the process for developing  recurrent SPOT training.  Our internal control standards and training  assessment guidance suggest that such refresher training should be  considered integral to an effective training program from the start because  work conditions and environments can be expected to change over time,  and additional or updated training is essential to ensuring that the program  mission continues to be accomplished.  According to the SPOT program  office, the recently deployed recurrent training will be semiannual.  TSA’s  Operational and Technical Training Division initially planned to pilot test  recurrent training in April 2009 followed by full implementation of the  course in approximately May 2009.  Because the Operational and  Technical Training Division focus was shifted to completing the revisions  for the SPOT basic certification course, recurrent training was delayed  until September 2009 when they released the training on TSA’s Online  Learning Center.
section_title: Instructor Evaluations Found Mixed Quality; Issues with Program Management Led to Instructor Retraining
paragraphs: Our previous work on elements of effective training states that instructors  must be both knowledgeable about the subject matter and issues involved,  as well as able to effectively transfer these skills and knowledge to  others.  Moreover, internal control standards state that all personnel  need to possess and maintain a level of competence that allows them to  accomplish their assigned duties.  Management needs to identify  appropriate knowledge and skills needed for various jobs and provide  needed training, as well as to ensure that those teaching the skills are  themselves competent.
TSA conducted internal assessments of SPOT instructors episodically  from 2006 through March 2008.  These assessments involved a few  instructors being rated at a time, and found a wide range of competency  among the instructors.  In January 2009, TSA’s Office of Inspections and  Investigations began an investigation of the SPOT training manager, who  resigned shortly thereafter.  TSA investigators determined that the training  manager and other trainers had created a hostile training environment that  intimidated some trainees.  To address this problem, TSA stated that the  program office reexamined the SPOT training program nationally.  This  included recertifying 47 of 54 SPOT instructors in March 2009, which  included evaluation by TSA’s Office of Human Capital, Quality Assurance  assessors.  Additionally, in July 2009, TSA centralized SPOT training at five  permanent, regional training facilities in Orlando, Florida; Houston, Texas;  Phoenix, Arizona; Denver, Colorado; and Philadelphia, Pennsylvania.   According to the SPOT program director, this will allow the SPOT  program office more oversight over training.  Previously, training was  provided at individual airports.
After the March 2009 recertification training, ratings scores of SPOT  instructors showed less variation than did previous ratings.  We reviewed  the quality assurance instructor evaluations of two SPOT instructors  conducted by TSA’s Office of Human Capital, Training Standards and  Evaluation Branch, and the 167 SPOT program instructor evaluations of 54  SPOT instructors conducted by the SPOT program office and TSA’s  Operational and Technical Training Division since the program started in  October 2006.  After the recertification training, 93 percent of instructors  were rated as exceeding expectations, compared to 30 percent in the 2006  to September 2008 ratings.  Table 3 shows the ratings of instructors for  March 2009 compared to the period of 2006 to September 2008.
In addition to the variation in numeric scores and rating levels for the 2006  to September 2008 period, as shown in table 3, we found substantial  variation in the comments about instructor competency for the same  period.  For example, in 32 out of 74 instructor evaluation forms that we  reviewed where comments were made about the instructor prior to 2009,  the comments ranged from superb to needs more experience as an  instructor, as well as needs more time performing the job as a BDO to be  able to teach others.  In the comments on an instructor who was rated as  “meets expectations,” the instructor was described as having “limited  experience within the SPOT program,” that this was “a major concern,”  and it was recommended that the instructor spend as much time as  possible functioning as a BDO.  In other cases, however, SPOT instructors  were described as competent, solid, and outstanding.  For example, one  instructor who received a rating of “exceeds expectations” was described  as a superb instructor who “is a valued member of the National Training  Team.”  As noted above, following the March 2009 recertification training,  93 percent of the instructors received a rating of “exceeds expectations”  with only 1 percent “needing improvement.”  Of the 94 instructor  evaluations completed in March 2009, 82 contained written comments.  Of  these, multiple SPOT instructors were described as excellent,  knowledgeable, and effective.  For example, an instructor who received a  rating of “exceeds expectations” was noted as demonstrating a high degree  of material knowledge and great presentation skills.  TSA attributed the  increase in instructor ratings to two factors.  The first is low turnover  among SPOT instructors, which allows instructors to hone both their  technical and instructor skills.  The second factor cited by TSA is that TSA  conducted a 2-day instructor refresher training immediately prior to the  evaluations in March 2009. To ensure all instructors were reevaluated  within a specific time frame, evaluations were scheduled and conducted in  a controlled environment.  Instructors knew in advance they were going to  be evaluated and delivered modules of the BDO certification course to  other BDO instructors.
section_title: TSA Has Taken Some Action, but Has Not Evaluated the SPOT Training Program for Effectiveness
paragraphs: We previously reported that evaluation is an integral part of training and  development efforts, and that agencies need to systematically plan for and  evaluate the effectiveness of training and development.  Employing  systematic monitoring and feedback processes can help by catching  potential problems at an early stage, thereby saving valuable time and  resources that a major redesign of training would likely entail.  Similarly,  in 2006, TSA’s Operational and Technical Training Division issued general  evaluation standards for training programs, stating that training programs  should be comprehensively evaluated on a periodic basis to identify  program strengths and weaknesses.  Moreover, standard practices in  program management for defining, designing, and executing programs  include developing a road map, or program plan, to establish an order for  executing specific projects needed to obtain defined programmatic results  within a specified time frame.
The former SPOT training manager told us that the SPOT program  internally evaluates the effectiveness of SPOT training through the job  knowledge tests that BDO candidates must pass following the classroom  portion of the training and the SPOT Proficiency/On-the-Job Training  Checklist following the on-the-job portion of the training.  Furthermore,  the former training manager told us that TSA knows that the SPOT training  is effective because BDOs are able to recognize behaviors at the  checkpoint, and because of BDOs’ demonstrated ability to identify  criminals—such as drug couriers or people with outstanding arrest  warrants—through the screening process.
Although TSA has not conducted a comprehensive analysis of the  effectiveness of the SPOT training program, TSA’s Office of Human  Capital, Training Standards and Evaluation Branch conducted training  evaluations to assess how students use what they were taught in the SPOT  basic training course.  Specifically, from July through September 2008, the  Training Standards and Evaluation Branch conducted evaluations at 5 of  the 161 airports where the SPOT program is currently operating.  Based on  BDO feedback at the 5 airports, the Training Standards and Evaluation  Branch’s final report contained a series of recommendations for improving  the SPOT training program.  These recommendations and TSA’s actions to  address them are summarized in table 4.
Additionally, in conjunction with S&T, TSA conducted a training  effectiveness evaluation on the Additional Behavior Detection Techniques  course, which showed a statistically significant increase in knowledge and  skills following completion of the course.
S&T is currently conducting a BDO job task analysis, which may be used  to evaluate and update the SPOT training curriculum.  Following the  completion of the job task analysis—anticipated in mid-May 2010—TSA’s  Operational and Technical Training Division intends to conduct an in- depth training gap analysis, which will take approximately 2 months to  complete.  Following completion of the training gap analysis, the agency  will develop project plans, including milestones for future development  efforts, to address any training concerns.  However, to date, the agency  does not have an evaluation plan including time frames and milestones for  completion.  According to the Operational and Technical Training  Division, TSA will conduct periodic evaluations as the BDO position  evolves.  By conducting a comprehensive evaluation of the effectiveness of  its training program, TSA will be in a better position to determine if BDOs  are being taught the knowledge and skills they need to perform their job.   Furthermore, by developing milestones and time frames for conducting  such evaluations systematically, as well as on a periodic basis, TSA could  help ensure that the SPOT training program is evaluated in accordance  with its directives to help ensure that the program continues to provide  BDOs with the necessary tools required to carry out their responsibilities.
section_title: Conclusions
paragraphs: TSA developed the SPOT program in the wake of September 11, 2001, in  an effort to respond quickly to potential threats to aviation security by  identifying individuals who may pose a threat to aviation security,  including terrorists planning or executing an attack who were not likely to  be identified by TSA’s other screening security measures.  Because TSA  did not ensure that SPOT’s underlying methodology and work methods  were scientifically validated prior to its nationwide deployment, an  independent panel of experts could help determine whether a scientific  foundation exists for the way in which the SPOT program uses behavior  detection analysis for counterterrorism purposes in the aviation  environment.
With approximately $5.2 billion devoted to screening passengers and their  property in fiscal year 2009, it is important that TSA provides effective  stewardship of taxpayer funds ensuring a return on investment for each  layer of its security system.  As one layer of aviation security, the SPOT  program has an estimated projected cost of about $1.2 billion over the  next 5 years if the administration’s requested funding of $232 million for  fiscal year 2011 remains at this level.  The nation’s constrained fiscal  environment makes it imperative that careful choices be made regarding  which investments to pursue and which to discontinue.  If an independent  expert panel determines that DHS’s study is sufficiently comprehensive to  determine whether the SPOT program is based on valid scientific  principles that can be effectively applied in an airport environment for  counterterrorism purposes, then conducting a comprehensive risk  assessment including threat, vulnerability, and consequence could  strengthen TSA’s ability in making resource allocation decisions and  prioritizing its risk mitigation efforts.  Moreover, conducting a cost-benefit  analysis could help TSA determine whether SPOT provides benefits  greater than or equal to other security alternatives and whether its level of  investment in the SPOT program is appropriate.  Revising its strategic plan  for SPOT to incorporate risk assessment information, cost and resource  analysis, and other essential components could enhance the plan’s  usefulness to TSA in making program management and resource  allocation decisions to effectively manage the deployment of SPOT.
Providing guidance on how to use TSA’s resources for running passenger  names against intelligence and criminal databases available to the  Transportation Security Operations Center and helping DHS to connect  disparate pieces of information using the Transportation Information  Sharing System and other related intelligence and crime database and data  sources could better inform DHS and TSA regarding the identity and  background of certain individuals and thereby enhance aviation security.   In addition, implementing the steps called for in the TSA Office of  Strategic Operations plan to provide BDOs with a real-time mechanism to  verify passenger identities and backgrounds via TSA’s Transportation  Security Operations Center could strengthen their ability to rapidly verify  the identity and background of passengers who have caused concern, and  increase the likelihood of detecting and disrupting potential terrorists  intending to cause harm to the aviation system.  Additionally, developing  outcome-oriented performance measures, making improvements to the  SPOT database, and studying the feasibility of utilizing video recordings of  individuals as they transited checkpoints and who were later charged with  or pleaded guilty to terrorism-related offenses, could help TSA evaluate  the SPOT program, identify potential vulnerabilities, and assess the  effectiveness of its BDOs.  Further, developing a plan for systematic and  periodic evaluation of the training provided to BDOs along with time  frames and milestones for its completion could help ensure that the SPOT  training program is evaluated in accordance with its directives to help  ensure that the program continues to provide BDOs with the necessary  tools required to carry out their responsibilities.
section_title: Recommendations for Executive Action
paragraphs: To help ensure that SPOT is based on valid scientific principles that can be  effectively applied in an airport environment, we recommend that the  Secretary of Homeland Security convene an independent panel of experts  to review the methodology of the DHS S&T Directorate study on the SPOT  program to determine whether the study’s methodology is sufficiently  comprehensive to validate the SPOT program.  This assessment should  include appropriate input from other federal agencies with expertise in  behavior detection and relevant subject matter experts.
If this research determines that the SPOT program has a scientifically  validated basis for using behavior detection for counterterrorism purposes  in the airport environment, then we recommend that the TSA  Administrator take the following four actions:    Conduct a comprehensive risk assessment to include threat,  vulnerability, and consequence of airports nationwide to determine the  effective deployment of SPOT if TSA’s ongoing Aviation Modal Risk  Assessment lacks this information.
Perform a cost-benefit analysis of the SPOT program, including a  comparison of the SPOT program with other security screening  programs, such as random screening, or already existing security  measures.
Revise and implement the SPOT strategic plan by incorporating risk  assessment information, identifying cost and resources, linking it to  other related TSA strategic documents, describing how SPOT is  integrated and implemented with TSA’s other layers of aviation  security, and providing guidance on how to effectively link the roles,  responsibilities, and capabilities of federal, state, and local officials  providing program support.
Study the feasibility of using airport checkpoint-surveillance video  recordings of individuals transiting checkpoints who were later  charged with or pleaded guilty to terrorism-related offenses to enhance  understanding of terrorist behaviors in the airport checkpoint  environment.
Concurrent with the DHS S&T Directorate study of SPOT, and an  independent panel assessment of the soundness of the methodology of the  S&T study, we recommend that the TSA Administrator take the following  six actions to ensure the program’s effective implementation:    To provide additional assurance that TSA utilizes available resources to  support the goals of deterring, detecting, and preventing security  threats to the aviation system, TSA should:    Provide guidance in the SPOT Standard Operating Procedures or  other TSA directive to BDOs, or other TSA personnel, on inputting  data into the Transportation Information Sharing System and set  milestones and a time frame for deploying Transportation  Information Sharing System access to SPOT airports so that TSA  and intelligence community entities have information from all SPOT  LEO referrals readily available to assist in “connecting the dots” and  identifying potential terror plots.  Implement the steps called for in the TSA Office of Security  Operations Business plan to develop a standardized process for  allowing BDOs or other designated airport officials to send  information to TSA’s Transportation Security Operations Center  about passengers whose behavior indicates that they may pose a  threat to security, and provide guidance on how designated TSA  officials are to receive information back from the Transportation  Security Operations Center.
Direct the TSA Transportation Security Operations Center to utilize  all of the law enforcement and intelligence databases available to it  when running passenger names, for passengers who have risen to  the level of a LEO referral.
To better measure the effectiveness of the program and evaluate the  performance of BDOs, TSA should:    Establish a plan that includes objectives, milestones, and time  frames to develop outcome-oriented performance measures to help  refine the current methods used by Behavior Detection Officers for  identifying individuals who may pose a risk to the aviation system.
Establish controls to help ensure completeness, accuracy,  authorization, and validity of data collected during SPOT screening.
To help ensure that TSA provides BDOs with the knowledge and skills  needed to perform their duties, TSA should:    Establish time frames and milestones for its plan to systematically  conduct evaluations of the SPOT training program on a periodic  basis.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of our report to DHS and TSA on March 19, 2010, for  review and comment.  On May 3, 2010, DHS provided written comments,  which are reprinted in appendix II.  In commenting on our report, DHS  stated that it concurred with 10 of our recommendations and identified  actions taken, planned, or under way to implement them.  However, the  actions DHS reported it plans to take and has underway do not fully  address the intent of our first recommendation.  DHS also concurred in  principle with an eleventh recommendation stating that it had convened a  working group to determine the feasibility of implementing it.  DHS  commented on the scientific basis underlying SPOT and on two statements  in our report that it believed were inaccurate—specifically, DHS disagreed  with our reliance on a 2008 National Research Council report published  under the auspices of the National Academy of Sciences on issues related  to behavior detection, and second, on issues related to unpublished  research they had cited as a partial validation of some aspects of the SPOT  program.  Finally, DHS commented on our conclusion regarding the use  of the SPOT referral data.
Regarding our first recommendation that DHS convene an independent  panel of experts to review the methodology of DHS’s Science and  Technology Directorate (S&T) study on SPOT, and to include appropriate  input from other federal agencies with relevant expertise, DHS concurred  and stated the current process includes an independent review of the  program that will include input from other federal agencies and relevant  experts.  Although DHS has contracted with the American Institutes for  Research to conduct its study, it remains unclear who will oversee this  review and whether they are sufficiently independent from the current  research process.  DHS’s response also does not describe how the review  currently planned is designed to determine whether the study’s  methodology is sufficiently comprehensive to validate the SPOT program.   As we noted in our report, research on other issues, such as determining  the number of individuals needed to observe a given number of passengers  moving at a given rate per day in an airport environment or the duration  that such observation can be conducted by BDOs before observation  fatigue affects effectiveness, could provide additional information on the  extent to which SPOT can be effectively implemented in airports.  Dr. Paul  Ekman, a leading research scientist in the field of behavior detection, told  us that additional research could help determine the need for periodic  refresher training since no research has yet determined whether behavior  detection is easily forgotten or can be potentially degraded with time or  lack of use.  Thus, questions exist as to whether behavior detection  principles can be reliably and effectively used for counterterrorism  purposes in airport settings to identify individuals who may pose a risk to  the aviation system.  To help ensure an objective assessment of the study’s  methodology and findings, DHS could benefit from convening an  independent panel of experts from outside DHS to determine whether the  study’s methodology is sufficiently comprehensive to validate the SPOT  program.
DHS also concurred with our second recommendation to conduct a  comprehensive risk assessment to determine the effective deployment of  SPOT.  DHS stated that TSA’s Aviation Modal Risk Assessment is designed  to evaluate overall transportation security risk, not deployment strategies.   However, DHS noted that TSA is in the process of conducting an initial  risk analysis using its risk management analysis tool and plans to update  this analysis in the future.  However, it is not clear from DHS’s comments  how this analysis will incorporate an assessment of TSA’s deployment  strategy for SPOT.
DHS also concurred with our third recommendation to perform a cost- benefit analysis of SPOT.  DHS noted that TSA is developing an initial cost- benefit analysis and that the flexibility of behavior detection officers  already suggests that behavior detection is cost-effective.  However, it is  not clear from DHS’s comments whether its cost-benefit analysis will  include a comparison of the SPOT program with other security screening  programs, such as random screening, or already existing security  measures as we recommended.  Completing its cost-benefit analysis and  comparing it to other screening programs should help establish whether  the SPOT program is cost-effective compared to other layers of security.
With regard to our fourth recommendation to revise and implement the  SPOT strategic plan using risk assessment information, DHS concurred  and noted that analysis facilitated by the risk management analysis tool  will allow the program to revise the SPOT strategic plan to incorporate the  elements identified in our recommendation.
DHS also concurred with our fifth recommendation to study the feasibility  of using airport checkpoint-surveillance video recordings to enhance its  understanding of terrorist behaviors.  DHS noted that TSA agrees this  could be a useful tool and is working with DHS’s S&T Directorate to utilize  video case studies of terrorists, if possible.  These cases studies could help  TSA determine what behaviors had been demonstrated by these persons  convicted of terrorist-related offenses who went through SPOT airports,  and what could be learned from the observed behaviors.
DHS concurred with our sixth recommendation that TSA provide guidance  in the SPOT SOP or other directives to BDOs, or other TSA personnel, on  how to input data into the Transportation Information Sharing System  database.  DHS stated that the SPOT SOP is undergoing revision, and that  the revised version will provide guidance directing the input of BDO data  into the Transportation Information Sharing System.  DHS anticipates  release of the updated SPOT SOP in fiscal year 2010.  DHS also agreed that  TSA should set milestones and a time frame for deploying Transportation  Information Sharing System access to SPOT airports so that TSA and  intelligence community entities have information from all SPOT LEO  referrals readily available to assist in “connecting the dots” and identifying  potential terror plots.  DHS stated that TSA is currently drafting a plan to  include milestones and a time frame for deploying System access to all  SPOT airports.
DHS concurred with our seventh recommendation to develop a  standardized process to allow BDOs or other designated airport officials to  send information to TSA’s Transportation Security Operations Center  about passengers whose behavior indicates they may pose a threat to  security, and to provide guidance on how designated TSA officials are to  receive information back from the Center.  DHS stated that TSA has  convened a working group to address this recommendation.  Moreover,  TSA is developing a system and procedure for sending and receiving  information from the Center and stated that it anticipates having a system  in place later in fiscal year 2010.
DHS concurred in principle with regard to our eighth recommendation  that the Transportation Security Operations Center utilize all of the  databases available to it when conducting checks on passengers who rise  to the level of a LEO referral against intelligence and criminal databases.   DHS stated that TSA has convened a working group to address this  recommendation.  According to DHS, this group will conduct a study  during fiscal year 2010 to determine the feasibility of fully implementing  this recommendation.  As such, the study is to review the various  authorities, permissions, and limitations of each of the databases or  systems cited in our report.  DHS stated that access to some of the  systems, requires more justification than a BDO referral.  Further,  according to DHS, because some of the databases or systems contain  classified information, TSA will also need to adopt a communication  strategy to transmit the passenger information between the BDO and  Transportation Security Operations Center.  DHS stated that TSA will  work on a process to collect the passenger information, verify the  passenger’s identity, through checks of databases, and analyze that  information to determine if the passenger is the subject of an investigation  and may pose a risk to aviation security.
With regard to our ninth recommendation to establish a plan with  objectives, milestones, and time frames to develop outcome-oriented  performance measures for BDOs, DHS concurred and stated that TSA  intends to consult with experts to develop outcome-oriented performance  measures.
DHS also concurred with our tenth recommendation to establish controls  for SPOT data.  DHS noted that TSA established additional controls as part  of the SPOT database migration to TSA’s Performance Management  Information System and is exploring an additional technology solution to  reduce possible errors.  As noted in our report, since these changes to the  database were not complete at the time of our audit, we could not assess  whether the problems we identified with the database had been corrected.
Regarding our eleventh recommendation to establish time frames and  milestones to systematically evaluate the SPOT training program on a  periodic basis, DHS concurred and stated that TSA intends to develop  such a plan following completion of DHS’s S&T Directorate’s BDO Job  Task Analysis, and TSA’s training gap analysis, which identifies gaps in the  training curriculum.
DHS also commented on the scientific basis underlying SPOT.   Specifically, DHS stated that decades of scientific research has shown the  SPOT behaviors to be “universal in their manifestation.”  However,  according to DHS, its S&T Directorate is examining the extent to which  behavior indicators are appropriate for screening purposes and lead to  appropriate and correct security decisions.  DHS also commented that the  results of this work, which is currently underway, will establish a scientific  basis of the extent to which the SPOT program instruments and methods  are valid.  Thus, DHS’s comments suggest that additional research is  needed to determine whether these behaviors can be used in an airport  environment for screening passengers to identify threats to the aviation  system.
Moreover, DHS took issue with our use of a report from the National  Research Council of the National Academy of Sciences stating that we  improperly relied upon this report.  We disagree.  DHS questioned the  findings of the National Research Council report and stated that it lacked  sufficient information for its conclusions because it principally focused on  privacy as it relates to data mining and behavioral surveillance and was  not intended to represent an exhaustive or definitive review of the  research or operational literature on behavioral screening, including  recent unpublished DHS, defense, and intelligence community studies.   DHS also stated that the National Research Council report did not study  the SPOT program and that the researchers did not conduct interviews  with SPOT personnel.
As we noted in our report,  although the National Research Council report  addresses broader issues related to privacy and data mining, a senior  Council official—and one of the authors of the study—stated that the  committee included behavior detection as a focus because any behavior  detection program could have privacy implications.  This official added  that the primary objective of the report was to develop a framework for  sound decision making for programs, such as SPOT, and help ensure a  sound scientific and legal basis.  According to this official, the National  Academy of Sciences’ Committee on Technical and Privacy Dimensions of  Information for Terrorism Prevention and Other National Goals—which  had oversight of the report—was briefed on the SPOT program as part of  the study.  The Committee also conducted meetings with three experts in  behavior detection as part of their research.  During the course of our  review, we interviewed three Committee members responsible for  developing the report’s findings, as well as four other behavior detection  experts, including the three who participated in the National Research  Council study.  Our discussions with these experts corroborated the  report’s findings.  Thus, we believe that our use of the Council report was  an appropriate and a necessary part of our review.
However, the National Research Council report was only one of many  sources that we analyzed with regard to the science of behavioral and  physiological screening, and its applicability to an airport environment.  As  we noted in the description of our methodology, our study included  interviews with officials from DHS as well as several of its components  and other U.S. government agencies—each of which use elements of  behavior detection in their daily work.  We also interviewed El Al airline  officials, a former director of security at Israel’s Ben-Gurion airport, and  seven nationally recognized experts in behavior detection as part of our  review.  Moreover, as we explained in the discussion of our scope and  methodology, we conducted a survey about the SPOT program of all 118  Federal Security Directors for all SPOT airports, and conducted site visits  to 15 SPOT airports.  In addition, we analyzed the SPOT referral database,  to the extent the data permitted, covering a 4-year period and the results  from 2 billion passengers passing through SPOT airports.  Moreover, we  attended both the basic and advanced training courses in behavior  detection provided by TSA to BDOs, in order to better understand how the  program is carried out.  Therefore, our analysis of the program was not  derived from or based on a single study by the National Research Council  as DHS suggested, but rather is based on all of the information we  gathered and synthesized from multiple, diverse, expert sources, each of  which provided different perspectives about the program, as well as about  behavior detection in general.
DHS also disagreed with the accuracy of a statement included in our  report that noted DHS S&T could not provide us with specific contacts  related to sources of information for certain research it cited as support  for the SPOT program.  In its comments, DHS stated that it had provided  us with all requested documents that represent DHS’s S&T Directorate- sponsored research.  We agree.  However, DHS did not provide us with  contact information for the sources of unpublished studies by the  Department of Defense and other intelligence community studies that DHS  S&T had cited as support for the SPOT program.  Without such  information, we are unable to verify the contents of these unpublished  studies.
Finally, DHS stated that while we were unable to use the SPOT referral  data to assess whether any behavior or combination of SPOT behaviors  could be used to reliably predict the final outcome of an incident involving  the use of SPOT, it was able to analyze the SPOT referral database  successfully after working with TSA to verify scores assigned to different  indicators.  Our concern with the data did not involve the question of  whether some behaviors were entered erroneously, nor whether errors in  coding were excessive or non-random.  Rather, we were concerned with  whether the data on behaviors were complete.  Specifically, it cannot be  determined from the SPOT referral database whether all behaviors  observed were included for each referred passenger by each BDO or  whether only the behaviors that were sufficient for a LEO referral were  recorded into the database.  It is not possible to determine from the  database if the number of observed behaviors entered for a given  passenger was the total number of observed behaviors, or whether  additional behaviors were observed.  A rigorous analysis of the relative  effects of the different behaviors on the outcomes of the use of SPOT  would require each BDO to record, for each of the observable behaviors,  whether it was or was not observed.
TSA also provided technical comments that we incorporated as  appropriate.
We will send copies of this report to the Secretary of Homeland Security;  the TSA Administrator (Acting); and interested congressional committees  as appropriate.  The report will also be available at no charge on the GAO  Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-4379 or lords@gao.gov.  Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page   of this report.  Key contributors to this report are acknowledged in  appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine the extent to which the Transportation Security  Administration (TSA) determined whether the Screening of Passengers By  Observation Techniques (SPOT) program had a scientifically-validated  basis for identifying passengers before deploying it, we reviewed literature  on behavior analysis by subject matter experts, interviewed seven experts  in behavior analysis, interviewed other federal agencies and entities about  how they use behavior detection techniques, and analyzed relevant reports  and books on the topic.  These included a 2008 study by the National  Research Council of the National Academy of Sciences that has a  discussion regarding deception and behavioral surveillance, as well as  other issues related to behavioral analysis.  We interviewed Dr. Herbert S.  Lin, who was a primary author of the report, as well as Dr. Robert W.  Levenson, and Dr. Stephen E. Fienberg, both members of the Academy  committee that oversaw the report, about the report’s findings with regard  to behavior detection, and the extent to which behavior detection in a  complex environment, such as an airport terminal, has been validated with  regard to its effectiveness in identifying persons who may be a risk to  aviation security.  Other behavior detection experts we consulted were Dr.  Paul Ekman; Dr. Mark Frank; Dr. David Givens; Dr. David Matsumoto;  and Mr. Rafi Ron, former director of security at Israel’s Ben-Gurion  Airport.  Dr. Ekman, Dr. Frank, and Mr. Ron provided expert advice for the  National Research Council study.  Dr. Givens was identified by TSA as  having been their principal source for the nonverbal behavior indicators  used by the SPOT program.  We also interviewed Dr. Lawrence M. Wein,  an expert in emergency responses to terror attacks and mathematical  models in operations management.  In addition, we interviewed officials  from the Department of Homeland Security’s (DHS) Science and  Technology (S&T) Directorate regarding their ongoing research into  behavior detection.  Although the views of these experts cannot be  generalized across all experts in behavior analysis because we selected  individuals based on their publications on behavioral analysis or related  topics, their recognized accomplishments and expertise, and, in some  cases, TSA’s use of their work or expertise to design and review the SPOT  program’s behaviors, they provided us with an overall understanding of  the fundamentals of behavior analysis, and how it could be applied.
To determine the basis for TSA’s strategy to develop and deploy SPOT and  evaluate to what extent SPOT was informed by a cost-benefit analysis and  a strategic plan, we reviewed program documentation, including briefings  prepared by the SPOT program office during the course of developing and  fielding SPOT, two versions of a strategic plan for SPOT, and the 2009  SPOT standard operating procedures guidance.  We compared the plans  and analyses used by TSA to develop and implement SPOT to criteria on  how to develop and implement programs in DHS’s 2006 Cost Benefit  Analysis Guidebook, as well as to Office of Management and Budget  guidance on the utility of cost-benefit analyses in program  implementation.  We also analyzed the development of SPOT in light of  the standards and criteria cited in DHS’s 2006 National Infrastructure  Protection Plan.  We met with relevant TSA officials to discuss these  issues.  To assess whether DHS developed an effective strategic plan for  SPOT prior to implementing the program, we interviewed TSA officials  involved in development of the SPOT strategic plan.  We analyzed whether  the SPOT plan incorporated the desirable characteristics of an effective  strategic plan as identified by previous GAO work on what strategic plans  should include to be considered effective, such as a risk assessment, cost  and resources analysis, and a means for collaboration with other key  entities.  We also examined it in light of the requirements of the  Government Performance and Results Act of 1993, which specifies the  elements of strategic plans for government programs.  We assessed  whether the SPOT strategic plan was followed by TSA.  As part of our  analysis of the planning for SPOT before it was implemented on a  nationwide basis, we reviewed TSA documentation related to the  development and pilot testing of SPOT, such as a TSA white paper on  SPOT, and interviewed key program officials from both headquarters and  field offices.
We also interviewed cognizant officials from other U.S. government  agencies and agency entities that utilize behavior detection in their work,  including U.S. Customs and Border Protection (CBP), the U.S. Secret  Service, the TSA’s Federal Air Marshal Service (FAMS) component, and  the Federal Bureau of Investigation (FBI).  We sought their views on the  utility of various behavior detection methods, their experience with  practicing behavior detection, and asked them about the extent to which  TSA had consulted with them in developing and implementing the SPOT  program.
To better understand how SPOT incorporated expertise about the use of  behavior detection in an airport setting, we interviewed officials from  Israel’s El Al Airlines, which is cited by TSA as having provided part of the  basis of the SPOT program.  We asked about El Al’s methods to ensure the  security of its passenger aircraft, and also interviewed a former head of  security at Israel’s Ben-Gurion airport, who has advised TSA on security  issues.  We asked TSA and SPOT program officials about their  consultations with El Al, and about the ways in which they had utilized El  Al’s expertise, as well as about any other entities whose expertise they  may have adopted into SPOT.
To determine the challenges, if any, that emerged during implementation  of the SPOT program, we interviewed headquarters and field personnel  about how the program has utilized the resources available to it to ensure  that it is effective.  These resources included the support of law  enforcement officers (LEOs), to whom passengers are referred by  Behavior Detection Officers (BDOs) for additional questioning.  In  addition, we interviewed SPOT program and TSA officials about the  databases available to them at TSA’s Transportation Security Operations  Center to determine if a suspect passenger is being sought by other U.S.  law enforcement or intelligence entities, and whether there is guidance for  BDOs on when and how to contact the Transportation Security Operations  Center.  We also asked about whether there is guidance and training for  BDOs on how to access TSA’s Transportation Information Sharing System  database, which is owned by FAMS, and is available through the  Transportation Security Operations Center.  To determine if any  management challenges had emerged related to management controls in  developing and implementing SPOT, we compared TSA’s approach for  implementing and managing the SPOT program with GAO’s Standards for  Internal Control in the Federal Government and with risk management  principles we had previously identified.  Our legal counsel office  reviewed court decisions relevant to the SPOT program.   In addition, we  interviewed attorneys from the American Civil Liberties Union, and  obtained and reviewed TSA’s Privacy Impact Assessments for SPOT, the   Transportation Security Operations Center, and the Transportation  Information Sharing System.  We also met with and discussed relevant  privacy and legal issues with TSA’s Offices of Privacy and Civil Rights/Civil  Liberties.  To obtain data about certain aspects of the SPOT program that  the SPOT program office did not have, we conducted a survey of Federal  Security Directorswhose responsibilities included security at all 161  SPOT airports at the time of our survey.  (Some Federal Security Directors  have responsibility for more than one airport.)  We obtained a 100 percent  response rate.  This survey asked, among other things, about whether  there were cameras at security checkpoints that record the interactions of  Transportation Security Officers (TSO), BDOs, and passengers; if the  airport authority had an agreement with TSA that specifies certain law  enforcement actions during a SPOT referral; and if there was an  agreement, or any other comparable guidance that specified a time limit  for LEOs to come to checkpoints after being called for help by BDOs.
To determine the extent to which TSA has measured SPOT’s effect on  aviation security, we obtained and analyzed the TSA SPOT referral  database, which records all incidents in which BDOs refer passengers to  secondary, more intensive questioning, and which also records all  incidents in which BDOs chose to refer passengers to LEOs.  We found  that the SPOT database was sufficiently reliable to count the number of  arrests resulting from referrals from BDOs to LEOs, for examining the  reasons for each arrest, and for counting the percentage of times that  LEOs responded to BDO calls for service, and the length of time required.   Use of these data required us to resolve apparent contradictions and  anomalies in the database to make the data useable.  Because of data  problems, we were unable to conduct analyses to assess whether any  behavior or combination of behaviors could be used to predict the final  outcome of an incident involving the use of SPOT.  In addition, we  reviewed relevant standardization team reports and observed a  standardization team visit in operation.
In addition, we spoke with BDO managers, Federal Security Directors, and  Assistant Federal Security Directors to determine how BDOs are  evaluated.  To do so, we conducted site visits to 15 commercial airports at  which BDOs and SPOT have been deployed, or almost 10 percent of the  161 airports with SPOT.  We chose these airports taking into account the  following criteria, among others:  (1) each airport had BDOs deployed, and  at each, the SPOT program had been in effect for no less than 3 months;  (2) airports were chosen to provide a variety of sizes, as measured in  annual passenger volume; physical location within the country (northeast,  southwest, central, Pacific Coast, rural, urban); and estimated risk of  terrorist incident, using DHS’s Current Airports Threat Assessment list  (visiting 6 that were in the top 10, and others much lower); (3) BDOs who  are employed by contractors, rather than employed directly by TSA; and  (4) airports with LEOs who were identified to us by TSA as having  received some form of behavior detection training and airports where they  were not known to have received such training.  In addition, we took into  account the location of the airports with regard their proximity to subject  matter experts on behavior detection whom we wished to interview, as  well as the time and cost required to reach certain airports.
At each of the airports we visited, we interviewed cognizant officials,  including the Federal Security Director or Assistant assigned to the  airport, the BDO program manager, one or two BDOs, and one or two  LEOs who have interacted with BDOs.  Since each of these airports differs  in terms of passenger volume, physical size and layout, geographic  location, and potential value as a target for terrorism, among other things,  the results from these visits are not generalizable to other airports.   However, these visits provided helpful insight into the operation of SPOT  at airports.
In addition, to determine if individuals had transited SPOT airports who  were later charged with or pleaded guilty to terrorism-related offenses, we  reviewed information contained in (1) the Treasury Enforcement  Communication System II database maintained by CBP; (2) Department  of Justice information and court documents, including indictments and  related documents; and (3) media accounts of individuals accused of  terrorism-related activities.  We compared information pertaining to these  individuals’ dates of transit to the dates when SPOT was deployed to the  various airports identified in the Treasury Enforcement Communication  System and Justice Department data to determine if SPOT had been  deployed at a given airport when the transits occurred.  Further, we used  our survey of Federal Security Directors at SPOT airports to determine the  extent to which video surveillance cameras are present at checkpoints.
To assess the extent that SPOT training incorporates the attributes of an  effective training program, we had training experts at TSA headquarters  complete a training assessment tool that we developed using our prior  work for assessing training courses and curricula.  To address training- related issues, including to understand better how other entities train their  employees in behavior detection, and what their curricula include, we  conducted site visits to the Secret Service, FAMS, CBP, and the FBI, and  also interviewed nongovernmental experts on behavior detection (our  selection of these experts is discussed above).  As part of our assessment  of SPOT training, we attended the basic SPOT training course given to  BDOs, as well as the advanced SPOT course on behavior detection.  We  interviewed BDOs and BDO managers about the SPOT training, as well as  officials of El Al airlines, with regard to how El Al trains and tests its  personnel who utilize behavior recognition and analysis as part of their  assessment of El Al passengers.
We conducted this performance audit from May 2008 through May 2010, in  accordance with generally accepted government auditing standards.   Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives.  We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: DHS Comments
section_title: Appendix III: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, David M. Bruno, Assistant  Director, and Jonathan R. Tumin, managed this assignment.  Ryan  Consaul, Jeff C. Jensen, Kevin Remondini, and Julie E. Silvers made  significant contributions to the work.  Arthur James, Jr., Amanda Miller,  and Douglas Sloane assisted with design, methodology, and data analysis.   Chris Dionis assisted with issues related to training.  Katherine Davis and  Debra Sebastian provided assistance in report preparation; Tracey King  and Tom Lombardi provided legal support; and Pille Anvelt and Barbara  Hills developed the report graphics.