Explosives Detection Canines: TSA Has Taken Steps to Analyze Canine Team Data and Assess the Effectiveness of Passenger Screening Canines
Released: 2014-06-24T00:00:00

Letter

Background

NEDCTP’s mission is to deter and detect the introduction of explosive devices into the transportation system. As of June 2014, NEDCTP has deployed 802 of 985 canine teams for which it is able to fund across the transportation system.type for which funding is available, as well as describes their roles, responsibilities, and costs to TSA. There are four types of LEO teams: aviation, mass transit, maritime, and multimodal, and three types of TSI teams: air cargo, multimodal, and PSC.

TSA Has Taken Steps to Analyze Canine Team Data to Identify Program Trends

Since our January 2013 report, TSA has taken steps to analyze key data on the performance of its canine teams to better identify program trends, as we recommended. In January 2013, we reported that TSA collected and used key canine program data in its Canine Website System (CWS), a central management database, but it could better analyze these data to identify program trends. Table 2 highlights some of the key data elements included in the CWS.

In January 2013, we found that NEDCTP was using CWS data to track and monitor canine teams’ performance. Specifically, field canine coordinators reviewed CWS data to determine how many training and utilization minutes canine teams conducted on a monthly basis. NEDCTP management used CWS data to determine, for example, how many canine teams were certified in detecting explosive odors, as well as the number of teams that passed short notice assessments. However, in our January 2013 report, we also found that TSA had not fully analyzed the performance data it collected in CWS to identify program trends and areas that were working well or in need of corrective action. For example:

Training minutes: TSA tracked the number of training minutes canine teams conducted on a monthly basis, as well as the types of explosives and search areas used when training, to ensure teams maintained their proficiency in detecting explosive training aids. However, we found that TSA did not analyze training minute data over time (from month to month) and therefore was unable to determine trends related to canine teams’ compliance with the requirement. On the basis of our analysis of TSA’s data, we determined that some canine teams were repeatedly not in compliance with TSA’s 240- minute training requirement, in some cases for 6 months or more in a 1-year time period.

Utilization minutes: We found that TSA collected and analyzed data monthly on the amount of cargo TSI air cargo canine teams screened in accordance with the agency’s requirement. However, it was unclear how the agency used this information to identify trends to guide longer-term future program efforts and activities, since our analysis of TSA’s cargo screening data from September 2011 through July 2012 showed that TSI air cargo teams nationwide generally exceeded their monthly requirement. We concluded that TSA could increase the percentage of cargo it required TSI canine teams to screen.

Certification rates: We found that TSA tracked the number of certified and decertified canine teams, but was unable to analyze these data to identify trends in certification rates because these data were not consistently tracked and recorded prior to 2011. Specifically, we could not determine what, if any, variances existed in the certification rates among LEO and TSI teams over time because CTES reported it was unable to provide certification rates by type of canine team for calendar years 2008 through 2010. According to CTES, the agency recognized the deficiency and was in the process of implementing procedures to address data collection, tracking, and record-keeping issues.

Short notice assessments (covert tests): We found that when TSA was performing short notice assessments (prior to their suspension in May 2012), it was not analyzing the results beyond the pass and fail rates. We concluded that without conducting the assessments and analyzing the results of these tests to determine if there were any search areas or type of explosives in which canine teams were more effective compared with others, and what, if any, training may have been needed to mitigate deficiencies, TSA was missing an opportunity to fully utilize the results.

Final canine responses: Our analysis of final canine responses and data on corresponding swab samples used to verify the presence of explosives odor revealed that canine teams were not submitting swab samples to NEDCTP’s Canine Explosives Unit (CEU). Specifically, we determined that the number of swab samples sent by canine handlers to CEU for scientific review was far lower than the number of final canine responses recorded in CWS. We concluded that without the swab samples, TSA was not able to more accurately determine the extent to which canine teams were effectively detecting explosive materials in real-world scenarios.

In January 2013, we recommended that TSA regularly analyze available data to identify program trends and areas that are working well and those in need of corrective action to guide program resources and activities. These analyses could include, but not be limited to, analyzing and documenting trends in proficiency training minutes, canine utilization, results of short notice assessments and final canine responses, performance differences between LEO and TSI canine teams, as well as an assessment of the optimum location and number of canine teams that should be deployed to secure the U.S. transportation system. TSA concurred with our recommendation and has taken actions to address it. Specifically, TSA is monitoring canine teams’ training minutes over time by producing annual reports. TSA also reinstated short notice assessments in July 2013, and in the event a team fails, the FCC completes a report that includes an analysis of the team’s training records to identify an explanation for the failure. In April 2013, TSA reminded canine handlers of the requirement to submit swab samples of their canines’ final responses, and reported that the number of samples submitted that same month, increased by 450 percent, when compared with sample submissions in April 2012. CEU is producing reports on the results of its analysis of the swab samples for the presence of explosives odor. In June 2014, TSA officials told us that in March 2014, NEDCTP stood up a new office, known as the Performance Measurement Section, to perform analyses of canine team data. We believe that these actions address the intent of our recommendation and could better position TSA to identify program trends to better target resources and activities based on what is working well and what may be in need of corrective action.

TSA Has Conducted Additional PSC Team Effectiveness Assessments and Deployed Some Teams to Highest- Risk Airports, but Additional Actions Are Needed

TSA Has Conducted Additional PSC Team Effectiveness Assessments, but Has Not Compared PSC Teams with Conventional Canine Teams

In our January 2013 report, we found that TSA began deploying PSC teams in April 2011 prior to determining the teams’ operational effectiveness. However, in June 2012, the DHS Science and Technology Directorate (S&T) and TSA began conducting effectiveness assessments On the basis of to help demonstrate the effectiveness of PSC teams.these assessments, DHS S&T and TSA’s NEDCTP recommended that the assessment team conduct additional testing and that additional training and guidance be provided to canine teams. See the hyperlink in the note for figure 2 for videos of training exercises at one airport showing instances when PSC teams detected, and failed to detect, explosives odor. In January 2013, we concluded that TSA could have benefited from completing effectiveness assessments of PSCs before deploying them on a nationwide basis to determine whether they are an effective method of screening passengers in the U.S. airport environment.

We also reported in January 2013 that TSA had not completed an assessment to determine where within the airport PSC teams would be most effectively utilized, but rather TSA leadership focused on initially deploying PSC teams to a single location within the airport—the sterile area—because it thought it would be the best way to foster stakeholders’, specifically airport operators’ and law enforcement agencies’, acceptance of the teams. Stakeholders were resistant to the deployment of PSC teams because they have civilian handlers, and TSA’s response resolution protocols do not require the teams to be accompanied by a law enforcement officer. According to TSA’s Assistant Administrator for the Office of Security Operations, to alleviate airport stakeholders’ concerns regarding TSA’s response resolution protocols, the agency initially deployed PSC teams to the sterile areas, thereby enabling TSA to gather data on the value of PSC teams in the airport environment while reducing the likelihood of a final response from a PSC, since an individual has already passed through several layers of screening when entering the sterile area. However, aviation stakeholders we interviewed raised concerns about this deployment strategy, stating that PSC teams would be more effectively utilized in non-sterile areas of the airport, such as curbside or in the lobby areas. TSA subsequently deployed PSC teams to the passenger screening checkpoints. However, DHS S&T did not plan to assess the effectiveness of PSCs on the public side, beyond the checkpoint, since TSA was not planning to deploy PSCs to the public side of the airport when DHS S&T designed its test plan. In January 2013, we concluded that comprehensive effectiveness assessments that include a comparison of PSC teams in both the sterile and public areas of the airport could help TSA determine if it is beneficial to deploy PSCs to the public side of airports, in addition to or in lieu of the sterile area and checkpoint.

During the June 2012 assessment of PSC teams’ effectiveness, TSA conducted one of the search exercises with three conventional canine teams. Although this assessment was not intended to be included as part of DHS S&T’s and TSA’s formal assessment of PSC effectiveness, the results of the assessment suggested, and TSA officials and DHS S&T’s Canine Explosives Detection Project Manager agreed, that a systematic assessment of PSCs with conventional canines could provide TSA with information to determine whether PSCs provide an enhanced security benefit compared with conventional LEO aviation canine teams that have already been deployed to airport terminals. In January 2013, we concluded that an assessment would help clarify whether additional investments for PSC training are warranted. We also concluded that since PSC teams are trained in both conventional and passenger screening methods, TSA could decide to convert existing PSC teams to conventional canine teams, thereby limiting the additional resource investments associated with training and maintaining the new PSC teams.

We recommended that TSA expand and complete testing, in conjunction with DHS S&T, to assess the effectiveness of PSCs and conventional canines in all airport areas deemed appropriate prior to making additional PSC deployments to help (1) determine whether PSCs are effective at screening passengers, and resource expenditures for PSC training are warranted, and (2) inform decisions regarding the type of canine team to deploy and where to optimally deploy such teams within airports. TSA concurred and has taken some actions to address our recommendation, but further action is needed to fully address it. Specifically, in June 2014, TSA reported that through its PSC Focused Training and Assessment Initiative, a two-cycle assessment to establish airport-specific optimal working areas, assess team performance, and train teams on best practices, it had assessed PSC teams deployed to 27 airports, cumulating in a total of 1,048 tests. On the basis of these tests, TSA determined that PSC teams are effective and should be deployed at the checkpoint queue. In February 2014, TSA launched a third PSC assessment cycle to determine how PSCs’ effectiveness changes over time in order to determine their optimal duration time when working the checkpoint queue (i.e., how many minutes they can work and continue to be effective).

Although TSA has taken steps to determine whether PSC teams are effective and where in the airport environment to optimally deploy such teams, as of June 2014, TSA has not compared the effectiveness of PSCs and conventional canines in order to determine if the greater cost of training canines in the passenger screening method is warranted. According to TSA, the agency does not plan to include conventional canine teams in PSC assessments because conventional canines have not been through the process used with PSC canines to assess their temperament and behavior when working in proximity to people. While we recognize TSA’s position that half of deployed conventional canines are of a breed not accepted for use in the PSC program, other conventional canines are suitable breeds, and have been paired with LEO aviation handlers working in proximity with people since they patrol airport terminals, including ticket counters and curbside areas. We continue to believe that TSA should conduct an assessment to determine whether conventional canines are as effective detecting explosives odor on passengers when compared with PSC teams working in the checkpoint queue. As we reported, since PSC teams are trained in both conventional and passenger screening methods, TSA could decide to convert existing PSC teams to conventional canine teams, thereby limiting the additional resource investments associated with training and maintaining PSC teams.

TSA Deployed Some PSC Teams to Highest-Risk Airports

In our January 2013 report, we found that TSA’s 2012 Strategic Framework calls for the deployment of PSC teams based on risk; however, airport stakeholder concerns about the appropriateness of TSA’s response resolution protocols for these teams resulted in PSC teams not being deployed to the highest-risk airports. TSA officials stated that PSC teams were not deployed to the highest-risk airports for various reasons, including concerns from an airport law enforcement association about TSA’s decision to deploy PSC teams with civilian TSI handlers and the appropriateness of TSA’s response resolution protocols. These protocols require the canine handler to be accompanied by two additional personnel that may, but not always, include a law enforcement officer. According to representatives from an airport law enforcement association, these protocols are not appropriate for a suicide bombing attempt requiring an immediate law enforcement response. TSA’s decision to deploy PSC teams only to airports where they would be willingly accepted by stakeholders resulted in PSC teams not being deployed to the highest- risk airports on its high-risk list. Moreover, PSC teams that were deployed to high-risk airports, specifically two airports we visited, were not being used for passenger screening because TSA and the local law enforcement agencies had not reached agreement on the PSC response resolution protocols.

We recommended that if PSCs are determined to provide an enhanced security benefit, TSA should coordinate with airport stakeholders to deploy future PSC teams to the highest-risk airports, and ensure that deployed PSC teams are utilized as intended, consistent with its statutory authority to provide for the screening of passengers and their property. TSA concurred with our recommendation, and has taken action to address it. Specifically, as of June 2014, the PSC teams for which TSA had funding and not already deployed to a specific airport at the time our report was issued have been deployed to, or allocated to, the highest-risk airports. According to TSA, it was successful in deploying PSC teams to airports where they were previously declined by aviation stakeholders for various reasons. For example, TSA officials explained that stakeholders have realized that PSCs are an effective means for detecting explosives odor, and no checkpoints have closed because of a nonproductive response. PSCs also help reduce wait times at airport checkpoints because PSC teams are one method by which TSA can operate Managed Inclusion—a tool that allows passengers who have not, for example, enrolled in TSA PreTM to access to PreTM screening lanes.

According to TSA, PSC teams provide an added layer of security, making it possible for TSA to provide expedited screening to passengers who have not enrolled in TSA PreTM and therefore have not had a background check. In November 2013, TSA also reported it was making progress in working with stakeholders to allow PSC teams to work at checkpoints at airports where PSC teams were not previously performing passenger screening, but rather were training and screening air cargo. In June 2014, TSA officials reported that of all the airports where PSC teams had been deployed, all but one airport agreed to allow TSA to conduct screening of individuals at passenger screening checkpoint queues. We believe that these actions address the intent of our recommendation, contingent upon TSA comparing PSC teams with conventional canine teams.

Chairman Hudson, Ranking Member Richmond, and members of the subcommittee, this completes my prepared statement. I would be happy to respond to any questions you may have at this time.

GAO Contact and Staff Acknowledgments

For questions about this statement, please contact Jennifer Grover at (202) 512-7141 or GroverJ@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this statement include Chris Ferencik (Assistant Director), Chuck Bausell, Lisa Canini, Josh Diosomito, Michele Fejfar, Eric Hauswirth, Richard Hung, Thomas Lombardi, Jessica Orr, and Michelle Woods. Key contributors to the previous work that this testimony is based on are listed in the report.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.