TSA Explosives Detection Canine Program: Actions Needed to Analyze Data and Ensure Canine Teams Are Effectively Utilized
Released: 2013-01-31T00:00:00

Letter

Background

NCP’s mission is to deter and detect the introduction of explosive devices into the transportation system. As of September 2012, NCP has deployed 762 of 921 canine teams for which it is able to fund across the transportation system. Table 1 shows the number of canine teams by type for which funding is available, as well as describes their roles, responsibilities, and costs to TSA. There are four types of LEO teams: aviation, mass transit, maritime, and multimodal, and three types of TSI teams: air cargo, multimodal, and PSC.

TSA’s start-up costs for LEO teams include the costs of training the canine and handler, and providing the handler’s agency a stipend. The annual costs to TSA for LEO teams reflect the amount of the stipend. TSA’s start-up and annual costs for TSI canine teams are greater than those for LEO teams, because TSI handlers are TSA employees, so the costs include the handlers’ pay and benefits, service vehicles, and cell phones, among other things. PSC teams come at an increased cost to TSA compared with other TSI teams because of the additional training costs associated with providing decoys (i.e., persons pretending to be passengers who walk around the airport with explosive training aids).

From fiscal year 2010 to 2012, TSA funding for NCP increased from approximately $52 million to $101 million. During this time, TSA requested and was provided funding to increase the stipend it provides to law enforcement agencies for participating in NCP, and of amounts appropriated in fiscal year 2012, $5 million was directed for 20 new canine teams. For fiscal year 2013, TSA is requesting approximately $96 million for its canine program, which is about $5 million less in funding than was appropriated in fiscal year 2012. TSA plans to continue to fund 921 canine teams, and cited savings on supplies, and efficiencies in operations, among others, as reasons for the reduced funding request.

Figure 1 shows LEO, TSI, and PSC teams performing searches in different environments.

TSA obtains canines at no cost to TSA through an interagency agreement with the Department of Defense. DOD reported that the most recent agreement, dated January 2012, requires TSA to request the number of canines it needs based on its annual requirement. TSA reported that on the basis of how many canine teams it plans to train, it requests up to 190 canines each year. TSA also breeds canines at its Canine Breeding and Development Center at Lackland Air Force Base. Canines undergo 10 weeks of explosives detection training before being paired with a handler at TSA’s CTES, also located at Lackland Air Force Base. Conventional canine handlers attend a 10-week training course, and PSC handlers attend a 12-week training course. Canines are paired with a LEO or TSI handler during their training course. After canine teams complete this initial training, and are acclimated to their home operating environment, they undergo a 10- to 14-day “training mission” with a CTES evaluator to obtain initial certification to work in their home operating environment. During the training mission, the evaluator uses a checklist as a guide to determine if a canine team should be certified. Some items on the checklist are considered critical, and the canine team must demonstrate those necessary skills in order to be certified. After initial certification, canine teams are evaluated on an annual basis to maintain certification.

During the conventional explosives detection evaluation, canine teams must demonstrate their ability to detect all the explosive training aids the canines were trained to detect in five search areas. The five search areas are randomly selected among all the possible areas, but according to CTES, include the area that is most relevant to the type of canine team (e.g., teams assigned to airports will be evaluated in areas such as aircraft and cargo). Canine teams must find a certain percentage of the explosive training aids to pass their annual evaluation. In addition, a specified number of nonproductive responses (NPR)—when a canine responds to a location where no explosives odor is present—are allowed to pass an evaluation and maintain certification. After passing the conventional evaluation, PSC teams are required to undergo an additional annual evaluation that includes detecting explosives on a person, or being carried by a person. PSC teams are tested in different locations within the sterile area of an airport. A certain number of persons must be detected, and a specified number of NPRs are allowed for PSC certification.

TSA Collects and Uses Data to Track Canine Program Performance, but Could Better Analyze These Data to Identify Program Trends

TSA collects and uses key canine program data in its Canine Website System, a central management database, but it could better analyze these data to identify program trends. Through CWS, TSA captures the amount of time canine handlers spend on proficiency training as well as screening for explosives, among other functions. Table 2 highlights some of the key data elements included in CWS.

NCP uses CWS data to track and monitor canine teams’ performance. Specifically, FCCs review CWS data to determine how many training and utilization minutes canine teams are conducting on a monthly basis. NCP management also analyzes CWS data to determine, for example, how many canine teams are certified in detecting explosive odors, as well as the number of teams that passed short notice assessments. However, TSA has not fully analyzed the performance data it collects in CWS to identify program trends and areas that are working well or in need of corrective action. Such analyses could better position TSA to determine canine teams’ proficiency, guide future deployment efforts, and help ensure that taxpayer funds are being used effectively.

Training Minutes

TSA tracks the number of training minutes canine teams conduct on a monthly basis, as well as the types of explosives and search areas used when training, to ensure teams maintain their proficiency in detecting explosive training aids. However, TSA does not analyze training minute data over time (from month to month) and therefore is unable to determine trends related to canine teams’ compliance with the requirement. On the basis of our analysis of TSA’s data, we determined that some canine teams were repeatedly not in compliance with TSA’s 240-minute training requirement, and identified differences in the rate of compliance between TSI and LEO teams.

According to senior NCP officials, reasons canine teams may not meet the requirement include the canine or handler taking leave, sometimes as a result of sickness or injury, and participating in an annual evaluation, among other reasons. While these circumstances may lead to a team not completing the required minutes certain months during a year, our analysis further revealed that a certain number of canine teams did not meet the training requirement for a more extended period, at least 6 months during the period analyzed. In light of this broader trend, NCP officials agreed that it would be useful to monitor compliance with training requirements over time to ensure canine teams maintain their proficiency in detecting explosives odor and stated that this information was previously unknown to TSA until we brought it to its attention. Analyzing the existing training data could help alert TSA to trends in noncompliance that could assist the agency in identifying canine team deficiencies and areas in need of corrective action. Specifically, such analyses could be used by program managers to place the canine handlers who do not meet the training requirement on a performance improvement plan, or ultimately remove the handlers from the canine program, among other actions.

Utilization Minutes

TSA tracks and monitors on a monthly basis the number of utilization minutes canine teams conduct searching for explosives, screening air cargo, and serving as a deterrent. TSA also collects monthly data on the amount of cargo TSI air cargo teams screen in accordance with the agency’s requirement, but hasn’t analyzed these data over time to determine if, for example, changes are in needed in the screening requirement or the number of teams deployed. Our analysis of CWS utilization data for the period from May 2011 through April 2012 showed that LEO teams consistently reported greater levels of monthly utilization minutes than TSI teams, and that on average, from September 2011 through July 2012, TSI air cargo teams exceeded their monthly screening requirement. However, according to senior NCP officials, it is difficult to compare utilization rates directly because LEO and TSI teams record utilization minutes differently. LEO teams report their utilization minutes based on the amount of time they spend patrolling transportation terminals, searching baggage and vehicles, and screening air cargo, among other things. TSI teams, on the other hand, report their utilization minutes based on the time spent primarily screening air cargo.

According to senior NCP officials, the differences in how utilization minutes are recorded explains why utilization minutes vary so widely between LEO and TSI teams. While variances exist in utilization minutes for canine teams, opportunities exist to determine whether LEO and TSI teams are being utilized effectively. Unlike TSA’s monthly 240-minute training requirement, there is no minimum monthly canine utilization requirement to help ensure that canine teams are effectively utilized. According to senior NCP officials, the agency is considering whether a minimum requirement for, and a consistent definition of, utilization minutes should be applied to LEO and TSI teams in response to our review. In the absence of a consistent definition of utilization minutes or goals for utilization, it will be difficult for TSA to determine whether canine teams are being utilized effectively. Analysis of utilization minutes in light of stated goals could help TSA determine where LEO and TSI canine teams are most effectively utilized and could leverage such analysis to drive future deployment efforts.

In addition to capturing utilization minutes, TSA also collects and analyzes data monthly on the amount of cargo TSI air cargo canine teams screen in accordance with the agency’s requirement. However, it is unclear how the agency uses this information to identify trends to guide longer-term future program efforts and activities. Our analysis of TSA’s cargo screening data from September 2011 through July 2012 showed that TSI air cargo teams nationwide generally exceeded their monthly requirement to screen air cargo placed on passenger aircraft. This suggests that TSA could increase the percentage of air cargo it requires canine teams to screen, or redeploy additional TSI air cargo teams. For example, while TSI air cargo teams have generally met or exceeded their monthly screening requirement, as of September 2012, the agency has yet to deploy all of its 120 TSI air cargo teams for which funding is available. However, TSA has not utilized the trends in cargo screening data to guide agency decision making as to whether the current and planned deployment number of TSI teams by location is appropriate. Three of the four supervisory TSI officials we interviewed stated that staffing levels of TSI teams were sufficient. However, one supervisory TSI official stated that his TSI staffing level was more than double what was needed to meet TSA’s screening requirement and provide adequate coverage for the airport. However, TSA plans to deploy additional TSI teams to this airport in the near future. As TSA continues to roll out TSI air cargo teams to reach its goal of 120 teams nationwide for which it has funding, an analysis of utilization data could better inform agency decision makers regarding future TSI air cargo screening requirements, and number of canine team deployments by location.

Certification Rates

TSA tracks the number of certified and decertified canine teams, but is unable to analyze these data to identify trends in certification rates because these data were not consistently tracked and recorded prior to 2011. Our analysis of CWS certification data revealed that approximately the same percentage of LEO and TSI teams were TSA certified in 2011 and 2012. While the certification data show a high percentage of certified canine teams for the time periods analyzed, it is unclear whether certifications rates have changed over time for LEO and TSI canine teams because reliable data for older periods of time were not available.

According to TSA officials, while LEO certification rates have remained constant for a 2-year time period, TSI certification rates have increased substantially since 2008, when TSI teams were initially deployed. TSA officials attributed the initially low TSI certification rates to lack of management support and limited availability of explosive training aids. However, we could not determine what, if any, variances existed in the certification rates among LEO and TSI teams over time because CTES reported it was unable to provide certification rates by type of canine team for calendar years 2008 through 2010. According to CTES, the agency recognizes this deficiency and is in the process of implementing procedures to address data collection, tracking, and record-keeping issues. It will be important for TSA to continue to capture and analyze these data to understand the differences that may exist between LEO and TSI canine teams; how these differences may affect their effectiveness; and what steps, if any, need to be taken to address any differences.

Short Notice Assessments

To help provide reasonable assurance that canine teams are proficient in detecting explosives odor, NCP conducts and collects information on covert tests conducted by FCCs that assess canine teams’ operational effectiveness in detecting and responding to possible explosives, which TSA refers to as short notice assessments. However, TSA has not analyzed the results of these assessments beyond the pass and fail rate, which limits NCP’s ability to determine the effectiveness of its canine teams. Furthermore, these assessments were suspended in May 2012 because of FCC staffing shortages. NCP began conducting short notice assessments in April 2011, and FCCs conducted 159 short notice assessments in 2011. While these tests can be helpful in identifying canine deficiencies, without analyzing the results of these tests to determine if there are any search areas or type of explosives in which canine teams are more effective compared with others, and what, if any, training may be needed to mitigate deficiencies, TSA is missing an opportunity to fully utilize them. Resuming and analyzing the results of short notice assessments will provide TSA with additional information on the effectiveness of its canine teams in detecting explosives odor, including determining the conditions under which canine teams are most or least effective. According to NCP officials, TSA is in the process of hiring 10 additional FCCs and expects all 10 will be employed in early 2013, at which time NCP plans to resume the assessments, and analyze the results.

Final Canine Responses

Our analysis of final canine responses and data on corresponding swab samples used to verify the presence of explosives odor revealed that canine teams are not submitting swab samples to NCP’s Canine Explosives Branch, which is an important component of TSA’s quality assurance process. In March 2011, TSA issued guidance to FCCs instructing canine handlers on how to collect and submit swab samples of final responses to CEB to verify the presence of explosives odor. Such samples can confirm canines are effectively detecting explosives odor, although there are limitations to taking swab samples, and a sample that does not confirm the presence of odor does not necessarily mean the canine falsely detected explosives odor (e.g., the sample may have been taken from the wrong part of the item). The purpose of the guidance was to develop a standardized process of collecting swab samples to determine canine teams’ effectiveness in detecting explosives odor.

However, we found that canine handlers did not always submit swab samples. Specifically, we determined that the number of swab samples sent by canine handlers to TSA’s CEB for scientific review was far lower than the number of final canine responses recorded in CWS. For instance, in 2011, of the total number of final canine responses recorded in CWS, LEO and TSI canine teams submitted swab samples for 8 percent of the final responses to CEB for scientific study after the guidance was issued. NCP stated that 49 percent of the final canine responses entered in CWS were improperly or incorrectly recorded by canine handlers, and the remaining 51 percent of cases did not have to be submitted to CEB for scientific study since submission under the guidance was not a requirement. However, NCP officials agreed that tracking and scientifically measuring final canine responses could help NCP measure canine teams’ effectiveness in detecting explosives in their operational environments, such as an airport or air cargo facility. According to these officials, the agency will take steps to require canine handlers to submit swab samples to CEB. For example, the requirement will be added to NCP’s Standard Operating Practices and Procedures, which is expected to be issued by March 2013. In addition, senior NCP officials told us the agency will clarify this policy with program supervisors and canine handlers, and modify CWS to more easily allow handlers to document when a swab sample is collected and sent to CEB for study. If these steps are implemented effectively, the issues we raised will be addressed.

Of the final canine responses that could have been submitted to CEB, NCP reported that almost half were resolved locally by identifying sources of odors consistent with those canines are trained to detect. However, NCP could not determine the resolution of the remaining cases that were not submitted to CEB, and for which no final resolution was documented at the local level. As our analysis has shown, capturing and analyzing additional information on final canine responses over time is an important quality assurance step and may help TSA and other interested stakeholders to more accurately determine the extent to which canine teams are effectively detecting explosive materials in real world scenarios.

Standards for Internal Control in the Federal Government requires agencies to ensure that ongoing monitoring occurs during the course of normal operations to help evaluate program effectiveness. TSA collects and uses key canine team performance data, but it has not fully analyzed these data, consisting of training and utilization minutes, certification rates, and results of short notice assessments and final canine responses. Such analyses could help TSA management identify program trends to better target resources and activities based on what is working well and what may be in need of corrective action. According to NCP officials, while FCCs review training and utilization data on a monthly basis, NCP does not have the time or staff resources to analyze such data over time. However, canine handlers collectively spend a significant amount of time providing the data to NCP, and such analyses could better position TSA to determine the proficiency of its canine teams, guide future deployment efforts, and help ensure taxpayer funds are being used effectively.

TSA Has Not Deployed Passenger Screening Canine Teams to the Highest- Risk Airports and Did Not Determine Their Effectiveness Prior to Deployment

TSA’s 2012 Strategic Framework calls for the deployment of PSC teams based on risk; however, airport stakeholder concerns about the appropriateness of TSA’s response resolution protocols for these teams have resulted in PSC teams not being deployed to the highest-risk airport terminals and concourses. Moreover, TSA began deploying PSC teams prior to determining the teams’ operational effectiveness and before identifying where within the airport these teams could be most effectively utilized to screen passengers.

TSA Has Deployed PSC Teams to Airports; However, PSC Teams Have Not Been Deployed to the Highest-Risk Terminals and Concourses

In April 2011, TSA began deploying PSC teams to airports terminals and concourses, and plans to deploy all 120 PSC teams for which it has funding by the end of calendar year 2013. TSA’s Strategic Framework calls for the deployment of PSC teams based on risk; however, we found that PSC teams have not been deployed to the highest-risk airport terminals and concourses based on TSA’s high-risk list. TSA officials stated that PSC teams were not deployed to the highest-risk terminals and concourses for various reasons, including concerns from an airport law enforcement association about TSA’s decision to deploy PSC teams with civilian TSI handlers and the appropriateness of TSA’s response resolution protocols. These protocols require the canine handler to be accompanied by two additional personnel that may, but not always, include a law enforcement officer. According to representatives from an airport law enforcement association, these protocols are not appropriate for a suicide bombing attempt requiring an immediate law enforcement response.

According to TSA officials, the lack of agreed-upon response resolution protocols with local law enforcement officials has complicated efforts to introduce PSCs at some airports. For example, seven airport operators have declined the deployment of PSC teams because of concerns expressed by airport stakeholders regarding TSA’s response resolution protocols and whether they are adequate for dealing with potential threats, such as suicide bombers. According to TSA’s Assistant Administrator for the Office of Security Operations, TSA’s goal is to deploy PSC teams to airports where there is a supportive working environment, and accordingly, TSA does not generally require that PSC teams be accepted for use at an airport. However, TSA reported that if it identified a specific threat to an airport, it would use its authority to deploy the PSC teams regardless of any airport stakeholder opposition to the decision, but it has not had to do so.

Representatives from three major aviation industry associations we interviewed support using PSC teams in airports, but also raised concerns surrounding TSA’s PSC response resolution protocols. These concerns were primarily related to TSA deploying PSCs with unarmed TSI handlers rather than LEO handlers because of the differences in their abilities to respond to a potential threat, such as a suicide bomber. According to TSA officials, TSA deployed PSCs with TSI handlers in a manner consistent with the report of the Committee on Appropriations of the House of Representatives, which accompanies the committee’s DHS fiscal year 2013 appropriations bill. Officials from an association representing airport law enforcement officers recommended that PSC teams be accompanied by three LEOs, among other reasons, to help reduce the potential liability and response times associated with responding to a possible suicide bomber. Representatives from an association representing airport law enforcement officers and TSA officials stated that they have been working together to try to address concerns regarding TSA’s response resolution protocols; however, they have not been able to reach an agreement on this issue, as of September 2012. TSA maintains that PSCs are similar to the agency’s other screening tools used to detect explosives, and thus a final response from a PSC should trigger the same type of law enforcement response to potential threats identified through other screening measures. Additionally, TSA officials stated that airport operators are required to adopt and maintain TSA-approved security programs that, among other things, provide for a law enforcement presence and capability at the airport that is adequate to ensure the safety of passengers, and that their current approach of using LEOs in support of TSI-led PSC operations is consistent with this requirement.

However, TSA’s decision to deploy PSC teams only to airports where they would be willingly accepted by stakeholders has resulted in PSC teams not being deployed to the highest-risk airport terminals and concourses on its high-risk list. Given that PSC teams cost $164,000 annually per team, TSA is not using the teams in the most cost-effective manner to enhance security if it is limited to deploying them at lower-risk airports and concourses. Moreover, PSC teams at the two high-risk airports we visited are not being used for passenger screening because TSA and the local law enforcement agencies have not reached agreement on the PSC response resolution protocols. Thus, rather than being utilized for their intended primary purpose—passenger screening— PSC teams are being used to screen air cargo or conduct training. TSA officials agreed that the decision to defer to airport stakeholders’ willingness to have PSC teams deployed to their airports has resulted in PSCs not being optimally utilized. While we recognize the value of obtaining airport stakeholder buy-in, deploying PSC teams to the highest- risk airport terminals and concourses in accordance with its Strategic Framework, could help TSA better deploy future PSC teams in a cost- effective manner, and utilize the PSC teams that have already been deployed for their intended purposes to enhance security.

TSA Plans to Complete PSC Effectiveness Assessments in 2013; Initial Results Indicate Performance Challenges

TSA began deploying PSC teams in April 2011 prior to determining the teams’ operational effectiveness and before identifying where within the airport environment these teams would be most effectively utilized. According to TSA officials, operational assessments did not need to be conducted prior to their deployment because canines were being used to screen passengers by other entities, such as airports in the United Kingdom, and TSA leadership focused on initially deploying PSC teams to a single location within the airport—the sterile area—because it thought it would be the best way to foster stakeholders’ acceptance of the teams. However, in June 2012, DHS S&T and TSA began conducting operational assessments to help demonstrate the effectiveness of PSC teams, and plans to complete additional PSC effectiveness testing in 2013 at additional airports. To assess the effectiveness of PSC teams in detecting explosives odor, TSA and DHS S&T are assessing their capabilities based on several factors. On the basis of these results, DHS S&T and TSA’s NCP recommended that the assessment team conduct additional testing and that additional training and guidance be provided to canine teams.

As part of our review, we visited two airports at which PSC teams have been deployed and observed training exercises in which PSC teams accurately detected explosives odor (i.e., positive response), failed to detect explosives odor (i.e., miss), and falsely detected explosives odor (i.e., non productive response). See the hyperlink in the note for figure 2 for videos of training exercises at one airport showing instances when PSC teams detected, and failed to detect explosives odor.

On the basis of the results of DHS’s assessments, TSA could have benefited from completing operational assessments of PSCs before deploying them on a nationwide basis to determine whether they are an effective method of screening passengers in the U.S. airport environment. TSA officials stated that operational assessments did not need to occur prior to deployment because the agency does not consider PSCs a new screening method. Moreover, according to DHS S&T’s Canine Explosives Detection Project Manager, some PSC teams needed to be deployed operationally at airports before their effectiveness could be assessed. While we agree that some PSC teams needed to be deployed for operational assessments to occur, this testing could have been completed before TSA started deploying additional PSC teams on a nationwide basis beyond the three pilot site airports. As discussed earlier, TSA plans to deploy all 120 teams for which it has funding by the end of calendar year 2013. Best practices for program performance call for agencies to use performance information to assess efficiency, identify performance gaps, and ensure intended goals are met. We previously reported on the need for TSA to validate detection methods before deploying new screening methods. For example, TSA proceeded with deploying its Screening of Passengers by Observation Techniques behavior detection program on a nationwide basis before determining whether the list of passenger behaviors and appearances underpinning the program were scientifically validated, and whether these techniques could be applied on a large scale for counterterrorism purposes in an airport environment. Assessing the effectiveness of PSCs in the operational environment could help provide TSA with reasonable assurance that PSCs are effective in identifying explosives odor on passengers and provide an enhanced security benefit.

Additionally, TSA has not completed an assessment to determine where within the airport PSC teams would be most effectively utilized. According to TSA’s Assistant Administrator for the Office of Security Operations, to alleviate airport stakeholders’ concerns regarding TSA’s response resolution protocols, the agency initially deployed PSC teams to the sterile areas, thereby enabling TSA to gather data on the value of PSC teams in the airport environment while reducing the likelihood of a final response from a PSC resulting in the closure of a checkpoint since an individual has already passed through several layers of screening when entering the sterile area. However, aviation stakeholders we interviewed, raised concerns about this deployment strategy, stating that PSC teams would be more effectively utilized in non sterile areas of the airport, such as curbside or in the lobby areas. Specifically, two of the three aviation industry representatives we interviewed stated that deploying PSC teams to the sterile area, rather than the public area, could limit their ability to detect and deter potential suicide bombers entering the airport. According to TSA’s Assistant Administrator for the Office of Security Operations, TSA initially deployed PSC teams to the sterile area because it thought it would foster stakeholders’ acceptance of PSC teams and does not generally consider the deployment of PSC teams to the sterile area as a permanent or most optimal use of the PSC teams. Instead, the agency maintains that the best use of PSC teams is in the public area or at the screening checkpoint.

TSA has since deployed PSC teams to the passenger screening checkpoints. According to NCP officials, the agency is also considering providing some PSCs to LEOs to work on the public side of the airport. However, DHS S&T does not plan to assess the effectiveness of PSCs on the public side, beyond the checkpoint, since TSA was not planning to deploy PSCs to the public side of the airport when DHS S&T designed its test plan. Comprehensive operational assessments that include a comparison of PSC teams in both the sterile and public areas of the airport could help TSA determine if it is beneficial to deploy PSCs to the public side of airports, in addition to or in lieu of the sterile area and checkpoint. Moreover, Standards for Internal Control in the Federal Government stresses the need for agencies to provide reasonable assurance of the effectiveness and efficiency of operations, including the use of the entity’s resources.

During the June 2012 assessment of PSC teams’ effectiveness, TSA conducted one of the search exercises with three conventional canine teams. Although this assessment was not intended to be included as part of DHS S&T and TSA’s formal assessment of PSC effectiveness, the results of this assessment suggest, and TSA officials and DHS S&T’s Canine Explosives Detection Project Manager agreed, that a systematic assessment of PSCs with conventional canines could provide TSA with information to determine whether PSCs provide an enhanced security benefit compared with conventional LEO aviation canine teams that have already been deployed to airport terminals. An assessment would help clarify whether additional investments for PSC training are warranted. Moreover, since PSC teams are trained in both conventional and passenger screening methods, TSA could decide to convert existing PSC teams to conventional canine teams, thereby limiting the additional resource investments associated with training and maintaining the new PSC teams. Additionally, as previously discussed, should TSA decide to deploy some of the PSCs scheduled to be deployed in 2013 with LEO handlers, rather than TSI handlers, TSA could further reduce program costs and increase costs savings, as a PSC team led by a LEO handler could cost TSA $53,000 annually per team, compared with $164,000, the current annual cost per TSI-led PSC team. Moreover, TSA officials and aviation industry stakeholders stated that deploying PSCs with experienced LEO canine handlers could reduce the learning curve associated with becoming a PSC handler.

Conclusions

Securing the nation’s vast and diverse transportation system is a challenging task that is complicated by the ever-changing and dynamic threat environment. TSA’s National Canine Program is an important and growing component of its efforts to achieve this goal. However, a systematic analysis of key program data that TSA currently collects, consisting of training and utilization minutes, certification rates, results of short notice assessments (covert tests), and canine responses to the possible presence of explosives, could better position TSA to identify program trends to better target resources and activities based on what is working well and what may be in need of corrective action. In addition, a comprehensive assessment of passenger screening canines and conventional canine teams in all areas of the airport could provide TSA with greater assurance that passenger screening canine teams are a cost-effective screening tool and are being optimally utilized. Such analyses could help TSA determine the appropriate number, type, and placement of canine teams needed to best secure the U.S. transportation system. Finally, if effective, deploying passenger screening canine teams to the highest-risk airport terminals and concourses will provide TSA with greater assurance that the PSC teams are deployed in a manner that targets the most pressing security needs in a cost-effective manner.

Recommendations for Executive Action

To help ensure TSA analyzes canine team data to identify program trends, and determines if PSC teams provide an added security benefit to the civil aviation system, and if so, deploys PSC teams to the highest-risk airports, we recommend that the Administrator of the Transportation Security Administration direct the Manager of the NCP to take the three following actions:

Regularly analyze available data to identify program trends and areas that are working well and those in need of corrective action to guide program resources and activities. These analyses could include, but not be limited to, analyzing and documenting trends in proficiency training, canine utilization, results of short notice assessments (covert tests) and final canine responses, performance differences between LEO and TSI canine teams, as well as an assessment of the optimum location and number of canine teams that should be deployed to secure the U.S. transportation system.

Expand and complete testing, in conjunction with DHS S&T, to assess the effectiveness of PSCs and conventional canines in all airport areas deemed appropriate (i.e., in the sterile area, at the passenger checkpoint, and on the public side of the airport) prior to making additional PSC deployments to help (1) determine whether PSCs are effective at screening passengers, and resource expenditures for PSC training are warranted, and (2) inform decisions regarding the type of canine team to deploy and where to optimally deploy such teams within airports.

If PSCs are determined to provide an enhanced security benefit, TSA should coordinate with airport stakeholders to deploy future PSC teams to the highest-risk airports, and ensure that deployed PSC teams are utilized as intended, consistent with its statutory authority to provide for the screening of passengers and their property.

Agency Comments and Our Evaluation

We provided a draft of the sensitive version of this report to the Departments of Defense and Homeland Security on November 13, 2012, for their review and comment. DOD did not provide comments. DHS provided written comments, which are reprinted in appendix III. DHS concurred with all three recommendations, and identified actions taken or planned to implement the recommendations.

DHS concurred with the first recommendation, that TSA regularly analyze canine team data to identify program trends, and areas working well and those in need of correction action. DHS stated that NCP is planning to reestablish yearly comprehensive assessments of canine teams’ compliance with program requirements, and implement corrective action, when necessary. Furthermore, TSA is upgrading CWS, and in March 2013 will be able to capture data on PSC teams, as it currently does for conventional canine teams. We support TSA’s actions to implement these yearly assessments, as well as the upgrade to CWS to collect information on PSC teams. We believe it is important for TSA to also review its analysis of canine teams’ performance screening air cargo in order to inform future decisions on the percentage of air cargo that it requires canine teams to screen, as well as the number of TSI air cargo canine teams needed to meet TSA’s air cargo screening requirement. TSA should also analyze its data, including results of short notice assessments and final canine responses, to assess canine team performance in order to mitigate any explosives detection challenges and determine canine teams’ strengths. Such actions, in conjunction with TSA’s planned efforts, would meet the intent of the recommendation.

DHS concurred with the second recommendation, to expand and complete testing to assess the effectiveness of PSC teams and conventional canines in areas of the airport deemed appropriate. DHS stated that TSA and DHS S&T will conclude testing of PSC teams in the sterile areas of airports in February 2013, and have begun testing PSC teams in the public areas of the airports. In addition, TSA will discuss with DHS S&T developing testing protocols for conventional canines. We support TSA’s actions to assess the effectiveness of PSC teams, and encourage TSA to incorporate conventional canines into future testing to demonstrate if PSCs provide an enhanced explosives detection benefit relative to conventional canines, given their increased cost.

DHS also stated that TSA anticipates deploying all 120 of the PSC teams for which it received authorization by the end of calendar year 2013. If PSC teams are not proven to be effective based on TSA and DHS S&T’s assessments, we strongly encourage TSA to reconsider deploying PSC teams for which it has not already allocated or obligated resources. Such action, in conjunction with TSA’s planned efforts, would meet the intent of the recommendation.

DHS concurred with the third recommendation, to deploy PSC teams to the highest-risk airports. DHS stated that TSA will deploy future teams to the highest-priority airports as identified by both its operational and risk- based analyses, and it continues to evaluate and modify its risk-based analysis as needed. We support TSA’s efforts to deploy PSC teams to the highest-risk airports in order to provide an additional layer of security where it is most needed, and continue to believe that TSA first needs to determine if PSCs provide an enhanced security benefit and, if so, coordinate such deployments with airport stakeholders. Such action, in conjunction with TSA’s planned efforts, should address the intent of the recommendation.

DHS also provided technical comments which we incorporated, as appropriate.

If you or your staff have any questions about this report, please contact me at (202) 512-4379 or lords@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV.

Appendix I: TSA Has Taken Actions to Enhance Program Training and Evaluation, and Research, Development, and Testing

Since our review of the Transportation Security Administration’s (TSA) National Canine Program (NCP) in 2008, the agency has taken actions to enhance the program in the areas of (1) training and evaluation, and (2) research, development, and testing with the DHS Science and Technology Directorate (S&T). Table 3 provides an overview of these efforts.

Training and evaluation. Canine handlers we interviewed in four locations (74 of 75 handlers), generally spoke positively of the instruction they had received during their training courses and missions. For example, some canine handlers cited the hands-on instruction and feedback they received from TSA Canine Training and Evaluation Section (CTES) evaluators during their training missions as particularly beneficial. However, some canine handlers we interviewed (26 of 67 evaluated) suggested that TSA seek ways to improve the consistency and objectivity of the evaluation process. For example, some canine handlers stated that while one evaluator may consider a canine sitting a few feet away from the explosive training aid a “fringe” response and count it positively, another evaluator would consider the same scenario a nonproductive response (NPR). As noted earlier, a specified number of NPRs are allowed to pass an evaluation and maintain certification. In July 2011, CTES formed a committee comprising various TSA officials and canine handlers to conduct a review of its evaluation process. The committee suggested creating a “Train the Evaluator” course to reduce inconsistencies in the evaluation process. CTES subsequently created an evaluation unit, which will be implementing the course on consistently applying evaluation guidelines later this calendar year. Furthermore, in January 2012, CTES began sending its most experienced evaluators to all annual evaluations to help improve the consistency and objectivity of the evaluation process. Specifically, according to CTES officials, one of the two evaluators conducting the evaluation is to be from the evaluation unit, which is composed of CTES’s 12 most experienced evaluators. The committee supported actions to provide greater consistency during the evaluation process but stated that the evaluations are fair, and ultimately not the reason for canine teams failing evaluations. Rather, the committee cited a lack of local training support for canine teams once they are deployed to their home units. As a result, CTES is in the process of deploying regional trainers to the field to assist canine teams with proficiency training. Thus far, 23 of 27 regional trainers have been deployed to the field. CTES planned to deploy all 27 regional trainers by December 2012.

Research, development, and testing. Through its Canine Explosives Detection Project, DHS S&T is working to enhance TSA canine effectiveness through two efforts: (1) conducting operational assessments of canine detection capabilities in the air cargo environment, and (2) determining the behavioral, physiological, and genetic identifiers of a successful explosives detection canine.

Operational assessments: DHS S&T has conducted a series of tests assessing the ability of canine teams to detect explosives in air cargo. During the most recent series of tests, conducted in 2009 and 2010, DHS S&T determined canines can be an effective tool in detecting explosives in the air cargo environment, but recommended more frequent and appropriate training.

DHS S&T and TSA also conducted a pilot test to determine if private, or third-party, canine teams could be used by the air cargo industry to screen air cargo for the purposes of meeting the mandate to screen 100 percent of air cargo. According to TSA, the pilot concluded in September 2011, and TSA determined that utilizing third-party canine teams to screen air cargo could be a viable option, but there would also be challenges, including providing the resources needed to maintain oversight of the program. TSA leadership is evaluating whether to move forward with the implementation of a third-party canine program. Allowing the air cargo industry to use canines could increase its ability to meet the screening mandate by overcoming challenges we have previously reported.

Behavior and genetics: DHS S&T, in collaboration with a private company, Dog Genetics, and the University of Texas, is researching behavioral, physiological, and genetic identifiers of a successful explosives detection canine. DHS S&T is reviewing the history of proven explosives detection canines within NCP and is attempting to identify behaviors and DNA markers that are indicative a canine will be successful at explosives detection. The goal of this research is to allow TSA to better select which canines to breed, and which to select for explosives detection training. This is a long-term effort and DHS S&T estimates this research will be completed in fiscal year 2015.

Appendix II: Objectives, Scope, and Methodology

This report addresses the following objectives: (1) What data does TSA have on its canine program, what do these data show, and to what extent has TSA analyzed these data to identify program trends? (2) To what extent has TSA deployed passenger screening canine teams using a risk-based approach and determined their effectiveness prior to deployment?

In addition, this report summarizes the actions TSA has taken to enhance its National Canine Program since we issued our last report, in 2008.

To determine what data TSA has on its canine program, what these data show, and the extent to which TSA has analyzed these data to identify program trends, we interviewed TSA’s Office of Security Operations headquarters and field officials about the type of data they collect and how they analyze the data to monitor canine team effectiveness consistent with Standards for Internal Control in the Federal Government. Specifically, we interviewed canine branch chiefs at headquarters who oversee the operations of NCP. We also interviewed an area canine coordinator and field canine coordinators, and supervisory transportation security inspectors who oversee canine teams in the field. Using data from the agency’s Canine Website System (CWS), we analyzed training and utilization minutes from May 2011 through April 2012—the most recent data available at the time of our analysis. We compared the number of training minutes canine teams conducted each month with TSA’s training requirement documented in its Canine Manual and cooperative agreements with law enforcement agencies. We analyzed the number of minutes canine teams were utilized by type of canine team to identify any differences between law enforcement officer (LEO) and transportation security inspector (TSI) teams. In addition, we collected and reviewed monthly reports on canine team training and utilization from an area canine coordinator and field canine coordinators. Further, we analyzed certification data from CWS for May 2012—the most recent data available at the time of our analysis—to determine the number of certified and decertified canine teams by type of team (LEO and TSI). Moreover, we reviewed TSA certification data by type of team from January 2011 through August 2012. To assess the reliability of the CWS data on canine team training, utilization, and certification, we interviewed knowledgeable officials, reviewed the data for obvious errors and anomalies, and reviewed documentation. We determined the data were sufficiently reliable for our purposes. We also reviewed and analyzed the number of final canine responses documented in CWS, and corresponding swab samples sent to TSA’s Canine Explosives Branch (CEB) to validate canines’ ability to detect explosives odor. Specifically, we compared the number of final canine responses recorded in CWS with the number of swab samples sent to CEB from the time the guidance was issued on collecting swab samples in March 2011 to July 2012—the most current data available at the time of our request. We found limitations in the final canine response data, as discussed in the report, but we believe the data to be sufficiently reliable for generally comparing the number of canine responses with the number of swab samples.

To assess the extent TSA deployed passenger screening canines (PSC) in a risk-based manner, consistent with TSA’s 2012 Strategic Framework, we compared the risk rankings TSA assigned to airport terminals with NCP’s September 2012 canine asset spreadsheet—which tracks the deployment location of TSA’s canine assets. We also interviewed senior TSA and NCP headquarters and field officials to obtain information on the extent to which risk and other factors, such as stakeholder concerns, were considered when deploying PSC teams. We also reviewed TSA’s Recommended Operational Procedures for Canine Screening of Individuals to understand the agency’s operational procedures and protocols for deploying and resolving final responses by PSC teams. In addition, we interviewed PSC handlers at two locations we visited where PSCs were initially deployed. We also interviewed officials from Amtrak police, and three major aviation industry associations—Airport Law Enforcement Agencies Network (ALEAN), Airports Council International- North America (ACI-NA), and American Association of Airport Executives (AAAE)—to discuss their views on PSC utilization and response protocols. While the results from our interviews are not generalizable to all PSC handlers and aviation stakeholders, they provided views on TSA’s utilization of PSC teams and protocols for resolving final responses.

To determine the extent to which TSA determined the effectiveness of PSC teams, we compared the PSC assessment approach used by TSA and DHS S&T with DHS guidance and best practices. We also interviewed senior TSA and DHS S&T officials to obtain information on the extent to which a comprehensive assessment of the effectiveness of PSC teams had been completed in the operational environment, including determining (1) where within the airport environment (i.e. sterile, checkpoint, and public areas) PSC teams would most effectively utilized, and (2) whether PSCs provide an added benefit in terms of cost and security to conventional canines. Moreover, we reviewed the test results from TSA and DHS S&T’s assessment to determine the extent to which PSC teams have been proven effective in different areas at the airport, consistent with the environments described in TSA’s Recommended Operational Procedures for Canine Screening of Individuals and best practices in Standards for Internal Control in the Federal Government. In addition, we interviewed DHS S&T officials regarding the testing methodology, and determined that the results are reliable. We also observed PSC teams in their operational environments during our site visits, and served as decoys—passengers with concealed explosive training aids—during PSC training exercises in airport terminals, and videotaped the PSC teams. We reviewed canine team costs data, as of January 2012, to identify the difference in costs among the various types of canine teams. To assess the reliability of the canine team cost data, we interviewed knowledgeable officials, reviewed the data for obvious errors and anomalies, and reviewed documentation. We determined the data were sufficiently reliable for our purposes.

To summarize the actions TSA has taken to enhance NCP since our 2008 review, we reviewed TSA’s canine team training and evaluation procedures. We discussed these procedures with trainers, instructors, and evaluators at TSA’s CTES. In addition, we observed canine training at CTES at Lackland Air Force Base in Texas, as well as Auburn University’s Canine Detection Training Center in Alabama because this is where some PSC teams were trained. We discussed canine training and evaluation practices with the Department of Defense’s Military Working Dog Program, and the Scientific Working Group on Dog and Orthogonal Detector Guidelines. Furthermore, we conducted site visits to the three locations with the greatest number of deployed canine teams in the mass transit, airport, and air cargo environments to observe canine teams training in their operational environments and interview LEO and TSI canine handlers. We also observed canine teams and interviewed handlers during a site visit we conducted to a fourth location so we could observe additional PSC teams. While the results from our interviews are not generalizable to all canine handlers, they provided a range of perspectives on canine handlers’ views on the training and evaluation process. We reviewed DHS S&T’s efforts to conduct research, development, and testing to enhance canine effectiveness through its Canine Explosives Detection Project. Specifically, we reviewed agreements and contracts with project participants to develop homemade explosive training aids to broaden threat detection, evaluate the effectiveness of canine teams in detecting explosives in air cargo, and conduct research on breeding and genetics to select canines best suited for explosives detection. We also discussed the status of these efforts with DHS S&T officials.

We conducted this performance audit from January 2012 through January 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix III: Comments from the Department of Homeland Security

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Steve Morris, Assistant Director, and Lisa Canini, Analyst-in-Charge, managed this review. Josh Diosomito and Michelle Woods made significant contributions to the work. Lydia Araya, Chuck Bausell, Michele Fejfar, Richard Hung, Thomas F. Lombardi, Douglas Manor, Linda Miller, Erin O’Brien, Jessica Orr and Jessica Smith also contributed to this report.