Advanced Imaging Technology: TSA Needs Additional Information before Procuring Next-Generation Systems
Released: 2014-04-30T00:00:00

Letter

Background

Roles and Responsibilities

DHS and TSA share responsibility for the research, development, and deployment of passenger checkpoint screening technologies. The Aviation and Transportation Security Act established TSA as the federal agency with primary responsibility for securing the nation’s civil aviation system, which includes the screening of all passengers and property transported to, from, and within the United States by commercial passenger aircraft. Additionally, the Homeland Security Act of 2002 established DHS and, within it, the Science and Technology Directorate for, among other things, conducting research, development, demonstration, and testing and evaluation activities relevant to DHS.DHS’s Science and Technology Directorate is responsible for testing and evaluating aviation security technologies, including AIT systems, at the TSL on behalf of TSA.

Types of Testing Conducted on AIT-ATR Systems

DHS and TSA conducted five types of tests to evaluate the performance of AIT-ATR systems.

Qualification testing. TSL conducted qualification tests in a laboratory setting to evaluate the technology’s capabilities against TSA’s procurement specification and detection standard that specified the required detection rate AIT systems must meet in order to qualify for procurement. Qualification tests evaluate the technology’s detection of threat items that are not artfully concealed as they are in covert tests, but do not test the entire system, including the SO’s interpretation and resolution of alarms. Qualification testing also includes testing of the system’s false alarm rate. For the purposes of this report, we refer to qualification testing as laboratory testing.

Operational testing. TSA conducted operational tests that assessed the technology’s detection performance, called threat-inject tests, at airports to evaluate the AIT-ATR systems’ ability to function in an operational environment. Operational testing also assesses how well AIT systems are suited for use in a real-world, aviation checkpoint environment after systems have successfully completed qualification testing in a laboratory setting. For example, operational testing includes determining whether the system interfered with other equipment fielded at the checkpoint and whether the system met TSA’s requirements. Further, DHS’s acquisition policy requires that operational tests be conducted prior to an agency procuring a technology. According to TSA testing documentation, threat- inject tests are not intended to evaluate effectiveness of the entire AIT- ATR system, which includes the technology, the personnel who use the technology, and the processes that govern screening, in an operational setting.

Covert testing. TSA’s Office of Inspection and the DHS Office of Inspector General conducted covert tests of AIT-ATR systems at the passenger checkpoint to identify vulnerabilities in TSA’s screening process. According to TSA officials, those tests were intended to identify weaknesses in the technology, the operators who used it, and TSO compliance with SOPs by artfully concealing threat objects intended to simulate a likely terrorist attack.

Performance assessments. TSA conducted covert performance assessments of TSO compliance with SOPs, under the Aviation Screening Assessment Program (ASAP), which TSA uses as a standard performance measurement for the Office of Management and Budget. According to TSA officials, ASAP assessments determine SO adherence to TSA’s SOPs and are not intended to test AIT-ATR system capabilities.

Checkpoint drills. In accordance with TSA’s IED checkpoint drill operational directive, TSA requires personnel at airports to conduct drills to assess TSO compliance with TSA’s screening SOPs and to train TSOs to better resolve anomalies identified by AIT-ATR systems. TSA conducts those drills at airports using test kits that contain inert bombs, bomb parts, and other threat items. According to TSA officials, IED checkpoint drills assess SO adherence to TSA’s SOPs and are not intended to test AIT-ATR system capabilities.

TSA’s Screening Process

TSA uses a multilayered security strategy aimed to enhance aviation security. Within those layers of security, TSA’s airport passenger checkpoint screening system includes, among other things, (1) screening personnel; (2) SOPs that guide screening processes conducted by TSOs; and (3) technology, such as AIT-ATR systems, used to conduct screening According to TSA, those elements collectively determine of passengers.the effectiveness and efficiency of passenger checkpoint screening. In strengthening one or more elements of its checkpoint screening system, TSA aims to balance its security goals with the need to efficiently process passengers.

Passenger screening is a process by which TSOs inspect individuals and their property to deter and prevent an act of violence, such as carrying an explosive, weapon, or other prohibited item onboard an aircraft or into the airport sterile area—in general, an area of an airport for which access is controlled through screening of persons and property. individuals for prohibited items at designated screening locations, referred to as checkpoints, where TSOs use technology and follow SOPs to screen passengers. According to TSA’s SOP for AIT-ATR systems, three TSOs are required to operate lanes equipped with AIT systems: one divestiture officer (of either gender), one male SO, and one female SO.

See 49 C.F.R. § 1540.5.

TSA’s Detection Standard and Tier Levels

As we reported in January 2012, TSA’s requirements for the AIT system have evolved over time. TSA continued to use those revised requirements to determine whether the AIT-ATR system met the agency’s needs. Additionally, TSA used those requirements to evaluate the next generation of AIT systems, referred to as AIT-2. Further, TSA’s requirements for AIT systems are based on tiers that correspond to the relative size of items that the AIT system must identify and requirements that the AIT system must meet, with Tier I being the level currently deployed AIT systems already meet and Tier IV being TSA’s anticipated goal for AIT systems to meet. TSA’s procurement of AIT-2 systems requires vendors to ensure AIT-2 systems meet Tier II requirements and provide faster throughput, among other things. TSA plans to seek proposals from AIT-2 vendors to provide Tier III and Tier IV capabilities by time frames specified in its AIT roadmap. TSA did not initially plan for AIT- IO systems to meet levels beyond Tier III, but included Tier IV in response to our recommendation.

TSA Does Not Collect or Analyze Data That Could Enhance System Performance

TSA does not collect or analyze three types of available information that could be used to enhance the effectiveness of the entire AIT-ATR system. First, TSA does not collect or analyze available airport-level IED checkpoint drill data on SO performance at resolving alarms detected by the AIT-ATR system to identify weaknesses and enhance SO performance at resolving alarms at the checkpoint. Second, TSA is not analyzing AIT-ATR systems’ false alarm rate in the field using data that could help it monitor the number of false alarms that occur on AIT-ATR systems to help monitor the potential impacts that AIT-ATR systems may have on operational costs. Third, TSA assesses the overall AIT-ATR system performance using laboratory test results that do not reflect the combined performance of the technology, the personnel that operate it, and the process that governs AIT-related security operations.

TSA Does Not Collect or Analyze Airport Data on SO Performance on AIT- ATR Systems

TSA does not collect or analyze IED checkpoint drill data, because it does not ensure compliance with its operational directive that requires each airport to conduct IED checkpoint drills each week. Specifically, the operational directive, originally issued in February 2010 and updated in November 2012, requires TSA personnel at airports to conduct a certain number of IED drills per checkpoint lane every week at each airport. The total number of drills per pay period must be split evenly between carry- on baggage and passenger screening. Additionally, for those airports equipped with AIT systems, a certain percentage of on-person drills must be conducted on AIT systems and a certain percentage must be conducted on walk-through metal detectors.

TSA is not enforcing compliance with its directive, and as a result, data on SO performance are not being consistently collected or reported by approximately half of airports with AIT-ATR systems. For example, according to TSA data, we found that TSA personnel at almost half of the airports with AIT-IO or AIT-ATR systems did not report any IED checkpoint drill results on those systems from March 2011 through February 2013. Of the airports at which TSA personnel conducted IED checkpoint drills, the number of drills conducted by TSA personnel at airports varied from 1 to 8,645. Further, roughly four-fifths of the on- person IED drills were conducted by screening passengers with metal detectors, with the rest of the IED drills conducted by screening passengers with AIT systems, which did not comply the directive’s specified requirements on the number of drills that must be conducted on each type of technology.

According to TSA officials, TSA’s Office of Security Operations is responsible for overseeing compliance with the operational directive at airports, but it does not analyze the IED checkpoint drill data at the headquarters level. Further, TSA officials told us that TSA formerly tracked the number of IED checkpoint drills in a monthly management report for federal security directors, but in fiscal year 2012, that report was replaced by an executive scorecard that tracks each airport’s IED checkpoint drill pass rate but does not include the number of drills conducted. TSA officials stated that federal security directors could conduct very few drills that are easy for SOs to identify in order to achieve a high pass rate, since the details of the drills are not provided to headquarters or analyzed beyond the pass rate. According to TSA officials, the agency does not ensure compliance with the directive at every airport because it is unclear which office within the Office of Security Operations should oversee enforcing the operational directive. According to officials from TSA’s Office of Training and Workforce Engagement, that office had the ability to monitor the program until TSA began using federal security director scorecards in 2012, which are reviewed by the Office of Security Operations. As a result, it is still unclear which office is ultimately responsible for overseeing whether TSA is in compliance with the operational directive at airports.

Data on IED checkpoint drills could provide insight into how well SOs resolve anomalies detected by the AIT systems, information that could be used to help strengthen the existing screening process. By not clarifying which office is responsible for overseeing TSA’s IED checkpoint drills operational directive, directing that office to ensure enforcement of the directive in conducting these drills, and analyzing the data, TSA is missing an opportunity to identify any potential weaknesses in the screening process, since performance depends in part on the ability of SOs to accurately resolve anomalies.

TSA Does Not Analyze AIT-ATR Systems’ False Alarm Rate in the Field

TSA is not analyzing available data on the number of secondary screening pat-downs that SOs conduct as a result of an AIT-ATR system alarm, which indicates that it has detected an anomaly. Analyzing this information could provide insight into the number of false alarms that occur in the field, which may affect operational costs. Specifically, when the AIT-ATR system identifies the presence of an anomaly, indicated by an alarm, the SO must resolve the anomaly by conducting a pat-down to determine whether the anomaly is a threat item. If the SO does not resolve the anomaly during the pat-down (i.e. by locating an item in the location identified by the AIT-ATR system alarm), this may be attributed to either a false alarm (the AIT-ATR system identified an anomaly when none actually existed) or SO error (the SO did not identify an anomaly that was present). By not analyzing such operational data, TSA is limited in its understanding of the operational effectiveness of deployed AIT-ATR systems.

TSA collected information on false alarm rates through laboratory testing conducted at TSL. These laboratory test results demonstrated that AIT- ATR systems have a higher false alarm rate than AIT-IO systems. Our analysis showed that the AIT-ATR system’s false alarm rate can be expected to range significantly based on the estimate’s 95 percent confidence interval, which could have implications for SO performance at resolving alarms and operational costs. Although TSA’s detection standard required AIT-ATR systems to meet a specific false alarm rate, TSL laboratory test results on the AIT-ATR system indicate that certain factors, such as body mass index (BMI) and headgear, such as turbans and wigs, may contribute to greater fluctuations in the false alarm rate, either above or below that threshold. For example, the false alarm rate for passengers with a normal BMI was less than the false alarm rates for overweight and obese passengers. Additionally, the AIT-ATR system had a higher false alarm rate when passengers wore turbans and wigs. While TSA did not include the false alarm rate as a key performance requirement that could be used as a basis to accept or reject AIT-ATR systems, higher false alarm rates could result in higher operational costs. According to TSA, the AIT-ATR systems’ current false alarm rate could produce an increase in annual staffing costs in the field, but it has not conducted studies on this issue. According to DHS’s Science and Technology Directorate, effective checkpoint screening technologies have lower false alarm rates, as well as higher throughput and lower costs of operations, which enhance the effectiveness and efficiency of how TSA screens passengers.

TSA’s Functional Requirements Document stated that AIT-ATR systems must have a data collection and reporting system that collects, stores, analyzes, and displays a summary report on the outcomes of scans. The AIT-ATR systems are required to provide, at a minimum, the total number of passengers scanned, total number of passengers on which the system detected anomalies, and the body location of where an anomaly was detected. TSA reported in its System Evaluation Report that the AIT-ATR system was equipped with that data collection and reporting system and the summary report. According to TSA, it verified that currently deployed AIT-ATR systems capture those data in operational testing and evaluation. However, TSA does not collect or analyze those data at headquarters. Rather, TSA gives TSA management at airports the discretion to determine how to use those data and whether to enter those data into TSA’s centralized information management system. TSA officials agreed that collecting and analyzing operational data would provide useful information related to the impact of false alarm rates on operational costs, and collecting those data could be done on a selective basis so that it would not be too labor-intensive. According to TSA officials, TSA is in the process of networking all AIT-ATR systems so that information can be collected at the headquarters level, and when this process is complete, TSA would be able to centrally collect operational data that could provide information on secondary screening outcomes, which provide insight into the operational false alarm rate. TSA officials were not able to provide an estimate of when this will be completed.

Given the potential staffing implications associated with a higher false alarm rate, it is important to fully understand the system’s false alarm rate in the field. Without a complete understanding of how the systems perform in the field, TSA may be at risk of incurring significantly higher operational costs than anticipated. Although TSA officials stated that collecting such data could be labor-intensive if not collected selectively, the agency agreed that evaluating operational screening data in the field could provide useful information, and that data could be collected in such a way that it does not negatively affect operations. Standards for Internal Control in the Federal Government calls for agencies to identify, capture, and distribute operational data to determine whether an agency is By not establishing meeting its goals and effectively using resources.protocols that facilitate capturing operational data on passengers at the checkpoint once the AIT-ATR systems are networked together, TSA is unable to determine the extent to which AIT-ATR system false alarm rates affect operational costs and has less information for its decision- making process related to checkpoint screening.

Assessments of AIT-ATR System Performance Do Not Include the Performance of All System Factors

According to TSA officials, checkpoint security is a function of technology, people, and the processes that govern them, but TSA does not include measures for each of those factors in determining overall AIT-ATR system performance. TSA evaluated the technology’s performance at meeting certain requirements in the laboratory to determine system effectiveness. Laboratory test results provide important insights but do not accurately reflect how well the technology will perform in the field with actual human operators. Figure 1 illustrates the multiple outcomes of the AIT-ATR screening process. Although TSA conducted operational tests on the AIT-ATR system prior to procurement, TSA does not assess how anomalies are resolved by considering how the technology, people, and processes function collectively as an entire system when determining AIT-ATR system performance.

TSA officials agreed that it is important to analyze performance by including an evaluation of the technology, operators, and processes, and stated that TSA is planning to assess the performance of all layers of security. According to TSA, the agency conducted operational tests on the AIT-ATR system, as well as follow-on operational tests as requested by DHS’s Director of Operational Test and Evaluation, but those tests were not ultimately used to assess effectiveness of the operators’ ability to resolve alarms, as stated in DHS’s Director of Operational Test and Evaluation’s letter of assessment on the technology. TSL officials also agreed that qualification testing conducted in a laboratory setting is not always predictive of actual performance at detecting threat items. Further, laboratory testing does not evaluate the performance of SOs in resolving anomalies identified by the AIT-ATR system or TSA’s current processes or deployment strategies.

According to best practices related to federal acquisitions, technologies should be demonstrated to work in their intended environment. According to DHS’s Acquisition Directive 102-01 and its associated guidebook, operational testing results should be used to evaluate the degree to which the system meets its requirements and can operate in the real world with real users like SOs. TSL’s Test Management Plan for AIT systems stated that effectiveness must reflect performance under realistic or near-realistic operating conditions. Additionally, a group of experts on testing best practices assembled by the National Academy of Sciences concluded that agencies should include the human element when evaluating system performance. That group of experts also determined that agencies should determine system effectiveness by conducting performance testing in an operational setting in addition to laboratory testing, which could include SOs during testing.

TSA conducted operational tests, but it did not use those tests to determine AIT-ATR effectiveness. Instead, TSA used laboratory tests that did not factor in performance of the entire system that includes technology, people, and processes. However, AIT-ATR system effectiveness relies on both the technology’s capability to identify threat items and its operators to resolve those threat items. Given that TSA is seeking to procure AIT-2 systems, DHS and TSA will be hampered in their ability to ensure that future procurements meet mission needs and perform as intended at airports without measuring system effectiveness based on the performance of the AIT-2 technology and SOs who operate the technology, while taking into account current processes and deployment strategies.

TSA Has Enhanced Passenger Privacy, but Could Take Additional Steps to Enhance AIT Capabilities

TSA has enhanced passenger privacy by completing the installation of ATR software upgrades for all deployed AIT systems but could do more to provide enhanced AIT capabilities to meet the agency’s mission needs. Moreover, the agency faces technological challenges in meeting its goals and milestones pertaining to enhancing AIT capabilities.

TSA Has Enhanced Privacy by Upgrading All Deployed Systems with ATR Software, but Has Not Met Its Goals Pertaining to Enhancing AIT Capabilities

TSA has met milestones as documented in its roadmap pertaining to the installation of ATR software upgrades that were intended to address privacy concerns and improve operational efficiency for all deployed AIT systems in accordance with the statutory deadline included as part of the Federal Aviation Administration Modernization and Reform Act of 2012.However, it did not meet proposed milestones documented in its AIT roadmap to provide enhanced capabilities to meet the agency’s mission needs. For example, the February 2012 AIT roadmap estimated that TSA would complete installation of Tier II ATR software upgrades for currently deployed AIT systems by December 2012. TSA’s updated October 2012 AIT roadmap revised this date to March 2013. According to TSA testing documentation, during operational testing conducted from May through June 2012 at an airport test site, the AIT-ATR Tier II system demonstrated limitations due to noncompliance with certain requirements.

Accordingly, TSA decided not to pursue fielding of the Tier II system based on particular deficiencies identified during operational testing. The vendor of this system submitted a new version of the AIT-ATR system for laboratory testing to TSL. In September 2013, the new version had passed laboratory testing and was undergoing operational test and evaluation. As shown in figure 2, TSA began operational test and evaluation for Tier II upgrades 17 months after the expected start date articulated in its October 2012 roadmap. According to TSA, it completed operational test and evaluation in January 2014. According to the timeframes in TSA’s revised roadmap, it would take an additional 7 months from January 2014 to complete Tier II upgrades. However, TSA had estimated that it would provide Tier III capabilities by the end of fiscal year 2014.

Although TSA experienced challenges and schedule slippages related to meeting Tier II requirements for the currently deployed AIT systems, in September 2012, TSA made contract awards to purchase and test the next generation of AIT systems (referred to as AIT-2) from three vendors. These systems are required to be equipped with ATR software and must be capable of meeting enhanced requirements (qualified at least at the Tier II level), among other things. The updated October 2012 roadmap contained milestones for testing and acquiring AIT-2 systems, which TSA has not met. Specifically, TSA is about 9 months behind schedule for AIT-2 testing and procurement, as depicted in figure 3. For example, the roadmap indicated that TSA would begin qualification testing and evaluation for AIT-2 during the first quarter of fiscal year 2013, would complete that testing by January 2013, and would complete deployment by March 2014. However, TSA did not initiate qualification testing until July 2013 (about 9 months behind schedule) because all three vendors had difficulty providing qualification data packages verifying that the vendors had met contractual requirements and the systems were ready to begin testing. Accordingly, as of March 2014, TSA is not on track to meet the March 2014 deployment milestone and these efforts have not resulted in enhancing AIT capabilities because currently deployed AIT- ATR systems are qualified at the same Tier I level as the systems originally deployed in 2009.

We have reported in the past few years that although AIT systems and the associated software have been in development for over two decades, TSA has faced challenges in developing and meeting program requirements in some of its aviation security programs, including AIT. Best practices for acquisition programs state that when key technologies are immature at the start of development, programs are at higher risk of being unable to deliver on schedule. As we concluded in January 2012, at the start of AIT development, TSA did not fully adhere to DHS acquisition guidance, and procured AIT systems without meeting all key requirements. According to best practices on major acquisitions, realistic program baselines with stable requirements for cost, schedule, and performance are important to delivering capabilities within schedule and cost estimates.

In its AIT roadmap, TSA describes the time frames as notional and explains that establishing definitive timelines for reaching defined, additional tiers is difficult to achieve because of intricate dependencies that are outside of the program’s control and may vary by manufacturer. However, TSA officials stated that they did not use available scientific research or evidence to help assess how long it would take to develop enhanced capabilities. In setting these time frames, TSA officials told us that TSA did not seek input from national laboratories that have conducted technology assessments and explosives research on behalf of DHS’s Science and Technology Directorate nor did it evaluate vendor data to determine the capabilities of the technology. According to experts we interviewed from Sandia National Laboratories, to accurately determine realistic time frames in which vendors would be able to provide enhanced capabilities, it would require an evaluation of proprietary vendor data to understand how well the technology can meet requirements at a specific tier level. Rather, according to TSA officials, since TSA did not have access to proprietary data, it relied on notional time frames proposed by the AIT vendors, which comprised estimates for when the vendors expected to be able to develop and deliver AIT systems that would meet TSA’s requirements.

TSA’s October 2012 AIT roadmap contains one key element of a technology roadmap—estimated time frames for achieving each milestone—and does not describe steps or activities needed to achieve each milestone. Moreover, in April 2012, the vendor for currently deployed AIT systems provided TSA with a detailed plan for delivering a system that could meet Tier III requirements that contained proposed milestones and time frames for achieving each milestone. Although TSA relied on discussions with this vendor to estimate roadmap time frames, the agency did not incorporate details from the vendor’s plan into its roadmap. According to a representative from this vendor, TSA did not consult with the vendor regarding the risks and limitations of its proposed time frames, including how long it might take to develop various hardware or software modifications, nor did it provide feedback to the vendor after the proposal was submitted. The vendor’s April 2012 plan states that after the Tier II system has met TSA’s requirements, it would take the vendor several years to develop and deliver a Tier III system for TSA to test, followed by an operational test and evaluation system validation phase that would take several months. In addition, according to experts we interviewed from the national laboratories that contributed to the development of imaging technology, the milestones contained in TSA’s October 2012 roadmap are not achievable because it did not reflect the time needed to make sufficient improvements to the technology to ensure that it would be able to meet additional tier levels. TSA did not incorporate available information from the national laboratories and vendors into its updated roadmap. As a result, the roadmap underestimated the length of time it would take to develop and deploy AIT-ATR Tier III systems.discussed later in this report, moving forward, it will be important for TSA to incorporate scientific evidence and information from DHS’s Science and Technology Directorate, and the national laboratories, as well as nonproprietary information and data provided by vendors into the next revision of its AIT roadmap to ensure that the time frames for achieving future goals and milestones are realistic and achievable.

TSA Faces Technological Challenges in Meeting Future Goals and Milestones

Consistent with the Homeland Security Act of 2002, as amended, the DHS Science and Technology Directorate has responsibility for coordinating and integrating the research, development, demonstration, testing, and evaluation activities of the department, as well as for working with private sector stakeholders to develop innovative approaches to produce and deploy the best available technologies for homeland security missions. Moreover, we have previously identified key practices that can help sustain agency collaboration and concluded that collaborating agencies can look for opportunities to address resource needs by leveraging each others’ resources, thus obtaining additional benefits that would not be available if they were working separately. According to TSA officials, the agency recognizes the need to develop achievable milestones based on scientific evidence and is in the process of developing a roadmap for the entire passenger screening program. They explained that they plan to collaborate with DHS Science and Technology Directorate to determine milestones for the new roadmap that will be based on a scientific analysis of technology capabilities as well as ongoing research and development efforts. TSA officials stated that they plan to update the AIT roadmap using this new approach and expect the AIT roadmap to be completed by September 30, 2014. A group of experts moderated by GAO in June 2013 stated that DHS must have personnel with technical expertise in ATR software for AIT systems and development who are engaged throughout the developmental process to ensure that vendors are providing improved capabilities over time. According to these expert comments, it is important to leverage the technical expertise of academia and the national laboratories to improve capabilities over time and provide insight into reasonable time frames for meeting future tiers.

In September 2011, we reported that given continuing budget pressures combined with the focus on performance envisioned in the Government Performance and Results Act (GPRA) Modernization Act of 2010, federal agencies must undertake fundamental reexaminations of their operations and programs to identify ways to operate more efficiently. While there are various approaches that vendors could take to make needed improvements to the technology, including hardware modifications, software developments, or incorporating new imaging techniques to provide enhanced capabilities, these approaches could take years to develop, and would require significant investment of resources. Moreover, according to scientists that we interviewed from the national laboratories, there are several ways to improve ATR software algorithms to enhance system capabilities; however, there is little market incentive for existing vendors to invest in making these improvements or for new vendors to enter the relatively small airport checkpoint market, since one vendor has already met TSA’s current requirements. Further, 2 of the 12 experts identified by the National Academy of Sciences with whom we spoke stated that establishing clear requirements would incentivize vendors to improve performance over time. Thus, according to these experts, it is unlikely that vendors will invest in making the needed improvements to meet TSA’s mission needs.

According to a representative from the vendor of currently deployed AIT systems, moving from Tier II to Tier III presents new technological challenges because meeting additional tiers will require the development of more targeted algorithms. Accordingly, to develop these new algorithms, vendors would have to build new data sets, conduct research, and invest additional resources before accurately determining realistic time frames for meeting Tier III and Tier IV requirements. Therefore, given the current state of the technology as well as the amount of research that has to be conducted on developing algorithms that can meet Tier III and Tier IV requirements, neither TSA nor the AIT vendors can reliably predict how long it will take to meet Tier IV requirements.

Because TSA revised its requirements over time, scientists from the national laboratories noted that vendors have little incentive to meet additional tier levels since they are meeting TSA’s current requirements. In addition, TSA has not obtained the necessary information to accurately understand the future state of the technology. Thus, the agency has little assurance that vendors will provide AIT-ATR systems that meet Tier IV requirements within TSA’s estimated time frames. As a result, the future capabilities of the technology and the time frames in which those capabilities will be delivered remain unknown. Given these challenges, TSA will be unable to ensure that its roadmap reflects the true capabilities of the next generation of AIT-2 systems without the use of scientific evidence and information from DHS’s Science and Technology Directorate, and the national laboratories, as well as nonproprietary information and data provided by vendors to develop a realistic schedule with achievable milestones that outlines the technological advancements, estimated time, and resources needed to achieve TSA’s Tier IV end state.

Conclusions

TSA has deployed nearly 740 AIT systems and will spend an estimated $3.5 billion in life cycle costs on deployed AIT-ATR systems and future AIT-2 systems. However, TSA faces challenges in managing its AIT program because it is not using all available data that it collects to inform its decisions. For example, TSA does not enforce compliance with its operational directive that requires each airport to conduct IED checkpoint drills each week, nor does it collect or use IED checkpoint drill data on SO performance. Additionally, TSA is not analyzing available data on the number of secondary screening pat-downs that SOs conduct when the system indicates that it has detected an anomaly, which could provide insight into the number of false alarms that occur in the field and the extent to which these alarms affect operational costs. TSA could improve the overall performance of the AIT system and better inform its decision- making process related to checkpoint screening by clarifying which office is responsible for overseeing TSA’s operational directive, directing that office to enforce compliance with the directive, and analyzing the IED checkpoint data to identify any potential weaknesses in the airport screening process, and also establishing protocols that facilitate capturing operational data on passengers at the checkpoint to determine the extent to which AIT-ATR system false alarm rates affect operational costs.

Although AIT systems and the associated software have been in development for over two decades, TSA has not used available information from the scientific community and vendors to understand the technological advancements that need to be made and determine the time frames in which AIT systems will meet Tier IV requirements. Therefore, the milestones that TSA uses to guide its procurement of this technology do not incorporate scientific evidence from the national laboratories or vendors that could be used to produce an accurate, realistic roadmap. TSA would have more assurance that its $3.5 billion investment in AIT provides effective security benefits by (1) measuring system effectiveness based on the performance of the AIT-2 technology and SOs who operate the technology, while taking into account current processes and deployment strategies and (2) using scientific evidence and information from DHS’s Science and Technology Directorate, and the national laboratories, as well as information and data provided by vendors, to develop a realistic schedule with achievable milestones that outlines the technological advancements, estimated time, and resources needed to achieve TSA’s Tier IV end state.

Recommendations for Executive Action

To help ensure that TSA improves SO performance on AIT-ATR systems and uses resources effectively, the Administrator of the Transportation Security Administration should take the following two actions: clarify which office is responsible for overseeing TSA’s IED screening checkpoint drills operational directive, direct the office to ensure enforcement of the directive in conducting these drills, and analyze the data to identify any potential weaknesses in the screening process, and establish protocols that facilitate the capturing of operational data on secondary screening of passengers at the checkpoint to determine the extent to which AIT-ATR system false alarm rates affect operational costs once AIT-ATR systems are networked together.

To help ensure that TSA invests in screening technology that meets mission needs, the Administrator of the Transportation Security Administration should ensure that the following two actions are taken before procuring AIT-2 systems: measure system effectiveness based on the performance of the AIT-2 technology and screening officers who operate the technology, while taking into account current processes and deployment strategies, and use scientific evidence and information from DHS’s Science and Technology Directorate, and the national laboratories, as well as information and data provided by vendors to develop a realistic schedule with achievable milestones that outlines the technological advancements, estimated time, and resources needed to achieve TSA’s Tier IV end state.

Agency Comments and Our Evaluation

We provided a draft of this report to DHS for comment. On March 21, 2014, DHS provided written comments, which are reprinted in appendix III and provided technical comments, which we incorporated as appropriate. DHS generally concurred with our four recommendations and described actions taken, underway, or planned, to implement each recommendation. Specifically, In response to the recommendation that TSA clarify which office is responsible for overseeing TSA’s Improvised Explosive Device Screening Checkpoint Drills operational directive, instruct the responsible office to enforce the directive, and analyze the drill data to identify any potential weaknesses in the screening process, DHS stated that TSA’s Office of Security Operations will initiate a review of programs that contribute to assessing screening performance with consideration of the findings identified in our report. TSA anticipates that it will complete this review by the end of fiscal year 2014, and by TSA also stated that by September 30, 2014, the operations directive will be amended to assign responsibility to one office. We believe that these are beneficial steps that would address our recommendation, provided that TSA directs the office to ensure enforcement of the directive in conducting the drills, and uses the data to identify any potential weaknesses in the screening process, as we recommended.

In response to our recommendation that TSA establish protocols to help determine the extent to which AIT-ATR system false alarm rates affect operational costs once AIT-ATR systems are networked together, DHS stated that TSA will monitor, update, and report the results of its efforts to capture operational data on the secondary screening of passengers resulting from AIT-ATR false alarms and evaluate the associated impacts to operational costs based on existing staffing levels. Once implemented, the new reporting mechanism will address our recommendation, provided that it captures sufficient information to determine the extent to which AIT- ATR system false alarm rates affect operational costs.

In response to the recommendation that TSA measure system effectiveness based on the performance of the AIT-2 technology and screening officers who operate the technology, while taking into account current processes and deployment strategies before procuring AIT-2 systems, DHS stated that TSA considers several factors when measuring system effectiveness, including documented deployment strategies, airport needs and conditions such as height and checkpoint space, TSA security operations processes and procedures, feedback from transportation security officers who operate the AIT-ATR systems, as well as concept of operations and formal operational and functional requirements documents. Further, DHS stated that TSA’s testing process enables TSA to determine if technologies meet required standards and are feasible for use in the airport environment, and that the system evaluation report for AIT-2— which will document system effectiveness using information from the laboratory and operational test reports—will state whether or not the next-generation AIT system has an acceptable operationally effective and suitable rating for use within an airport environment. While these are beneficial practices, we believe that it would be preferable for TSA to measure the AIT-2 system’s overall probability of detection by including an evaluation of screening officer performance at resolving alarms detected by the technology in its assessment, as we recommended, since AIT system effectiveness relies on both the technology’s capability to detect items and screening officers ability to correctly resolve alarms. In addition, DHS stated that TSA is currently implementing the Transportation Security Capability Analysis Process, which will be used to better understand TSA’s requirements and better articulate those requirements and needs for acquisition and requirements documentation. This is an important first step toward addressing our recommendation, provided that TSA uses this process to determine the overall effectiveness of its system based on the performance of the AIT-2 technology as well as the screening officers who operate the technology and not solely on the capabilities of current AIT technology as has been done in the past.

In response to the recommendation that TSA use scientific evidence and information from DHS’s Science and Technology Directorate, and the national laboratories, as well as information and data provided by vendors to develop a realistic schedule with achievable milestones that outline the technological advancements, estimated time, and resources needed to achieve TSA’s Tier IV end state, DHS stated that TSA has initiated an effort to complete a more comprehensive technology roadmap that forecasts technology progression through detection tiers, estimates cost to mature the technology, and includes a timeline with supporting narrative. TSA expects this roadmap to be completed by September 30, 2014. We believe that these are beneficial actions that could help TSA address the weaknesses identified in this report and we will continue to work with TSA to monitor progress on the proposed steps as the agency progresses.

As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after its issue date. At that time, we will send copies of this report to the Secretary of Homeland Security, the TSA Administrator, the House Homeland Security Committee, the House Subcommittee on Oversight and Management Efficiency, the House Subcommittee on Transportation Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4379 or lords@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Transportation Security Administration (TSA) Has Taken Action toward Addressing Advanced Imaging Technology Utilization and Planning Challenges Identified in our January 2012 Report

In January 2012, we concluded that TSA had acquired advanced imaging technology (AIT) systems that were not being used on a regular basis and thus were not providing a security benefit. For example, we found that 32 of 486 AIT systems had been used less than 5 percent of the days since their deployment, and that 112 of 486 AIT systems had been used on less than 30 percent of the days since their deployment. Further, we observed that at 5 of the 12 airports we visited, AIT systems were deployed but were not regularly used. For example, at 1 airport we observed that TSA had deployed 3 AIT systems in an area that typically handles approximately 230 passengers. TSA officials informed us at the time that 2 of the AIT systems were seldom used because of the lack of passengers and mentioned that they believed the AIT systems were deployed based on the availability of space. In addition, we observed instances in which AIT systems were not being used because of maintenance problems that affected how often the deployed AIT system screened passengers. We concluded, on the basis of our observations on AIT utilization, that there were concerns about how effectively deployed AIT systems were being used. Accordingly, we recommended that TSA evaluate the utilization of currently deployed AIT systems and potentially redeploy AIT systems based on utilization data, so that those systems not being extensively used could provide enhanced security benefits at airports. The Department of Homeland Security (DHS) agreed, and TSA has taken steps to address our recommendation but has not fully addressed the intent of our recommendation. Specifically, TSA took the following actions.

Develop and track AIT utilization metrics. TSA officials we spoke with in October 2012 stated that they revised TSA’s metric for measuring utilization based on our January 2012 report to more accurately reflect the amount of time AIT systems were being used. According to TSA’s field guide issued in March 2012, TSA measures AIT utilization as the percentage of passengers that are screened by AIT systems. To track AIT utilization based on this metric, TSA developed specific targets to meet that are based on passenger throughput and hours that AIT systems are in operation at an airport.

However, the target TSA establishes for an airport is reduced to account for AIT systems that are not operational because of maintenance problems or that are not being used because of lane closures, staffing restrictions, or low passenger volume. Accordingly, the methodology employed by TSA to measure AIT utilization does not accurately measure the extent to which AIT systems are being used since the metric tracks AIT system utilization only when they are being used. Furthermore, to calculate airport targets and track AIT utilization, TSA relies on data submitted by airports into its centralized information management system. However, in September 2013, the DHS Office of Inspector General (DHS OIG) reported that TSA did not have adequate internal controls to ensure accurate data on AIT utilization. Specifically, the OIG found that TSA’s utilization data were unreliable because (1) AIT throughput data recorded in its centralized information management system were different from data in the source document, (2) AIT throughput data on the source document were not recorded in its centralized information management system, (3) the starting AIT count was different from the previous day’s ending AIT count, and (4) AIT throughput source documentation was missing. Further, since airports record and enter AIT throughput in its centralized information management system manually, this may lead to inaccurate recording of information and does not provide an audit trail to validate data accuracy. Accordingly, without reliable throughput data, TSA decision makers cannot accurately measure AIT utilization at airports.

Backscatter X-ray technology uses a low-level X-ray to produce an X-ray image, while millimeter-wave technology beams the millimeter-wave radio-frequency energy over the body’s surface to produce a three-dimensional image. Since the backscatter vendor was unable to develop Automated Target Recognition (ATR) software by the June 2013 statutory deadline, as extended by TSA, to upgrade all deployed AIT systems with the software, TSA terminated its contract with this vendor and removed all of these systems from airports in order to meet the requirement. number of AIT systems that should be deployed to which airports. Accordingly, TSA is not using the data it collects on utilization to inform its deployment decisions. While the actions TSA has taken represent important steps toward addressing our recommendation, ensuring that the utilization data it collects are accurate, and using these data to inform future deployment decisions, would help ensure the effective utilization and redistribution of AIT systems and efficient use of taxpayer resources.

Appendix II: Objectives, Scope, and Methodology

This report answers the following questions: 1. To what extent does TSA collect and analyze available information that could be used to enhance the performance of AIT systems equipped with ATR (AIT-ATR)? 2. To what extent has TSA made progress toward enhancing AIT capabilities to detect concealed explosives and other threat items, and what challenges, if any, remain?

To determine the extent to which TSA collects and analyzes available information to improve the performance of screening officers (SO) responsible for resolving anomalies identified by ATR software, we analyzed improvised explosive device (IED) checkpoint drills conducted by TSA personnel at airports that submitted data to TSA from March 1, 2011, through February 28, 2013, under TSA’s IED checkpoint drill operational directive. TSA’s IED checkpoint drill operational directive requires personnel at airports to conduct drills to assess Transportation Security Officer (TSO) compliance with TSA’s screening standard operating procedures (SOP) and to train TSOs to better resolve anomalies identified by AIT-ATR systems. We analyzed those data to determine whether airports were in compliance with TSA’s operational directive by analyzing the number and percentage of tests that were conducted on AIT systems and on other passenger screening methods at the checkpoint to evaluate whether, overall, airports with AIT systems had conducted the required proportion of drills between AIT drills and other passenger-screening drills. Additionally, we evaluated airport compliance with TSA’s operational directive and Standards for Internal Control in the Federal Government to determine the extent to which TSA is monitoring We also reviewed TSA’s AIT deployment compliance with its directive.schedules to determine which type of AIT-ATR system airports had, the dates those systems were first deployed, and the dates systems were upgraded with ATR capability to assess how airport performance varied at resolving anomalies identified by the AIT-ATR system.

Further, we analyzed laboratory test results of the AIT-ATR system and the AIT systems that used IOs (AIT-IO) from calendar years 2009 through 2012 conducted by the Transportation Security Laboratory (TSL). We analyzed these data using statistical methods that estimated how the false alarm rates varied according to various characteristics of the mock passenger. We assessed whether the laboratory tests complied with statistical principles by comparing the testing design to generally accepted statistical principles used for data collection. We calculated the false alarm rates using two specific statistical calculations, called bias- corrected cluster bootstrap resampling and random effects methods, to estimate the sampling error of the AIT-ATR systems’ estimated false alarm rates. We used each of these methods to estimate the 95 percent confidence intervals of the false alarm rates, and achieved similar results using either method.

GAO, Homeland Security: DHS Requires More Disciplined Investment Management to Help Meet Mission Needs, GAO-12-833 (Washington, D.C.: Sept. 18, 2012). We identified key acquisition management practices by reviewing 17 prior GAO reports examining DHS, the Department of Defense, the National Aeronautics and Space Administration, and private sector organizations. and reviewing testing reports and related documentation. We determined these data were sufficiently reliable for the purposes of this report.

Furthermore, we compared the extent to which TSA evaluated the performance of the entire system to key acquisition practices established by GAO, DHS’s Acquisition Directive 102-01, and TSL’s Test Management Plan. We also visited a nonprobability sample of four U.S. airports to observe AIT-ATR systems and interview relevant TSA personnel. We interviewed a total of 46 TSA personnel who operate AIT- ATR systems selected by airport officials to obtain their views on system performance, and six Transportation Security Specialists for Explosives to discuss airport IED checkpoint drills. We selected these airports based on airport category and AIT-ATR system deployment. The information we obtained from these visits cannot be generalized to other airports, but provided us with information on the perspectives of various participants in the deployment of AIT units at airports across the country. We also interviewed TSA officials involved in AIT-ATR deployment, training, and covert testing. We visited TSL in Atlantic City, New Jersey, to interview laboratory scientists responsible for testing and evaluating AIT-ATR systems and reviewed TSL documentation related to laboratory test plans, records, and final reports. We interviewed knowledgeable agency officials from TSA, TSL, and DHS’s Science and Technology Directorate to better understand how AIT-ATR and AIT-IO system performance was assessed.

To determine progress TSA has made and any challenges that remain toward enhancing AIT capabilities, we analyzed TSA’s original AIT roadmap dated February 2012, as well as the October 2012 revision. To determine the extent to which TSA has met its projected time frames for AIT-ATR system upgrades and development of the next generation of AIT systems, referred to as AIT-2, we reviewed actions taken by TSA testing officials and compared the actual dates for each milestone with the estimated dates documented in TSA’s AIT roadmap. We also reviewed a leading AIT vendor’s technology plan for meeting additional tiers to determine the extent to which TSA’s AIT roadmap contained achievable time frames for meeting future tier levels. We further reviewed several technology roadmaps for large-scale acquisition programs developed by other agencies and organizations, such as the Department of Defense, as well as technology roadmapping guidance developed by Sandia National Laboratories to enhance our understanding of the fundamental elements of technology roadmaps. We then compared this guidance with TSA’s AIT roadmap to determine the extent to which TSA’s roadmap contained these elements. We also reviewed prior GAO reports on (1) major acquisition programs to identify best practices for delivering capabilities within schedule and cost estimates and (2) key practices that can help sustain agency collaboration to leverage each others’ resources and obtain additional benefits that would not be available if they were working separately.

To determine challenges TSA faces toward enhancing AIT capabilities, we interviewed scientists from the Department of Energy’s Sandia National Laboratories and Pacific Northwest National Laboratory to obtain their views on current and future capabilities of the technology and the scientific advancements that would need to occur to enable the development of future tier levels. We also interviewed a leading AIT vendor to obtain its views on the extent to which TSA obtained input from the vendor related to its ability to meet future tiers within expected time frames as well as the risks and limitations associated with pursuing alternative approaches for developing successive tiers. We further interviewed TSA acquisition officials to obtain the agency’s views on the vendors’ ability to meet future tiers within estimated time frames. Last, we interviewed 12 experts identified by the National Academy of Sciences to obtain their views on best practices for testing detection technologies, such as AIT-ATR systems. Our interviews with these experts are illustrative and provide insights about testing best practices.

We conducted this performance audit from September 2012 to March 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix III: Comments from the Department of Homeland Security

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Stephen M. Lord at (202) 512-4379 or at lords@gao.gov.

Staff Acknowledgments

In addition to the contact named above, David Bruno, Assistant Director; David Alexander; Carl Barden; Carissa Bryant; Susan Czachor; Emily Gunn; Tom Lombardi; Lara Miklozek; Tim Persons; Doug Sloane; and Jeff Tessin made key contributions to this report.