TSA Acquisitions: Further Actions Needed to Improve Efficiency of Screening Technology Test and Evaluation
Released: 2015-12-17T00:00:00

Letter

Background

TSA Acquires Security- Related Technologies to Screen Passengers and Baggage for Threats

TSA uses security-related technologies to help secure approximately 1.8 million passengers, 1.2 million checked bags, and 3 million carry-on bags on 25,000 flights at roughly 440 federally regulated airports every day. As of August 2015, TSA had deployed about 15,000 units of security-related technology to airports nationwide and anticipates spending a significant portion of its $3.6 billion security capability budget on technologies such as these over the next 5 years. Within TSA, the Office of Security Capabilities (OSC) provides security-related technology solutions through two major DHS acquisition programs, the Passenger Screening Program (PSP) and the Electronic Baggage Screening Program (EBSP):

PSP technologies, such as advanced imaging technology and bottled liquid scanners, work in combination at airport checkpoints to screen passengers and their carry-on baggage for threats.

EBSP technologies, such as explosives trace detectors and explosives detection systems, help ensure that TSA screens 100 percent of checked baggage for explosives, as mandated by law.

For each technology type, vendors develop their own specific version or system. See table 1 for an overview of key PSP and EBSP technology types.

According to TSA, EBSP and PSP are transitioning into a sustainment mode, meaning they are largely not procuring new types of screening technologies and are instead focused on the recapitalization of over 2,400 systems that are reaching their end-of-life over the next 5 years.

In our past work, we found key challenges related to TSA’s efforts to acquire technologies. For example, in March 2014, we found that TSA’s performance assessments of advanced imaging technology systems with automatic target recognition capability did not account for all factors affecting the effectiveness of the system, such as how well the system would perform with human operators, because the assessments were conducted in a laboratory. We recommended that TSA establish protocols to facilitate capturing operational data on secondary passenger screening at the checkpoint to determine the extent to which rates of false alarms for various advanced imaging technology systems affect operational costs once the systems with automatic target recognition capability are networked. TSA concurred with this recommendation. In its comments to our report, TSA stated that it will monitor, update, and report the results of its efforts to capture operational data and evaluate its associated impacts to operational costs. In January 2012, we found that TSA did not fully follow DHS acquisition policies when acquiring a different version of advanced imaging technology, which resulted in DHS approving nationwide deployment without full knowledge of TSA’s revised specifications. Among other things, we found that TSA had changed key performance parameters during the acquisition process without formally informing DHS acquisition officials, in violation of DHS policy. We recommended that TSA develop a road map describing when vendors will meet milestones for further developing advanced imaging technology and that TSA brief Congress as part of the budget process. Findings from this report resulted in a reduction in planned advanced imaging technology purchases amounting to approximately $1.4 billion.

DHS Acquisition Process

DHS policies and processes for managing its major acquisition programs, such as TSA’s PSP and EBSP, are primarily set forth in DHS Acquisition Management Directive 102-01 and DHS Instruction Manual 102-01-001, Acquisition Management Instruction/Guidebook. Within TSA, the PSP and EBSP program management offices are responsible for planning and executing the acquisition programs within the cost, schedule, and performance parameters in their program baselines, which are approved by DHS. DHS’s Deputy Secretary or Under Secretary for Management serve as the decision authority for PSP and EBSP acquisitions since both are large acquisition programs with life cycle cost estimates of over $1 billion. The decision authority reviews programs at five predetermined acquisition decision events to assess whether the programs are ready to proceed through each of the four acquisition life cycle phases.

During the first phase of the acquisition life cycle—the “need” phase— TSA develops a mission need statement for a potential solution to an identified problem, such as threats concealed on passengers and in their carry-on items. If the acquisition decision authority concludes that the need is of sufficiently high priority to DHS, the program moves to the second phase known as the “analyze/select” phase. The TSA program management offices—such as PSP and EBSP—communicate this need to potential vendors through requests for information, requests for proposals, and broad agency announcements in search of potential solutions, such as innovative screening technologies. Program managers select the best technology option and, if the acquisition decision authority approves the proposed technology, the program moves to phase three. The third phase is the “obtain” phase in which TSA develops, tests, and evaluates systems—vendors’ versions of the selected technology. With acquisition decision authority approval, programs enter the final phase, the “produce/deploy/support” phase, and TSA can proceed with procurement and full deployment of tested and qualified systems.

TSA Test and Evaluation Process

The goal of test and evaluation is to make sure that a product works as intended before it is provided to end-users, such as TSA’s transportation security officers. According to DHS policy, the primary purpose of test and evaluation is to provide timely, accurate information to managers, decision makers, and other stakeholders to reduce programmatic, financial, schedule, and performance risk. TSA’s test and evaluation process is guided by DHS and TSA policies. DHS issued its test and evaluation policy in May 2009 and TSA issued its policy in July 2010.

Within OSC, the Operations Support Division is responsible for testing and evaluating screening technologies in coordination with DHS’s Transportation Security Laboratory. Table 2 provides an overview of DHS and TSA test and evaluation roles and responsibilities.

As noted in the table, DHS and TSA both have responsibilities for evaluating vendors’ qualification data packages and determining whether to accept systems for testing. Once accepted, the systems undergo TSA’s three phase pre-acquisition test and evaluation process to verify requirements. Responsibility for verifying requirements is assigned to testers for each of the three phases: (1) qualification and certification testing at the Transportation Security Laboratory, (2) qualification testing at the TSA Systems Integration Facility, and (3) operational testing at selected airports. 1. Qualification and certification testing at the Transportation Security Laboratory. Qualification testing is the formal verification and validation of a system’s performance and must provide confidence that the system will satisfy desired capabilities in an operational environment. Qualification testing is typically conducted at two facilities, each serving distinct functions. Testing conducted at the DHS Science and Technology Directorate’s Transportation Security Laboratory in Atlantic City, New Jersey is focused on qualifying or certifying that a system meets the probability of detection of all categories of explosives while meeting TSA’s designated false alarm requirement. Typically, systems complete detection testing at the Transportation Security Laboratory prior to undergoing the balance of qualification testing at the TSA Systems Integration Facility. 2. Qualification testing at the TSA Systems Integration Facility. TSA conducts additional qualification testing at the TSA Systems Integration Facility in Arlington, Virginia to verify system performance against defined functional requirements, such as system capacities, human factors, physical characteristics, user safety, and system reliability, availability, and maintainability. This testing may include transportation security officers who exercise standard operating procedures to provide feedback regarding the system’s functions in a controlled environment. Systems must pass qualification testing and undergo an operational test readiness review before they can proceed to operational testing. 3. Operational testing at select airports. TSA conducts operational testing at select airports to evaluate a system’s operational effectiveness and suitability in a realistic environment and to ensure that the airport infrastructure is ready to accept the system. According to TSA, it selects test sites, in coordination with various stakeholders, using criteria intended to facilitate an unbiased evaluation of the systems. TSA injects mock threats during operational testing to assess performance of the entire system, including users and standard operating procedures. Operational tests focus on demonstrating that operational requirements have been met and that all critical operational issues have been resolved.

Following operational testing, TSA prepares a system evaluation report, which provides effectiveness and suitability determinations along with potential system improvement recommendations. The DHS Director of Operational Test and Evaluation reviews the report and prepares a letter of assessment. The letter assesses the system evaluation report and the adequacy of TSA’s operational test and concludes whether the system is operationally suitable and effective for procurement. TSA’s program management offices place approved systems on qualified products lists, which contain systems that have successfully completed the test and evaluation process and have been approved by DHS. Once a system is on a qualified products list, which TSA maintains for most technology types, such as explosives detection systems, the vendor can participate in TSA’s procurement process. However, placement on a qualified products list does not guarantee that the vendor will receive a procurement contract.

TSA conducts acceptance testing on systems at vendors’ factories following production and again once installed at airports to ensure consistency of the manufacturing process, system configuration, and functionality. Post deployment, TSA’s Office of Inspection and the DHS Office of Inspector General conduct covert tests in airports to identify vulnerabilities in screening processes. TSA and DHS test and evaluate systems prior to procurement, but covert testing has demonstrated that systems, when integrated into TSA’s screening process, do not always work as intended once deployed. TSA’s test and evaluation activities are funded by the program offices, and TSA’s life cycle cost estimate for EBSP reflects average annual testing costs of approximately $24 million. PSP officials estimate that the program averages $10 million in annual testing costs.

DHS and TSA Partner with Industry to Develop and Deploy Screening Technologies in a Dynamic Threat Environment

TSA’s security screening must adapt to meet evolving threats since potential terrorists use their knowledge of aviation security measures when planning aviation-related attacks. New explosives threats can result in heightened detection standards, which may require software upgrades to existing systems or the development of new technologies. DHS and TSA partner with industry to build, test, field, and sustain security-related technologies. For example, DHS makes available testing facilities and expertise at the Transportation Security Laboratory that vendors can use, by their choice, to further the development and evaluation of their systems. TSA’s August 2015 Strategic Five-Year Technology Investment Plan for Aviation Security noted the market for security-related technologies is quite limited, with only a handful of vendors, which creates a significant challenge for small businesses. According to TSA, substantial funding and time are required by industry to develop, qualify, and produce TSA’s screening technologies, which represents a significant barrier to entry. For example, the procurement cost of an explosives detection system can exceed over half a million dollars. TSA’s approximately 15,000 systems were manufactured by 13 different vendors, with 5 vendors accounting for approximately 81 percent of deployed systems.

TSA Has Identified Technologies to Meet Mission Needs, but Failures during Testing Have Contributed to Acquisition Inefficiencies

Consistent with departmental guidance and acquisition best practices, TSA’s test and evaluation process supports its acquisition decisions by providing information regarding the ability of passenger and baggage screening technologies to meet mission needs prior to full production decisions. From June 2010 to July 2015, only half of the 22 systems that TSA and DHS tested successfully passed qualification and operational testing and were therefore deemed effective and suitable for deployment. TSA procured all but one of the successful systems. Technology failures during testing, as a result of vendors’ immature technologies entering the test and evaluation process, often required significant fixes and have contributed to inefficiencies in TSA acquiring the technologies for use in airports.

Test and Evaluation Process Is Consistent with Guidance and Best Practices and Identifies Technologies That Meet Mission Needs

TSA’s test and evaluation process is a critical means of providing DHS and TSA officials with accurate information about security-related technologies to support acquisition decisions. We found that the process is consistent with DHS and TSA policies and guidance and helps TSA ensure that the technologies it acquires fulfill mission needs upon deployment. For example, TSA has integrated end users, such as transportation security officers into its test and evaluation process, consistent with TSA test and evaluation policy. Also, TSA officials said they have designated program office liaisons to improve communication between testing officials and the TSA program offices responsible for procuring the screening technologies to help ensure program offices have the testing information they need to inform the procurement decision, in line with DHS test and evaluation policy. In addition, TSA establishes system evaluation teams for each system undergoing test and evaluation; the teams consist of the TSA and DHS officials that plan for and evaluate system effectiveness and suitability throughout the testing process. Further, TSA’s test and evaluation process is designed to identify and evaluate existing technologies that can be adjusted or repurposed to meet the agency’s needs. This goal is in line with department guidance underlining the importance of pursuing technologies, specifically commercial off-the-shelf technologies that do not require a substantial investment of time and money to ensure they are effective once procured.

We previously found that the validation of product knowledge early in the acquisition process and before key investments are made is consistent with best practices by commercial firms. Specifically, leading commercial firms strive to detect as many problems as possible during testing, which in turn leads to easier and less expensive improvements to products down the road. Leading commercial firms highlighted the importance of perceiving problems during testing, not as failures, but as knowledge which can be used to improve a product. Consistent with this practice, while vendors’ proposed systems may fail TSA’s test and evaluation process, these failures can help the government identify needed fixes to better ensure that they will be effective and suitable prior to being deployed to airports nationwide. Further, we previously found that cost overruns and underperformance of technologies are exacerbated when problems discovered during testing are not resolved. For example, costly redesigns and retrofits could be required to achieve satisfactory performance of units already deployed in the field. By conducting operational testing on systems prior to procurement, TSA is working to prevent such issues.

Only Half of Screening Systems Tested in the Last 5 Years Passed Testing and TSA Procured Almost All of the Successful Systems

Of the 22 PSP and EBSP systems TSA and DHS tested from June 2010 to July 2015, 11 successfully completed qualification and operational testing, and TSA procured all but one of the 11. An additional 8 systems were tested during this period and testing remains ongoing. In addition, during this period one vendor withdrew its system from the testing process. These 9 systems are not depicted in figure 1 below, which shows the number of systems that made it through each stage of TSA’s test and evaluation process during this period.

Additional detail on the outcomes of vendors’ proposed systems follows.

Four additional systems were proposed by vendors during the 5-year period, but did not proceed to qualification testing because TSA did not accept the vendors’ qualification data packages. TSA officials told us they return nearly all qualification data packages to vendors at least once for additional clarification and documentation.

Four systems did not pass qualification testing and therefore did not proceed to operational testing. For example, one proposed passenger screening system did not meet 15 of TSA’s 165 functional requirements. In this case, TSA identified 6 of the 15 failures as issues that could adversely affect an operational or mission essential capability and for which there is no known work-around solution. Deficiencies for this system included issues with safety, information security, maintenance, and system dimensions, among others.

Seven systems either did not pass operational testing or had operational testing halted by TSA during this 5-year period. Operational testing is the first time a system is subject to realistic operational demands and used by a variety of transportation security officers; thus, problems with some systems are generally not detected until this phase of testing.

Eleven systems successfully passed qualification and operational testing; TSA procured 10 of them. Of these, TSA placed 9 on its qualified products lists, meaning that they were eligible for procurement. TSA procured the 10th system, second generation advanced imaging technology, independent of a qualified products list.

TSA officials stated that test and evaluation results are a primary input into their acquisition decisions, but that they also consider factors such as mission, threats, and cost. A senior TSA acquisition official explained that TSA could decide to procure systems that have minor issues during testing and perform additional testing after the system is in airports to ensure that the problems are resolved. Similarly, TSA may choose not to procure systems that successfully complete TSA’s test and evaluation process. Placement on TSA’s qualified products list allows a vendor to participate in TSA’s procurement process, but it does not guarantee a procurement contract. However, TSA has awarded a contract to buy all 9 systems it tested and placed on its qualified products list since June 2010.

Industry officials noted that there is an expectation among vendors that if they meet the requirements as outlined by TSA they will at least be able to compete for procurement. We found that, from June 2010 to July 2015, one system—a portable explosives trace detector system—passed operational testing but was not procured. This system was intended to enhance the ability of transportation security officers to randomly screen passengers’ hands and their accessible property for traces of explosives residue. In July 2012, following approximately three and a half months of testing, TSA found that this vendor’s system met requirements, including probability of detection of required detection threats, and it was found to be operationally effective and operationally suitable with limitations. However, according to senior TSA officials, a new threat subsequently emerged. The TSA Acquisition Review Board, which reviewed the potential acquisition in advance of the DHS acquisition decision authority, decided to wait for a system that met TSA’s new detection requirements; thus, TSA issued an acquisition decision memorandum in November 2012 deferring the procurement. A senior TSA acquisition official stated that this decision was a positive action because the system did not meet TSA’s mission needs and therefore should not have been procured at that time.

Immature Technologies Have Contributed to Acquisition Inefficiencies

TSA officials emphasized that immature technologies submitted by vendors are a key driver of testing failures and therefore delays in TSA’s ability to buy screening systems for use in airports. Because immature technologies often experience multiple failures during testing and require multiple retests, testing costs more and takes longer than originally anticipated. After a testing related failure, vendors will usually modify their systems to address deficiencies and then re-submit their systems for additional TSA testing. According to TSA leadership, the security-related technologies industry is still maturing, since it primarily developed after the terrorist attacks of September 11, 2001. As a result, TSA has worked extensively with industry to help develop systems that will meet TSA’s mission needs. However, TSA’s extensive work with vendors has contributed to inefficiencies in purchasing screening systems. TSA’s Assistant Administrator for OSC, who serves as the TSA Chief Technology Officer, stated that TSA wants to reduce its role in system development in order to increase efficiency and certainty in the test and evaluation process so that it can be a true purchaser of these systems.

We found that 4 of the 11 systems that successfully passed TSA’s testing process since June 2010 required at least two formal rounds of qualification or operational testing. Testing officials noted that most of the systems tested during this period required even further testing in addition to these formal rounds. For example, TSA or DHS also conducted regression testing to verify that corrections made to the previously failed systems did not adversely affect system performance, according to these officials. TSA testing officials stated that nearly all systems need some modification by the vendor during testing. Industry representatives involved in testing these systems also told us that systems are not always mature when they enter TSA’s test and evaluation process, and that they can require significant modifications and retesting before they are ready to be bought and deployed to airports.

TSA officials provided us with examples of three explosives detection systems that required multiple retests, which resulted in acquisition delays of several years. TSA also ended up spending over $3 million in additional costs incurred in retesting to ensure the systems were effective and suitable. All three systems had met explosives detection certification and false alarm rate criteria during qualification testing at the Transportation Security Laboratory, but failed to meet additional requirements during qualification testing at the TSA Systems Integration Facility and operational testing at selected airports. Some details of these delays and additional costs are below.

One explosives detection system experienced failures during testing that led to a 19 month delay to the scheduled full rate production decision and an additional $1.2 million in testing costs. During operational testing in summer 2011, TSA found that the system was effective with minor limitations, but not suitable because it failed to meet TSA’s reliability requirements and negatively impacted the ability of transportation security officers to detect threats at a certain level. The vendor made significant modifications to the system in fall 2012, and following additional testing DHS approved full rate production of the system in August 2014.

The second system experienced a 39 month delay to its scheduled full rate production decision and incurred an additional $1.1 million in testing costs as a result of multiple rounds of retests. The system began testing in fall 2010 and after follow-on operational testing in fall of 2012, TSA found the system to be effective with limitations and not suitable due to, among other things, issues with resetting or restarting the system after bag jam failures and reliability. The system underwent testing by a third party entity in late 2014 to verify that modifications made to the system improved its performance, and DHS approved full rate production in May 2015.

At the time of our engagement, the third explosives detection system’s scheduled full rate production decision had not yet occurred, but it had already been delayed by more than 60 months, resulting in an additional $1.2 million in testing costs. TSA began qualification testing of the system in fall 2010, and, after operational testing in fall 2011, TSA found the system to be effective with major limitations and suitable with minor limitations. TSA encountered issues with bag tracking and reliability, among other things. As of summer 2015, after five rounds of retests, the vendor was pursuing a third party test of the system.

According to TSA and DHS testing officials, the actual single rounds of qualification and operational testing timeframes are not long, but it takes vendors a significant amount of time to correct issues identified during testing, which results in a delay to the start of TSA’s next round of testing. We found in June 2014 that the difficulties in getting vendors systems to meet the higher level detection for advanced imaging technology required the vendors to go back and conduct remedial developmental testing activities. We also found that the qualification testing environment is not conducive to remediating deficiencies that require extensive additional development work because feedback to vendors during this phase is limited. TSA testing officials stated they believe that vendors submit immature technologies and use the testing process to inform further system development, but TSA expects systems to be fully mature at the start of testing.

Too Soon to Tell Whether TSA’s Actions to Ensure Technology Maturity Will Address All Factors Contributing to Acquisition Inefficiencies

TSA has acknowledged the need to better ensure technology maturity at the start of testing to improve the efficiency of its test and evaluation process. In addition to other efforts, a key action TSA is taking to achieve this goal is developing a third party testing strategy—through which a third party tester will help ensure systems are mature prior to entering TSA’s test and evaluation process. TSA plans to implement this strategy in 2016, but it is too soon to tell whether the strategy will address all of the factors that contribute to acquisition inefficiencies because TSA has yet to finalize its key aspects. For example, TSA has not identified whether there are a sufficient number of eligible third party testers or established a mechanism to oversee the testing they will perform. Further, TSA did not conduct a comprehensive assessment of testing data prior to developing its third party testing strategy and may be missing opportunities to identify additional areas for improvement in its acquisition process.

TSA Is Taking Actions to Better Ensure Technology Maturity at the Start of Testing

As discussed earlier in the report, TSA’s test and evaluation process is consistent with policies and guidance outlined by DHS and TSA as well as acquisition best practices; however, TSA has acknowledged the need to improve the efficiency of the process by better ensuring technology maturity at the start of testing. For example, the Office of Security Capabilities’ most recent strategic plan as well as TSA’s overall 2015 Strategic Technology Investment Plan both note that the test and evaluation process needs to be more efficient and agile in responding to emerging threats.

TSA has recently initiated two specific reforms to improve the efficiency of the test and evaluation process: increased transparency with vendors and a third party testing strategy. First, to increase transparency, TSA officials told us that they are sharing test plans with vendors to better prepare them for testing. While industry officials agreed that TSA has become more transparent, they said that the number of test plans that TSA has shared thus far have been limited. TSA also created a Test and Evaluation Process Guide to inform stakeholders—particularly industry— of the test and evaluation process, including the test and evaluation phases and roles/responsibilities of TSA, DHS, and industry. TSA also holds periodic industry days to update vendors on planned procurements and changes to acquisition processes, among other things. However, TSA test and evaluation guidance necessarily limits the extent to which testing officials can share information with industry throughout the testing process. TSA’s Director of Test and Evaluation stressed that, to maintain the integrity of the test process, they do not intend to provide vendors with detailed information that could be used to “game” the tests. For example, according to officials from the Transportation Security Laboratory’s Independent Test and Evaluation division, the certification and qualification testing it conducts—unlike developmental testing assistance made available by the laboratory to vendors—provides, by design, limited feedback between testers and the vendors, to ensure the integrity of the test and evaluation process.

Second, TSA is developing a third party testing strategy, which it has partially implemented for technologies that have already entered the test and evaluation process. According to interim guidance effective in July 2014, a vendor experiencing a significant failure during testing is required to fund and undergo third party testing and provide results to TSA, demonstrating that the system has met the previously failed requirements before the system is allowed to resume TSA’s testing process. Officials told us that the guidance exempts detection-related testing. At this time, according to TSA and DHS officials, third party testing cannot replace the explosives detection testing conducted by the Transportation Security Laboratory because no other testing facility has the same capabilities. Since TSA issued the guidance, one vendor has utilized a third party tester to verify that an operational availability problem discovered during follow-on operational testing was resolved. TSA is in the process of implementing the primary piece of the strategy, which establishes additional third party testing requirements that vendors must meet before entering the test and evaluation process. Figure 2 provides an overview of TSA’s use of third party testing following a technology failure during testing.

TSA Is Implementing a Third Party Testing Strategy but Has Not Finalized Its Approach or Conducted a Comprehensive Assessment of Testing Data

TSA Has Yet to Finalize Key Aspects of the Third Party Testing Strategy

TSA is developing a third party testing strategy and plans to implement it for select technologies in 2016. The strategy is a key TSA effort to improve technological maturity and ensure readiness for testing in order to reduce the number of technology failures during testing. A key aspect of this strategy is to strengthen requirements for vendors’ qualification data packages, which are required to enter the test and evaluation process. Currently, TSA requires some third party testing to confirm certain requirements in these data packages, such as compliance with emissions and radiation standards. But for other requirements, at present vendors simply can attest that they have been met rather than provide independent verification. The strategy could increase the number of requirements that must be independently verified in qualification data packages for some technologies, though TSA has not yet worked out the details.

TSA has taken some steps to develop the strategy. For example, TSA is working in coordination with the National Institute of Standards and Technology to develop requirements for third party testers in accordance with international standards for test laboratories. TSA has communicated the developing strategy to key stakeholders, including DHS and industry, through meetings with DHS officials, industry days, and a request for information for potential third party testers. Also, TSA is taking steps to prioritize the requirements that should be independently verified in vendor qualification data packages.

However, TSA has yet to finalize key aspects of the strategy to ensure that it will be successful before it is formally required next year.

TSA officials are unsure of how many third party testers may be able to provide the necessary services to vendors. Without knowledge of the number of third party testers that can provide the necessary services, it is unclear whether there will be enough testers to satisfy the demand for this service. Further, if the number of testers is limited, it may increase the cost of testing services.

TSA has not yet established a process for approving or monitoring third party testers, or established how often they will need to be recertified. TSA officials stated they may choose to have an independent party approve third party testers after initial implementation of the strategy. At the time of our review, TSA was finalizing the third party test application process for potential third party testers to apply for TSA approval. TSA officials estimated that this process would be completed by the end of March 2016.

TSA has not established a mechanism to oversee third party testing.

DHS and TSA officials stated that the extent of DHS’s role is dependent on how the third party testing strategy is implemented.

TSA officials believe that the strategy will reduce the number of rounds of retests and decrease the time it takes for TSA to test systems. However, officials stated that there is no guarantee that third party testing will shorten acquisition timelines.

TSA officials are unsure whether the third party testing strategy will save overall acquisition costs, which they have highlighted as a potential benefit. Specifically, while third party testing will shift the costs of retesting from TSA to the vendors, industry officials told us it is probable that vendors will reflect these additional costs in their pricing. TSA officials responsible for planning and implementing the third party testing strategy told us they had not assessed the potential impacts of vendors paying for testing on a system’s costs, or the possibility that third party testing costs could be a barrier to entering the market for new vendors.

As we established in prior work, components of sound planning include, among other items, identifying: problems and causes; resources, investments, and risks; roles, responsibilities, and coordination; and integration among and with other entities. If TSA does not finalize key aspects of its strategy prior to implementing further third party testing requirements for vendors to enter testing, it may cause unintended consequences, such as increasing acquisition costs or creating a barrier to market entry for new vendors, such as small businesses.

TSA Has Not Conducted a Comprehensive Assessment of Testing Data

At the time of our review, TSA had not conducted a comprehensive assessment of testing data, such as timeframes for completing testing and costs incurred, because it lacked a mechanism to track and consolidate testing data across all technologies, such as testing delays, costs, timeframes, and results. Thus, TSA does not have any documented assessment supporting the decision to implement the third party testing strategy and was unable to provide us with testing timeframes for each of the 22 systems tested in the past five years. TSA tracks testing delays and costs in separate systems and provides actual testing timeframes in the test reports for each system. However, after we raised this point during the course of our review, TSA officials developed a master testing tracker to more comprehensively track testing data. The master tracker consolidates testing data to generate a holistic view of specific technologies by summarizing testing timeframes, delays, costs, and progression through testing, among other things. TSA testing officials stated that the master tracker will help them to support TSA leadership by consolidating data for each system in one place, thereby eliminating the need to go through individual test reports to gather specific testing information, such as testing timeframes.

As noted above, in OSC’s Strategic Plan for 2013 to 2016, it expressed a commitment to risk-informed, data driven analysis and decision-making. While the master testing tracker TSA plans to develop is a positive first step towards more informed decision-making, officials have not established a plan for assessing the information collected from the tracker. Conducting a comprehensive assessment of these testing data is the critical next step. TSA has previously failed to assess information it collected with respect to third party testing. Specifically, while developing its third party testing strategy TSA issued a request for information in December 2012 about potential third party testers and their capabilities. However, senior TSA testing officials stated that TSA did not evaluate any of the responses that it received from potential third party testers because TSA intended the request to be a mechanism to provide vendors with information. According to Standards for Internal Control in the Federal Government, ongoing monitoring should occur in the course of normal operations. In addition, we previously found that agencies can use performance information to identify problems in existing programs, to try to identify the causes of problems, and/or to develop corrective actions. The benefit of collecting performance information is only fully realized when this information is actually used by agencies to make decisions oriented toward improving results.

TSA’s actions to address the repeated technology failures during testing which have led to acquisition inefficiencies—in large part through its third party testing strategy—focus on improving technological maturity. However, TSA and industry officials we spoke with identified additional issues that may be contributing to the inefficiencies, which third party testing may not address. Specifically, TSA and industry officials highlighted issues pertaining to the program management offices’ development of acquisition schedules. Some industry officials we met with stated that the testing timeframes in acquisition schedules, specifically for explosives detection systems, are unrealistic. For example, TSA has communicated to vendors that testing for explosives detection systems will take around three years; however, according to an industry official this testing typically takes four or more years. Testing officials stated that they provide the program offices with time estimates for how long a single round of testing should take, but that it is ultimately the program office’s decision regarding how many rounds of testing to include in the schedule. TSA program officials acknowledged that they typically set optimistic schedules, assuming that technologies will not require multiple rounds of retesting. However, testing officials have stated it is rare for technologies to pass testing the first time.

In addition, industry officials identified issues with how requirements have been defined and interpreted in the past. For example, one industry official stated that there is a disconnect among TSA, DHS, and vendors about how operational and functional requirements will be interpreted and evaluated during the testing process. This official stated that when TSA tested a system submitted by the company that the official represents, the vendor and TSA interpreted a requirement about operational availability differently, specifically with regards to whether the requirement applied to the system on its own or to the system as integrated into the airport infrastructure. The vendor failed testing and was required to make system modifications prior to TSA’s qualification of the system.

TSA testing officials also noted problems with how TSA has defined, interpreted, and shared requirements. For example, TSA developed two sets of requirements for its 2010 competitive procurement for explosives detection systems—requirements that systems must meet and requirements that TSA would like the systems to meet, which are referred to as minimum “shall” and non-minimum “shall” requirements, respectively. However, key stakeholders have disagreed on how these requirements should be interpreted as systems go through the testing process. In the case of one explosives detection system, the program office disagreed with how testing officials evaluated the test results, noting that the non-minimum “shall” requirements should not have been considered in the evaluation of effectiveness or suitability. Testing officials stated that systems needed to meet some of the non-minimum “shall” requirements in order to fulfill minimum “shall” requirements and therefore those should have been identified as minimum “shalls.” In addition, TSA did not release the operational requirements document for explosives detection systems to vendors prior to testing, and, as a result, vendors did not know what to expect during operational testing, according to testing officials.

TSA is taking steps to address some of these issues. For example, testing officials stated that TSA plans to have one set of requirements for the next competitive procurement of explosives detection systems. In addition, TSA’s Mission Analysis Division is focused on improving the requirements development process through a more rigorous and structured method that involves the participation of relevant stakeholders, such as industry, DHS, and end-users, who TSA officials stated have not historically been involved in the process. In March 2015, industry officials identified additional actions that TSA could take to improve the transparency of requirements. Specifically, industry officials provided formal recommendations to TSA regarding how to further improve the transparency of requirements, such as developing a limited set of key requirements, which are critical to executing TSA’s mission, sharing requirements documents in draft form prior to final release, and defining and prioritizing requirements for each phase of the testing process, among other things.

Without conducting and documenting an assessment of testing data available to date, such as testing timeframes, costs incurred, and testing delays across all technologies and sharing it with key stakeholders, it is too soon to tell to what extent TSA’s reforms will address factors contributing to any acquisition inefficiencies. Specifically, TSA may be missing opportunities to identify other factors, in addition to technology immaturity, that are outside the purview of testing officials. An overall assessment of testing data could inform potential areas for improvement in the acquisition process, such as incomplete or unclear requirements and unrealistic acquisition schedules.

Conclusions

For over a decade, TSA has secured the nation’s civil aviation system amid continued threats. Due to the significant challenge the agency faces in balancing security concerns with efficient passenger movement, it is important that TSA procure and deploy effective passenger and baggage screening technologies. As the security-related technology industry evolves to meet new and changing threats, TSA’s test and evaluation processes are also evolving. Since 2010, though, only half the systems tested have successfully passed testing and been qualified by TSA for procurement. Technology failures during testing have resulted in multiple retests and inefficiencies in acquiring systems that could potentially help TSA execute its mission.

TSA has taken steps that could improve the maturity of technologies put forth by industry and reduce the burden on TSA’s own testing resources. TSA plans to implement its third party testing strategy for selected technologies in 2016. However, because TSA has not finalized certain aspects of this strategy—for example, determining how many potential third party testers are qualified to assist TSA, and how TSA will oversee this component of testing—there is a risk that the strategy will not be as effective as envisioned. Unintended consequences, such as increased acquisition costs, could result. While the third party testing strategy is likely to help improve system readiness for entering the test process, TSA has not conducted a comprehensive assessment of testing data for baggage and passenger screening systems, and therefore cannot be certain whether the steps currently planned will address other factors contributing to acquisition inefficiencies. Given TSA’s emphasis on improving its acquisition and test and evaluation processes, an assessment of TSA’s testing data would help guide future TSA reforms.

Recommendations for Executive Action

We recommend that the Administrator of TSA take the following two actions: 1. To help ensure that actions taken to improve the test and evaluation process address identified challenges, finalize all aspects of the third party testing strategy before implementing further third party testing requirements for vendors to enter testing. 2. To help ensure that the reforms TSA has underway are informed by existing information, conduct and document a comprehensive assessment of testing data available to date, such as timeframes for completing testing, costs incurred, and testing delays across all technology areas to identify key factors contributing to any acquisition inefficiencies and potential areas for reform.

Agency Comments and Our Evaluation

We provided a draft of this product to DHS for comment. In its written comments, reproduced in appendix II, DHS concurred with both of our recommendations and provided plans of action and estimated completion dates for them. In response to our first recommendation—that TSA finalize all aspects of the third party testing strategy before implementing further third party testing requirements for vendors to enter testing—DHS stated that TSA plans to begin implementing the third party test program in a phased approach by December 31, 2016. TSA’s planned actions include, among other things, finalizing the third party tester application process, communicating the new approach to industry, and developing third party testing requirements. We believe that these are positive steps to finalize the strategy, but that TSA should also consider the potential consequences of implementing the strategy, such as the cost impacts to TSA and vendors, which will help to ensure the effectiveness of the strategy.

DHS also provided technical comments that we incorporated into the report as appropriate.

We are sending copies of this report to the Secretary of Homeland Security, the TSA Administrator, and the appropriate congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4841 or mackinm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to this report are listed in appendix III.

Appendix I: Scope and Methodology

The Transportation Security Acquisition Reform Act included a provision for GAO to examine the Transportation Security Administration’s (TSA) test and evaluation process for security-related technologies. Specifically, this report examines the extent to which (1) TSA’s test and evaluation process helps TSA meet mission needs through the acquisition of passenger and baggage screening technologies, and (2) TSA’s planned actions to improve the test and evaluation process address factors contributing to inefficiencies, if any, in acquiring those technologies.

For our first objective, we reviewed Department of Homeland Security (DHS) and TSA acquisition and testing policies and guidance to determine the role of test and evaluation in TSA’s acquisition process and compared them to acquisition best practices. We also analyzed testing documentation provided to us by TSA for the 22 Passenger Screening Program (PSP) and Electronic Baggage Screening Program (EBSP) systems TSA has tested since June 2010, such as test and evaluation plans, test reports, system evaluation reports, data packages for entry into testing, and information regarding the number of retests during testing to determine the number of systems that passed each stage of TSA’s testing process. In addition, we reviewed TSA and DHS acquisition documentation for passenger and baggage screening technologies, including requirements documents, schedules, letters of assessment, acquisition decision memorandums, contract award notices, and information regarding the number of each technology deployed in airports nationwide to determine the number of systems that TSA has tested, qualified for use, and procured since June 2010. Additionally, we conducted interviews with TSA officials from the Office of Security Capabilities (OSC), the Office of Acquisitions, PSP, EBSP, and the Mission Analysis Division, regarding the acquisition and test and evaluation processes, the influence of test and evaluation results on TSA’s decisions about which technologies to procure, and reasons for any acquisition inefficiencies. We also met with DHS officials from the Science and Technology Directorate and the Office of Program Accountability and Risk Management regarding their roles and responsibilities related to TSA’s test and evaluation process. Further, we attended a joint industry and TSA workshop hosted by the Airport Consultants Council focused on TSA’s acquisition process for security- related technologies, including its test and evaluation process. Finally, we visited the two primary testing facilities for TSA’s security-related technologies—the TSA Systems Integration Facility, in Arlington, Virginia and DHS’s testing facility, the Transportation Security Laboratory, in Atlantic City, New Jersey to interview testing officials and observe the testing process. We chose these locations because they are TSA’s two primary testing facilities for security-related technologies.

For our second objective, we reviewed TSA strategic planning documents, including TSA’s August 2015 Strategic Five-Year Technology Investment Plan for Aviation Security, OSC’s Strategic Plan for 2013 to 2016, and OSC’s May 2014 Transportation Security Strategic Capability Investment Plan, to identify TSA’s stated challenges with the test and evaluation process and its plans for improvement. We also reviewed TSA’s interim third party testing guidance, implementation timeline, and briefings to industry as well as testing examples, which served as TSA’s rationale for implementing the third party testing strategy. In addition, we conducted interviews with TSA and DHS testing officials, including testers and evaluators, regarding challenges of TSA’s acquisition and test and evaluation processes as well as ongoing and planned efforts to improve testing efficiency and transparency. We also met with representatives from two industry groups with experience related to TSA’s test and evaluation process to obtain industry views on any challenges in the test and evaluation process for screening technologies, areas for potential improvement, and their perspectives on TSA’s third party testing strategy. We met with representatives from the Airport Consultants Council’s Security Manufacturers Coalition, who represented three of TSA’s largest vendors receiving 8 of PSP and EBSP’s 10 largest contracts to date. We also met with officials from the Homeland Security and Defense Business Council, who represented companies providing test and evaluation support services to DHS and TSA. Additionally, we conducted an interview with the DHS Director of Operational Test and Evaluation regarding planned efforts to expand DHS oversight of component-level testing, including TSA, from operational to developmental testing and its involvement in the development of TSA’s third party testing strategy. Further, we met with officials from the National Institute of Standards and Technology involved in the development of TSA’s third party testing strategy to determine TSA’s coordination with stakeholders in the development and implementation of the strategy. Finally, we conducted interviews with TSA testing officials regarding TSA’s existing and planned mechanisms for tracking systems throughout the test and evaluation process and any efforts to assess testing data, such as timeframes for completing testing and costs incurred and compared them to Standards for Internal Control in the Federal Government.

We conducted this performance audit from April 2015 to December 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Homeland Security

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Katherine Trimble (Assistant Director), Charlie Shivers III (Analyst-in-Charge), Peter Anderson, Molly Callaghan, William Carrigg, Kristine Hassinger, Mark Hoover, Michael Kaeser, Jean McSween, Lindsay Taylor, and Ozzy Trevino made significant contributions to this report.