TSA Acquisitions: Further Actions Needed to Improve Efficiency of Screening Technology Test and Evaluation
Published: 2015-12-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: TSA, within the Department of Homeland Security, is responsible for securing the nation's civil aviation system while facilitating the movement of passengers and commerce at approximately 440 airports nationwide. TSA tests passenger and baggage screening technologies developed by industry to ensure they support TSA missions. In reviews from 2010 to 2014, GAO found that TSA encountered challenges in acquiring and deploying technologies. The Transportation Security Acquisition Reform Act contained a provision for GAO to assess TSA's test and evaluation activities for security-related technologies. This report assesses the extent to which (1) TSA's test and evaluation process helps it meet mission needs through the acquisition of passenger and baggage screening technologies, and (2) TSA has taken steps to improve the test and evaluation process.
GAO reviewed DHS and TSA acquisition and test policies, analyzed testing and acquisition documentation for technologies tested in the past five years, observed the testing process at DHS and TSA facilities, and spoke with DHS, TSA, and industry officials.
section_title: What GAO Found
paragraphs: The Transportation Security Administration's (TSA) test and evaluation process has enabled TSA and Department of Homeland Security (DHS) officials to identify passenger and baggage screening technologies that will meet mission needs, but technology failures during testing have contributed to inefficiencies in the acquisition process. Consistent with departmental guidance and acquisition best practices, TSA's test and evaluation process provides information regarding the ability of technologies to meet mission needs before agency officials decide whether to begin full production, saving the agency from investing in potentially expensive yet ineffective equipment. From June 2010 to July 2015, half of the 22 systems that TSA tested successfully completed qualification and operational testing. TSA procured all but 1 of the 11 successful systems. Technologies that entered the test and evaluation process and were immature required significant modifications and retesting.
TSA has taken steps to improve its test and evaluation process by helping ensure technologies are mature before entering testing, but it is too soon to tell whether these actions will address all of the factors that contribute to acquisition inefficiencies. A key action TSA is taking involves developing a third party testing strategy, through which a third party will help ensure systems are mature prior to entering TSA's test and evaluation process. TSA plans to implement its approach in 2016, but it has yet to finalize key aspects of the strategy. For example, TSA has not identified whether there are a sufficient number of eligible third party testers or established a mechanism to oversee that testing. Without a finalized strategy, TSA risks unintended consequences, such as increasing acquisition costs. Further, TSA has not conducted or documented a comprehensive assessment of testing data and thus may be missing opportunities to identify additional areas for improvements to its acquisition process. An assessment of this data, such as costs incurred, could help TSA guide future reforms to the test and evaluation process to help ensure they address factors contributing to any acquisition inefficiencies.
section_title: What GAO Recommends
paragraphs: TSA should finalize its third party testing strategy before implementation and conduct and document a comprehensive assessment of testing data to identify key factors contributing to any acquisition inefficiencies. DHS concurred with GAO's recommendations.

section_title: Letter
paragraphs: Commercial aircraft have long been a target of terrorists, who have  carried out attacks with explosives smuggled onboard in checked  baggage as well as with passenger-carried contraband and explosives.  The 1988 bombing of a U.S. aircraft over Lockerbie, Scotland as well as  the terrorist attacks of September 11, 2001, illustrate the real threat posed  by terrorists against aircraft. The threat continues today as new and  innovative methods—such as the use of artfully concealed homemade  explosives—proliferate. The Transportation Security Administration  (TSA), a component of the Department of Homeland Security (DHS), is  responsible for overseeing security operations at the nation’s roughly 440  commercial airports as part of its mission to protect the nation’s civil  aviation system. TSA screens individuals, their carry-on luggage, and  their checked baggage to deter, detect, and prevent carriage of any  prohibited items, such as explosives and contraband, on board  commercial aircraft. To carry out these activities, the agency relies to a  large extent on security-related screening technologies, such as  explosives detection systems and advanced imaging technology devices.
In our past work, we have found that TSA encountered challenges in  effectively acquiring and deploying passenger and baggage screening  technologies and had not consistently implemented DHS policy and best  practices for procurement. In light of these concerns, among other  issues, Congress enacted the Transportation Security Acquisition Reform  Act in December 2014, which contained a provision for GAO to assess  TSA’s test and evaluation activities for security-related technologies. This  report assesses the extent to which (1) TSA’s test and evaluation process  helps TSA meet mission needs through the acquisition of passenger and  baggage screening technologies, and (2) TSA’s planned actions to  improve the test and evaluation process address factors contributing to  inefficiencies, if any, in acquiring those technologies.
To assess the extent to which TSA’s test and evaluation process helps  the agency meet mission needs, we reviewed DHS and TSA acquisition  documentation for passenger and baggage screening technologies tested  since June 2010. This documentation included, for example, letters of  assessment and acquisition decision memorandums. We conducted  interviews with TSA officials to determine the influence of test and  evaluation results on TSA’s decisions about which technologies to  procure and any reasons for repeated testing, which affected TSA’s ability  to procure the technologies. We also met with relevant DHS officials  regarding their roles and responsibilities related to TSA’s test and  evaluation process, which included site visits to the two primary testing  facilities for TSA’s security-related technologies—the TSA Systems  Integration Facility in Arlington, Virginia and the DHS Transportation  Security Laboratory in Atlantic City, New Jersey. To identify the extent to  which TSA’s planned actions to improve the test and evaluation process  address factors contributing to acquisition inefficiencies, if any, we  reviewed TSA’s documentation related to the planned actions. We also  met with representatives from two industry groups with experience related  to TSA’s test and evaluation process to obtain industry views on any  challenges in the test and evaluation process for screening technologies,  areas for potential improvement, and TSA’s planned actions to improve  the process. Finally, we conducted interviews with TSA testing officials  regarding TSA’s existing and planned mechanisms for tracking  technologies throughout the test and evaluation process, and any efforts  to assess testing data by looking at, for example, timeframes for  completing testing and costs incurred. This report focuses on the test and  evaluation process TSA uses as it acquires screening systems and the  extent to which it procures the systems that it tests, and does not discuss  the ability of systems to meet mission needs once deployed in the field.  Additional details about our scope and methodology are discussed in  appendix I.
We conducted this performance audit from April 2015 to December 2015  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: TSA Acquires Security- Related Technologies to Screen Passengers and Baggage for Threats
paragraphs: TSA uses security-related technologies to help secure approximately 1.8  million passengers, 1.2 million checked bags, and 3 million carry-on bags  on 25,000 flights at roughly 440 federally regulated airports every day. As  of August 2015, TSA had deployed about 15,000 units of security-related  technology to airports nationwide and anticipates spending a significant  portion of its $3.6 billion security capability budget on technologies such  as these over the next 5 years. Within TSA, the Office of Security  Capabilities (OSC) provides security-related technology solutions through  two major DHS acquisition programs, the Passenger Screening Program  (PSP) and the Electronic Baggage Screening Program (EBSP):
PSP technologies, such as advanced imaging technology and bottled  liquid scanners, work in combination at airport checkpoints to screen  passengers and their carry-on baggage for threats.
EBSP technologies, such as explosives trace detectors and  explosives detection systems, help ensure that TSA screens 100  percent of checked baggage for explosives, as mandated by law.
For each technology type, vendors develop their own specific version or  system. See table 1 for an overview of key PSP and EBSP technology  types.
According to TSA, EBSP and PSP are transitioning into a sustainment  mode, meaning they are largely not procuring new types of screening  technologies and are instead focused on the recapitalization of over 2,400  systems that are reaching their end-of-life over the next 5 years.
In our past work, we found key challenges related to TSA’s efforts to  acquire technologies. For example, in March 2014, we found that TSA’s  performance assessments of advanced imaging technology systems with  automatic target recognition capability did not account for all factors  affecting the effectiveness of the system, such as how well the system  would perform with human operators, because the assessments were  conducted in a laboratory. We recommended that TSA establish  protocols to facilitate capturing operational data on secondary passenger  screening at the checkpoint to determine the extent to which rates of false  alarms for various advanced imaging technology systems affect  operational costs once the systems with automatic target recognition  capability are networked. TSA concurred with this recommendation. In its  comments to our report, TSA stated that it will monitor, update, and report  the results of its efforts to capture operational data and evaluate its  associated impacts to operational costs. In January 2012, we found that  TSA did not fully follow DHS acquisition policies when acquiring a  different version of advanced imaging technology, which resulted in DHS  approving nationwide deployment without full knowledge of TSA’s revised  specifications. Among other things, we found that TSA had changed key  performance parameters during the acquisition process without formally  informing DHS acquisition officials, in violation of DHS policy. We  recommended that TSA develop a road map describing when vendors will  meet milestones for further developing advanced imaging technology and  that TSA brief Congress as part of the budget process. Findings from this  report resulted in a reduction in planned advanced imaging technology  purchases amounting to approximately $1.4 billion.
section_title: DHS Acquisition Process
paragraphs: DHS policies and processes for managing its major acquisition programs,  such as TSA’s PSP and EBSP, are primarily set forth in DHS Acquisition  Management Directive 102-01 and DHS Instruction Manual 102-01-001,  Acquisition Management Instruction/Guidebook. Within TSA, the PSP and  EBSP program management offices are responsible for planning and  executing the acquisition programs within the cost, schedule, and  performance parameters in their program baselines, which are approved  by DHS. DHS’s Deputy Secretary or Under Secretary for Management  serve as the decision authority for PSP and EBSP acquisitions since both  are large acquisition programs with life cycle cost estimates of over $1  billion. The decision authority reviews programs at five predetermined  acquisition decision events to assess whether the programs are ready to  proceed through each of the four acquisition life cycle phases.
During the first phase of the acquisition life cycle—the “need” phase— TSA develops a mission need statement for a potential solution to an  identified problem, such as threats concealed on passengers and in their  carry-on items. If the acquisition decision authority concludes that the  need is of sufficiently high priority to DHS, the program moves to the  second phase known as the “analyze/select” phase. The TSA program  management offices—such as PSP and EBSP—communicate this need  to potential vendors through requests for information, requests for  proposals, and broad agency announcements in search of potential  solutions, such as innovative screening technologies. Program managers  select the best technology option and, if the acquisition decision authority  approves the proposed technology, the program moves to phase three.  The third phase is the “obtain” phase in which TSA develops, tests, and  evaluates systems—vendors’ versions of the selected technology. With  acquisition decision authority approval, programs enter the final phase,  the “produce/deploy/support” phase, and TSA can proceed with  procurement and full deployment of tested and qualified systems.
section_title: TSA Test and Evaluation Process
paragraphs: The goal of test and evaluation is to make sure that a product works as  intended before it is provided to end-users, such as TSA’s transportation  security officers. According to DHS policy, the primary purpose of test and  evaluation is to provide timely, accurate information to managers,  decision makers, and other stakeholders to reduce programmatic,  financial, schedule, and performance risk. TSA’s test and evaluation  process is guided by DHS and TSA policies. DHS issued its test and  evaluation policy in May 2009 and TSA issued its policy in July 2010.
Within OSC, the Operations Support Division is responsible for testing  and evaluating screening technologies in coordination with DHS’s  Transportation Security Laboratory. Table 2 provides an overview of DHS  and TSA test and evaluation roles and responsibilities.
As noted in the table, DHS and TSA both have responsibilities for  evaluating vendors’ qualification data packages and determining whether  to accept systems for testing. Once accepted, the systems undergo  TSA’s three phase pre-acquisition test and evaluation process to verify  requirements. Responsibility for verifying requirements is assigned to  testers for each of the three phases: (1) qualification and certification  testing at the Transportation Security Laboratory, (2) qualification testing  at the TSA Systems Integration Facility, and (3) operational testing at  selected airports.  1.  Qualification and certification testing at the Transportation  Security Laboratory. Qualification testing is the formal verification  and validation of a system’s performance and must provide  confidence that the system will satisfy desired capabilities in an  operational environment. Qualification testing is typically conducted at  two facilities, each serving distinct functions. Testing conducted at the  DHS Science and Technology Directorate’s Transportation Security  Laboratory in Atlantic City, New Jersey is focused on qualifying or  certifying that a system meets the probability of detection of all  categories of explosives while meeting TSA’s designated false alarm  requirement. Typically, systems complete detection testing at the  Transportation Security Laboratory prior to undergoing the balance of  qualification testing at the TSA Systems Integration Facility.  2.  Qualification testing at the TSA Systems Integration Facility. TSA  conducts additional qualification testing at the TSA Systems  Integration Facility in Arlington, Virginia to verify system performance  against defined functional requirements, such as system capacities,  human factors, physical characteristics, user safety, and system  reliability, availability, and maintainability. This testing may include  transportation security officers who exercise standard operating  procedures to provide feedback regarding the system’s functions in a  controlled environment. Systems must pass qualification testing and  undergo an operational test readiness review before they can proceed  to operational testing.  3.  Operational testing at select airports. TSA conducts operational  testing at select airports to evaluate a system’s operational  effectiveness and suitability in a realistic environment and to ensure  that the airport infrastructure is ready to accept the system. According  to TSA, it selects test sites, in coordination with various stakeholders,  using criteria intended to facilitate an unbiased evaluation of the  systems. TSA injects mock threats during operational testing to  assess performance of the entire system, including users and  standard operating procedures. Operational tests focus on  demonstrating that operational requirements have been met and that  all critical operational issues have been resolved.
Following operational testing, TSA prepares a system evaluation report,  which provides effectiveness and suitability determinations along with  potential system improvement recommendations. The DHS Director of  Operational Test and Evaluation reviews the report and prepares a letter  of assessment. The letter assesses the system evaluation report and the  adequacy of TSA’s operational test and concludes whether the system is  operationally suitable and effective for procurement. TSA’s program  management offices place approved systems on qualified products lists,  which contain systems that have successfully completed the test and  evaluation process and have been approved by DHS. Once a system is  on a qualified products list, which TSA maintains for most technology  types, such as explosives detection systems, the vendor can participate  in TSA’s procurement process. However, placement on a qualified  products list does not guarantee that the vendor will receive a  procurement contract.
TSA conducts acceptance testing on systems at vendors’ factories  following production and again once installed at airports to ensure  consistency of the manufacturing process, system configuration, and  functionality. Post deployment, TSA’s Office of Inspection and the DHS  Office of Inspector General conduct covert tests in airports to identify  vulnerabilities in screening processes. TSA and DHS test and evaluate  systems prior to procurement, but covert testing has demonstrated that  systems, when integrated into TSA’s screening process, do not always  work as intended once deployed. TSA’s test and evaluation activities are  funded by the program offices, and TSA’s life cycle cost estimate for  EBSP reflects average annual testing costs of approximately $24 million.  PSP officials estimate that the program averages $10 million in annual  testing costs.
section_title: DHS and TSA Partner with Industry to Develop and Deploy Screening Technologies in a Dynamic Threat Environment
paragraphs: TSA’s security screening must adapt to meet evolving threats since  potential terrorists use their knowledge of aviation security measures  when planning aviation-related attacks. New explosives threats can result  in heightened detection standards, which may require software upgrades  to existing systems or the development of new technologies. DHS and  TSA partner with industry to build, test, field, and sustain security-related  technologies. For example, DHS makes available testing facilities and  expertise at the Transportation Security Laboratory that vendors can use,  by their choice, to further the development and evaluation of their  systems. TSA’s August 2015 Strategic Five-Year Technology Investment  Plan for Aviation Security noted the market for security-related  technologies is quite limited, with only a handful of vendors, which creates  a significant challenge for small businesses. According to TSA,  substantial funding and time are required by industry to develop, qualify,  and produce TSA’s screening technologies, which represents a significant  barrier to entry. For example, the procurement cost of an explosives  detection system can exceed over half a million dollars. TSA’s  approximately 15,000 systems were manufactured by 13 different  vendors, with 5 vendors accounting for approximately 81 percent of  deployed systems.
section_title: TSA Has Identified Technologies to Meet Mission Needs, but Failures during Testing Have Contributed to Acquisition Inefficiencies
paragraphs: Consistent with departmental guidance and acquisition best practices,  TSA’s test and evaluation process supports its acquisition decisions by  providing information regarding the ability of passenger and baggage  screening technologies to meet mission needs prior to full production  decisions. From June 2010 to July 2015, only half of the 22 systems that  TSA and DHS tested successfully passed qualification and operational  testing and were therefore deemed effective and suitable for deployment.  TSA procured all but one of the successful systems. Technology failures  during testing, as a result of vendors’ immature technologies entering the  test and evaluation process, often required significant fixes and have  contributed to inefficiencies in TSA acquiring the technologies for use in  airports.
subsections: section_title: Test and Evaluation Process Is Consistent with Guidance and Best Practices and Identifies Technologies That Meet Mission Needs
paragraphs: TSA’s test and evaluation process is a critical means of providing DHS  and TSA officials with accurate information about security-related  technologies to support acquisition decisions. We found that the process  is consistent with DHS and TSA policies and guidance and helps TSA  ensure that the technologies it acquires fulfill mission needs upon  deployment. For example, TSA has integrated end users, such as  transportation security officers into its test and evaluation process,  consistent with TSA test and evaluation policy. Also, TSA officials said  they have designated program office liaisons to improve communication  between testing officials and the TSA program offices responsible for  procuring the screening technologies to help ensure program offices have  the testing information they need to inform the procurement decision, in  line with DHS test and evaluation policy. In addition, TSA establishes  system evaluation teams for each system undergoing test and evaluation;  the teams consist of the TSA and DHS officials that plan for and evaluate  system effectiveness and suitability throughout the testing process.  Further, TSA’s test and evaluation process is designed to identify and  evaluate existing technologies that can be adjusted or repurposed to  meet the agency’s needs. This goal is in line with department guidance  underlining the importance of pursuing technologies, specifically  commercial off-the-shelf technologies that do not require a substantial  investment of time and money to ensure they are effective once procured.
We previously found that the validation of product knowledge early in the  acquisition process and before key investments are made is consistent  with best practices by commercial firms. Specifically, leading commercial  firms strive to detect as many problems as possible during testing, which  in turn leads to easier and less expensive improvements to products  down the road. Leading commercial firms highlighted the importance of  perceiving problems during testing, not as failures, but as knowledge  which can be used to improve a product. Consistent with this practice,  while vendors’ proposed systems may fail TSA’s test and evaluation  process, these failures can help the government identify needed fixes to  better ensure that they will be effective and suitable prior to being  deployed to airports nationwide. Further, we previously found that cost  overruns and underperformance of technologies are exacerbated when  problems discovered during testing are not resolved. For example, costly  redesigns and retrofits could be required to achieve satisfactory  performance of units already deployed in the field. By conducting  operational testing on systems prior to procurement, TSA is working to  prevent such issues.
section_title: Only Half of Screening Systems Tested in the Last 5 Years Passed Testing and TSA Procured Almost All of the Successful Systems
paragraphs: Of the 22 PSP and EBSP systems TSA and DHS tested from June 2010  to July 2015, 11 successfully completed qualification and operational  testing, and TSA procured all but one of the 11. An additional 8 systems  were tested during this period and testing remains ongoing. In addition,  during this period one vendor withdrew its system from the testing  process. These 9 systems are not depicted in figure 1 below, which  shows the number of systems that made it through each stage of TSA’s  test and evaluation process during this period.
Additional detail on the outcomes of vendors’ proposed systems follows.
Four additional systems were proposed by vendors during the 5-year  period, but did not proceed to qualification testing because TSA did  not accept the vendors’ qualification data packages. TSA officials told  us they return nearly all qualification data packages to vendors at  least once for additional clarification and documentation.
Four systems did not pass qualification testing and therefore did not  proceed to operational testing. For example, one proposed passenger  screening system did not meet 15 of TSA’s 165 functional  requirements. In this case, TSA identified 6 of the 15 failures as  issues that could adversely affect an operational or mission essential  capability and for which there is no known work-around solution.  Deficiencies for this system included issues with safety, information  security, maintenance, and system dimensions, among others.
Seven systems either did not pass operational testing or had  operational testing halted by TSA during this 5-year period.  Operational testing is the first time a system is subject to realistic  operational demands and used by a variety of transportation security  officers; thus, problems with some systems are generally not detected  until this phase of testing.
Eleven systems successfully passed qualification and operational  testing; TSA procured 10 of them. Of these, TSA placed 9 on its  qualified products lists, meaning that they were eligible for  procurement. TSA procured the 10th system, second generation  advanced imaging technology, independent of a qualified products  list.
TSA officials stated that test and evaluation results are a primary input  into their acquisition decisions, but that they also consider factors such as  mission, threats, and cost. A senior TSA acquisition official explained that  TSA could decide to procure systems that have minor issues during  testing and perform additional testing after the system is in airports to  ensure that the problems are resolved. Similarly, TSA may choose not to  procure systems that successfully complete TSA’s test and evaluation  process. Placement on TSA’s qualified products list allows a vendor to  participate in TSA’s procurement process, but it does not guarantee a  procurement contract. However, TSA has awarded a contract to buy all 9  systems it tested and placed on its qualified products list since June  2010.
Industry officials noted that there is an expectation among vendors that if  they meet the requirements as outlined by TSA they will at least be able  to compete for procurement. We found that, from June 2010 to July 2015,  one system—a portable explosives trace detector system—passed  operational testing but was not procured. This system was intended to  enhance the ability of transportation security officers to randomly screen  passengers’ hands and their accessible property for traces of explosives  residue. In July 2012, following approximately three and a half months of  testing, TSA found that this vendor’s system met requirements, including  probability of detection of required detection threats, and it was found to  be operationally effective and operationally suitable with limitations.  However, according to senior TSA officials, a new threat subsequently  emerged. The TSA Acquisition Review Board, which reviewed the  potential acquisition in advance of the DHS acquisition decision authority,  decided to wait for a system that met TSA’s new detection requirements;  thus, TSA issued an acquisition decision memorandum in November  2012 deferring the procurement. A senior TSA acquisition official stated  that this decision was a positive action because the system did not meet  TSA’s mission needs and therefore should not have been procured at that  time.
section_title: Immature Technologies Have Contributed to Acquisition Inefficiencies
paragraphs: TSA officials emphasized that immature technologies submitted by  vendors are a key driver of testing failures and therefore delays in TSA’s  ability to buy screening systems for use in airports. Because immature  technologies often experience multiple failures during testing and require  multiple retests, testing costs more and takes longer than originally  anticipated. After a testing related failure, vendors will usually modify their  systems to address deficiencies and then re-submit their systems for  additional TSA testing. According to TSA leadership, the security-related  technologies industry is still maturing, since it primarily developed after  the terrorist attacks of September 11, 2001. As a result, TSA has worked  extensively with industry to help develop systems that will meet TSA’s  mission needs. However, TSA’s extensive work with vendors has  contributed to inefficiencies in purchasing screening systems. TSA’s  Assistant Administrator for OSC, who serves as the TSA Chief  Technology Officer, stated that TSA wants to reduce its role in system  development in order to increase efficiency and certainty in the test and  evaluation process so that it can be a true purchaser of these systems.
We found that 4 of the 11 systems that successfully passed TSA’s testing  process since June 2010 required at least two formal rounds of  qualification or operational testing. Testing officials noted that most of the  systems tested during this period required even further testing in addition  to these formal rounds. For example, TSA or DHS also conducted  regression testing to verify that corrections made to the previously failed  systems did not adversely affect system performance, according to these  officials. TSA testing officials stated that nearly all systems need some  modification by the vendor during testing. Industry representatives  involved in testing these systems also told us that systems are not always  mature when they enter TSA’s test and evaluation process, and that they  can require significant modifications and retesting before they are ready  to be bought and deployed to airports.
TSA officials provided us with examples of three explosives detection  systems that required multiple retests, which resulted in acquisition  delays of several years. TSA also ended up spending over $3 million in  additional costs incurred in retesting to ensure the systems were effective  and suitable. All three systems had met explosives detection certification  and false alarm rate criteria during qualification testing at the  Transportation Security Laboratory, but failed to meet additional  requirements during qualification testing at the TSA Systems Integration  Facility and operational testing at selected airports. Some details of these  delays and additional costs are below.
One explosives detection system experienced failures during testing  that led to a 19 month delay to the scheduled full rate production  decision and an additional $1.2 million in testing costs. During  operational testing in summer 2011, TSA found that the system was  effective with minor limitations, but not suitable because it failed to  meet TSA’s reliability requirements and negatively impacted the ability  of transportation security officers to detect threats at a certain level.  The vendor made significant modifications to the system in fall 2012,  and following additional testing DHS approved full rate production of  the system in August 2014.
The second system experienced a 39 month delay to its scheduled  full rate production decision and incurred an additional $1.1 million in  testing costs as a result of multiple rounds of retests. The system  began testing in fall 2010 and after follow-on operational testing in fall  of 2012, TSA found the system to be effective with limitations and not  suitable due to, among other things, issues with resetting or restarting  the system after bag jam failures and reliability. The system  underwent testing by a third party entity in late 2014 to verify that  modifications made to the system improved its performance, and DHS  approved full rate production in May 2015.
At the time of our engagement, the third explosives detection system’s  scheduled full rate production decision had not yet occurred, but it had  already been delayed by more than 60 months, resulting in an  additional $1.2 million in testing costs. TSA began qualification testing  of the system in fall 2010, and, after operational testing in fall 2011,  TSA found the system to be effective with major limitations and  suitable with minor limitations. TSA encountered issues with bag  tracking and reliability, among other things. As of summer 2015, after  five rounds of retests, the vendor was pursuing a third party test of the  system.
According to TSA and DHS testing officials, the actual single rounds of  qualification and operational testing timeframes are not long, but it takes  vendors a significant amount of time to correct issues identified during  testing, which results in a delay to the start of TSA’s next round of testing.  We found in June 2014 that the difficulties in getting vendors systems to  meet the higher level detection for advanced imaging technology required  the vendors to go back and conduct remedial developmental testing  activities. We also found that the qualification testing environment is not  conducive to remediating deficiencies that require extensive additional  development work because feedback to vendors during this phase is  limited. TSA testing officials stated they believe that vendors submit  immature technologies and use the testing process to inform further  system development, but TSA expects systems to be fully mature at the  start of testing.
section_title: Too Soon to Tell Whether TSA’s Actions to Ensure Technology Maturity Will Address All Factors Contributing to Acquisition Inefficiencies
paragraphs: TSA has acknowledged the need to better ensure technology maturity at  the start of testing to improve the efficiency of its test and evaluation  process. In addition to other efforts, a key action TSA is taking to achieve  this goal is developing a third party testing strategy—through which a  third party tester will help ensure systems are mature prior to entering  TSA’s test and evaluation process. TSA plans to implement this strategy  in 2016, but it is too soon to tell whether the strategy will address all of the  factors that contribute to acquisition inefficiencies because TSA has yet to  finalize its key aspects. For example, TSA has not identified whether  there are a sufficient number of eligible third party testers or established a  mechanism to oversee the testing they will perform. Further, TSA did not  conduct a comprehensive assessment of testing data prior to developing  its third party testing strategy and may be missing opportunities to identify  additional areas for improvement in its acquisition process.
subsections: section_title: TSA Is Taking Actions to Better Ensure Technology Maturity at the Start of Testing
paragraphs: As discussed earlier in the report, TSA’s test and evaluation process is  consistent with policies and guidance outlined by DHS and TSA as well  as acquisition best practices; however, TSA has acknowledged the need  to improve the efficiency of the process by better ensuring technology  maturity at the start of testing. For example, the Office of Security  Capabilities’ most recent strategic plan as well as TSA’s overall 2015  Strategic Technology Investment Plan both note that the test and  evaluation process needs to be more efficient and agile in responding to  emerging threats.
TSA has recently initiated two specific reforms to improve the efficiency of  the test and evaluation process: increased transparency with vendors and  a third party testing strategy. First, to increase transparency, TSA officials  told us that they are sharing test plans with vendors to better prepare  them for testing. While industry officials agreed that TSA has become  more transparent, they said that the number of test plans that TSA has  shared thus far have been limited. TSA also created a Test and  Evaluation Process Guide to inform stakeholders—particularly industry— of the test and evaluation process, including the test and evaluation  phases and roles/responsibilities of TSA, DHS, and industry. TSA also  holds periodic industry days to update vendors on planned procurements  and changes to acquisition processes, among other things. However,  TSA test and evaluation guidance necessarily limits the extent to which  testing officials can share information with industry throughout the testing  process. TSA’s Director of Test and Evaluation stressed that, to maintain  the integrity of the test process, they do not intend to provide vendors with  detailed information that could be used to “game” the tests. For example,  according to officials from the Transportation Security Laboratory’s  Independent Test and Evaluation division, the certification and  qualification testing it conducts—unlike developmental testing assistance  made available by the laboratory to vendors—provides, by design, limited  feedback between testers and the vendors, to ensure the integrity of the  test and evaluation process.
Second, TSA is developing a third party testing strategy, which it has  partially implemented for technologies that have already entered the test  and evaluation process. According to interim guidance effective in July  2014, a vendor experiencing a significant failure during testing is required  to fund and undergo third party testing and provide results to TSA,  demonstrating that the system has met the previously failed requirements  before the system is allowed to resume TSA’s testing process. Officials  told us that the guidance exempts detection-related testing. At this time,  according to TSA and DHS officials, third party testing cannot replace the  explosives detection testing conducted by the Transportation Security  Laboratory because no other testing facility has the same capabilities.  Since TSA issued the guidance, one vendor has utilized a third party  tester to verify that an operational availability problem discovered during  follow-on operational testing was resolved. TSA is in the process of  implementing the primary piece of the strategy, which establishes  additional third party testing requirements that vendors must meet before  entering the test and evaluation process. Figure 2 provides an overview  of TSA’s use of third party testing following a technology failure during  testing.
section_title: TSA Is Implementing a Third Party Testing Strategy but Has Not Finalized Its Approach or Conducted a Comprehensive Assessment of Testing Data
subsections: section_title: TSA Has Yet to Finalize Key Aspects of the Third Party Testing Strategy
paragraphs: TSA is developing a third party testing strategy and plans to implement it  for select technologies in 2016. The strategy is a key TSA effort to  improve technological maturity and ensure readiness for testing in order  to reduce the number of technology failures during testing. A key aspect  of this strategy is to strengthen requirements for vendors’ qualification  data packages, which are required to enter the test and evaluation  process. Currently, TSA requires some third party testing to confirm  certain requirements in these data packages, such as compliance with  emissions and radiation standards. But for other requirements, at present  vendors simply can attest that they have been met rather than provide  independent verification. The strategy could increase the number of  requirements that must be independently verified in qualification data  packages for some technologies, though TSA has not yet worked out the  details.
TSA has taken some steps to develop the strategy. For example, TSA is  working in coordination with the National Institute of Standards and  Technology to develop requirements for third party testers in accordance  with international standards for test laboratories. TSA has communicated  the developing strategy to key stakeholders, including DHS and industry,  through meetings with DHS officials, industry days, and a request for  information for potential third party testers. Also, TSA is taking steps to  prioritize the requirements that should be independently verified in vendor  qualification data packages.
However, TSA has yet to finalize key aspects of the strategy to ensure  that it will be successful before it is formally required next year.
TSA officials are unsure of how many third party testers may be able  to provide the necessary services to vendors. Without knowledge of  the number of third party testers that can provide the necessary  services, it is unclear whether there will be enough testers to satisfy  the demand for this service. Further, if the number of testers is limited,  it may increase the cost of testing services.
TSA has not yet established a process for approving or monitoring  third party testers, or established how often they will need to be  recertified. TSA officials stated they may choose to have an  independent party approve third party testers after initial  implementation of the strategy. At the time of our review, TSA was  finalizing the third party test application process for potential third  party testers to apply for TSA approval. TSA officials estimated that  this process would be completed by the end of March 2016.
TSA has not established a mechanism to oversee third party testing.
DHS and TSA officials stated that the extent of DHS’s role is  dependent on how the third party testing strategy is implemented.
TSA officials believe that the strategy will reduce the number of  rounds of retests and decrease the time it takes for TSA to test  systems. However, officials stated that there is no guarantee that third  party testing will shorten acquisition timelines.
TSA officials are unsure whether the third party testing strategy will  save overall acquisition costs, which they have highlighted as a  potential benefit. Specifically, while third party testing will shift the  costs of retesting from TSA to the vendors, industry officials told us it  is probable that vendors will reflect these additional costs in their  pricing. TSA officials responsible for planning and implementing the  third party testing strategy told us they had not assessed the potential  impacts of vendors paying for testing on a system’s costs, or the  possibility that third party testing costs could be a barrier to entering  the market for new vendors.
As we established in prior work, components of sound planning include,  among other items, identifying: problems and causes; resources,  investments, and risks; roles, responsibilities, and coordination; and  integration among and with other entities. If TSA does not finalize key  aspects of its strategy prior to implementing further third party testing  requirements for vendors to enter testing, it may cause unintended  consequences, such as increasing acquisition costs or creating a barrier  to market entry for new vendors, such as small businesses.
section_title: TSA Has Not Conducted a Comprehensive Assessment of Testing Data
paragraphs: At the time of our review, TSA had not conducted a comprehensive  assessment of testing data, such as timeframes for completing testing  and costs incurred, because it lacked a mechanism to track and  consolidate testing data across all technologies, such as testing delays,  costs, timeframes, and results. Thus, TSA does not have any  documented assessment supporting the decision to implement the third  party testing strategy and was unable to provide us with testing  timeframes for each of the 22 systems tested in the past five years. TSA  tracks testing delays and costs in separate systems and provides actual  testing timeframes in the test reports for each system. However, after we  raised this point during the course of our review, TSA officials developed  a master testing tracker to more comprehensively track testing data. The  master tracker consolidates testing data to generate a holistic view of  specific technologies by summarizing testing timeframes, delays, costs,  and progression through testing, among other things. TSA testing officials  stated that the master tracker will help them to support TSA leadership by  consolidating data for each system in one place, thereby eliminating the  need to go through individual test reports to gather specific testing  information, such as testing timeframes.
As noted above, in OSC’s Strategic Plan for 2013 to 2016, it expressed a  commitment to risk-informed, data driven analysis and decision-making.  While the master testing tracker TSA plans to develop is a positive first  step towards more informed decision-making, officials have not  established a plan for assessing the information collected from the  tracker. Conducting a comprehensive assessment of these testing data is  the critical next step. TSA has previously failed to assess information it  collected with respect to third party testing. Specifically, while developing  its third party testing strategy TSA issued a request for information in  December 2012 about potential third party testers and their capabilities.  However, senior TSA testing officials stated that TSA did not evaluate any  of the responses that it received from potential third party testers because  TSA intended the request to be a mechanism to provide vendors with  information. According to Standards for Internal Control in the Federal  Government, ongoing monitoring should occur in the course of normal  operations. In addition, we previously found that agencies can use  performance information to identify problems in existing programs, to try  to identify the causes of problems, and/or to develop corrective actions.  The benefit of collecting performance information is only fully realized  when this information is actually used by agencies to make decisions  oriented toward improving results.
TSA’s actions to address the repeated technology failures during testing  which have led to acquisition inefficiencies—in large part through its third  party testing strategy—focus on improving technological maturity.  However, TSA and industry officials we spoke with identified additional  issues that may be contributing to the inefficiencies, which third party  testing may not address. Specifically, TSA and industry officials  highlighted issues pertaining to the program management offices’  development of acquisition schedules. Some industry officials we met  with stated that the testing timeframes in acquisition schedules,  specifically for explosives detection systems, are unrealistic. For example,  TSA has communicated to vendors that testing for explosives detection  systems will take around three years; however, according to an industry  official this testing typically takes four or more years. Testing officials  stated that they provide the program offices with time estimates for how  long a single round of testing should take, but that it is ultimately the  program office’s decision regarding how many rounds of testing to include  in the schedule. TSA program officials acknowledged that they typically  set optimistic schedules, assuming that technologies will not require  multiple rounds of retesting. However, testing officials have stated it is  rare for technologies to pass testing the first time.
In addition, industry officials identified issues with how requirements have  been defined and interpreted in the past. For example, one industry  official stated that there is a disconnect among TSA, DHS, and vendors  about how operational and functional requirements will be interpreted and  evaluated during the testing process. This official stated that when TSA  tested a system submitted by the company that the official represents, the  vendor and TSA interpreted a requirement about operational availability  differently, specifically with regards to whether the requirement applied to  the system on its own or to the system as integrated into the airport  infrastructure. The vendor failed testing and was required to make system  modifications prior to TSA’s qualification of the system.
TSA testing officials also noted problems with how TSA has defined,  interpreted, and shared requirements. For example, TSA developed two  sets of requirements for its 2010 competitive procurement for explosives  detection systems—requirements that systems must meet and  requirements that TSA would like the systems to meet, which are referred  to as minimum “shall” and non-minimum “shall” requirements,  respectively. However, key stakeholders have disagreed on how these  requirements should be interpreted as systems go through the testing  process. In the case of one explosives detection system, the program  office disagreed with how testing officials evaluated the test results,  noting that the non-minimum “shall” requirements should not have been  considered in the evaluation of effectiveness or suitability. Testing officials  stated that systems needed to meet some of the non-minimum “shall”  requirements in order to fulfill minimum “shall” requirements and therefore  those should have been identified as minimum “shalls.” In addition, TSA  did not release the operational requirements document for explosives  detection systems to vendors prior to testing, and, as a result, vendors did  not know what to expect during operational testing, according to testing  officials.
TSA is taking steps to address some of these issues. For example,  testing officials stated that TSA plans to have one set of requirements for  the next competitive procurement of explosives detection systems. In  addition, TSA’s Mission Analysis Division is focused on improving the  requirements development process through a more rigorous and  structured method that involves the participation of relevant stakeholders,  such as industry, DHS, and end-users, who TSA officials stated have not  historically been involved in the process. In March 2015, industry officials  identified additional actions that TSA could take to improve the  transparency of requirements. Specifically, industry officials provided  formal recommendations to TSA regarding how to further improve the  transparency of requirements, such as developing a limited set of key  requirements, which are critical to executing TSA’s mission, sharing  requirements documents in draft form prior to final release, and defining  and prioritizing requirements for each phase of the testing process,  among other things.
Without conducting and documenting an assessment of testing data  available to date, such as testing timeframes, costs incurred, and testing  delays across all technologies and sharing it with key stakeholders, it is  too soon to tell to what extent TSA’s reforms will address factors  contributing to any acquisition inefficiencies. Specifically, TSA may be  missing opportunities to identify other factors, in addition to technology  immaturity, that are outside the purview of testing officials. An overall  assessment of testing data could inform potential areas for improvement  in the acquisition process, such as incomplete or unclear requirements  and unrealistic acquisition schedules.
section_title: Conclusions
paragraphs: For over a decade, TSA has secured the nation’s civil aviation system  amid continued threats. Due to the significant challenge the agency faces  in balancing security concerns with efficient passenger movement, it is  important that TSA procure and deploy effective passenger and baggage  screening technologies. As the security-related technology industry  evolves to meet new and changing threats, TSA’s test and evaluation  processes are also evolving. Since 2010, though, only half the systems  tested have successfully passed testing and been qualified by TSA for  procurement. Technology failures during testing have resulted in multiple  retests and inefficiencies in acquiring systems that could potentially help  TSA execute its mission.
TSA has taken steps that could improve the maturity of technologies put  forth by industry and reduce the burden on TSA’s own testing resources.  TSA plans to implement its third party testing strategy for selected  technologies in 2016. However, because TSA has not finalized certain  aspects of this strategy—for example, determining how many potential  third party testers are qualified to assist TSA, and how TSA will oversee  this component of testing—there is a risk that the strategy will not be as  effective as envisioned. Unintended consequences, such as increased  acquisition costs, could result. While the third party testing strategy is  likely to help improve system readiness for entering the test process, TSA  has not conducted a comprehensive assessment of testing data for  baggage and passenger screening systems, and therefore cannot be  certain whether the steps currently planned will address other factors  contributing to acquisition inefficiencies. Given TSA’s emphasis on  improving its acquisition and test and evaluation processes, an  assessment of TSA’s testing data would help guide future TSA reforms.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Administrator of TSA take the following two  actions:  1.  To help ensure that actions taken to improve the test and evaluation  process address identified challenges, finalize all aspects of the third  party testing strategy before implementing further third party testing  requirements for vendors to enter testing.  2.  To help ensure that the reforms TSA has underway are informed by  existing information, conduct and document a comprehensive  assessment of testing data available to date, such as timeframes for  completing testing, costs incurred, and testing delays across all  technology areas to identify key factors contributing to any acquisition  inefficiencies and potential areas for reform.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this product to DHS for comment. In its written  comments, reproduced in appendix II, DHS concurred with both of our  recommendations and provided plans of action and estimated completion  dates for them. In response to our first recommendation—that TSA  finalize all aspects of the third party testing strategy before implementing  further third party testing requirements for vendors to enter testing—DHS  stated that TSA plans to begin implementing the third party test program  in a phased approach by December 31, 2016. TSA’s planned actions  include, among other things, finalizing the third party tester application  process, communicating the new approach to industry, and developing  third party testing requirements. We believe that these are positive steps  to finalize the strategy, but that TSA should also consider the potential  consequences of implementing the strategy, such as the cost impacts to  TSA and vendors, which will help to ensure the effectiveness of the  strategy.
DHS also provided technical comments that we incorporated into the  report as appropriate.
We are sending copies of this report to the Secretary of Homeland  Security, the TSA Administrator, and the appropriate congressional  committees. In addition, the report is available at no charge on the GAO  website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-4841 or mackinm@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made significant contributions  to this report are listed in appendix III.
section_title: Appendix I: Scope and Methodology
paragraphs: The Transportation Security Acquisition Reform Act included a provision  for GAO to examine the Transportation Security Administration’s (TSA)  test and evaluation process for security-related technologies. Specifically,  this report examines the extent to which (1) TSA’s test and evaluation  process helps TSA meet mission needs through the acquisition of  passenger and baggage screening technologies, and (2) TSA’s planned  actions to improve the test and evaluation process address factors  contributing to inefficiencies, if any, in acquiring those technologies.
For our first objective, we reviewed Department of Homeland Security  (DHS) and TSA acquisition and testing policies and guidance to  determine the role of test and evaluation in TSA’s acquisition process and  compared them to acquisition best practices. We also analyzed testing  documentation provided to us by TSA for the 22 Passenger Screening  Program (PSP) and Electronic Baggage Screening Program (EBSP)  systems TSA has tested since June 2010, such as test and evaluation  plans, test reports, system evaluation reports, data packages for entry  into testing, and information regarding the number of retests during  testing to determine the number of systems that passed each stage of  TSA’s testing process. In addition, we reviewed TSA and DHS acquisition  documentation for passenger and baggage screening technologies,  including requirements documents, schedules, letters of assessment,  acquisition decision memorandums, contract award notices, and  information regarding the number of each technology deployed in airports  nationwide to determine the number of systems that TSA has tested,  qualified for use, and procured since June 2010. Additionally, we  conducted interviews with TSA officials from the Office of Security  Capabilities (OSC), the Office of Acquisitions, PSP, EBSP, and the  Mission Analysis Division, regarding the acquisition and test and  evaluation processes, the influence of test and evaluation results on  TSA’s decisions about which technologies to procure, and reasons for  any acquisition inefficiencies. We also met with DHS officials from the  Science and Technology Directorate and the Office of Program  Accountability and Risk Management regarding their roles and  responsibilities related to TSA’s test and evaluation process. Further, we  attended a joint industry and TSA workshop hosted by the Airport  Consultants Council focused on TSA’s acquisition process for security- related technologies, including its test and evaluation process. Finally, we  visited the two primary testing facilities for TSA’s security-related  technologies—the TSA Systems Integration Facility, in Arlington, Virginia  and DHS’s testing facility, the Transportation Security Laboratory, in  Atlantic City, New Jersey to interview testing officials and observe the  testing process. We chose these locations because they are TSA’s two  primary testing facilities for security-related technologies.
For our second objective, we reviewed TSA strategic planning  documents, including TSA’s August 2015 Strategic Five-Year Technology  Investment Plan for Aviation Security, OSC’s Strategic Plan for 2013 to  2016, and OSC’s May 2014 Transportation Security Strategic Capability  Investment Plan, to identify TSA’s stated challenges with the test and  evaluation process and its plans for improvement. We also reviewed  TSA’s interim third party testing guidance, implementation timeline, and  briefings to industry as well as testing examples, which served as TSA’s  rationale for implementing the third party testing strategy. In addition, we  conducted interviews with TSA and DHS testing officials, including testers  and evaluators, regarding challenges of TSA’s acquisition and test and  evaluation processes as well as ongoing and planned efforts to improve  testing efficiency and transparency. We also met with representatives  from two industry groups with experience related to TSA’s test and  evaluation process to obtain industry views on any challenges in the test  and evaluation process for screening technologies, areas for potential  improvement, and their perspectives on TSA’s third party testing strategy.  We met with representatives from the Airport Consultants Council’s  Security Manufacturers Coalition, who represented three of TSA’s largest  vendors receiving 8 of PSP and EBSP’s 10 largest contracts to date. We  also met with officials from the Homeland Security and Defense Business  Council, who represented companies providing test and evaluation  support services to DHS and TSA. Additionally, we conducted an  interview with the DHS Director of Operational Test and Evaluation  regarding planned efforts to expand DHS oversight of component-level  testing, including TSA, from operational to developmental testing and its  involvement in the development of TSA’s third party testing strategy.  Further, we met with officials from the National Institute of Standards and  Technology involved in the development of TSA’s third party testing  strategy to determine TSA’s coordination with stakeholders in the  development and implementation of the strategy. Finally, we conducted  interviews with TSA testing officials regarding TSA’s existing and planned  mechanisms for tracking systems throughout the test and evaluation  process and any efforts to assess testing data, such as timeframes for  completing testing and costs incurred and compared them to Standards  for Internal Control in the Federal Government.
We conducted this performance audit from April 2015 to December 2015  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Homeland Security
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Katherine Trimble (Assistant  Director), Charlie Shivers III (Analyst-in-Charge), Peter Anderson, Molly  Callaghan, William Carrigg, Kristine Hassinger, Mark Hoover, Michael  Kaeser, Jean McSween, Lindsay Taylor, and Ozzy Trevino made  significant contributions to this report.