TSA ACQUISITIONS

Further Actions 

Needed to Improve 

Efficiency of 

Screening Technology 

Test and Evaluation 

Report to Congressional Committees

December 2015

GAO-16-117

United States Government Accountability Office
United States Government Accountability Office

Highlights of GAO-16-117, a report to 

congressional committees

December 2015

TSA ACQUISITIONS

Further Actions Needed to Improve Efficiency of 

Screening Technology Test and Evaluation

Why GAO Did This Study

TSA, within the Department of 

Homeland Security, is responsible for

securing the nation’s civil aviation 

system while facilitating the movement 

of passengers and commerce at 

approximately 440 airports nationwide. 

TSA tests passenger and baggage 

screening technologies developed by 

industry to ensure they support TSA 

missions. In reviews from 2010 to 

2014, GAO found that TSA 

encountered challenges in acquiring 

and deploying technologies. The 

Transportation Security Acquisition 

Reform Act contained a provision for 

GAO to assess TSA’s test and 

evaluation activities for security-related 

technologies. This report assesses the 

extent to which (1) TSA’s test and 

evaluation process helps it meet 

mission needs through the acquisition 

of passenger and baggage screening

technologies, and (2) TSA has taken 

steps to improve the test and 

evaluation process.

GAO reviewed DHS and TSA 

acquisition and test policies, analyzed 

testing and acquisition documentation 

for technologies tested in the past five 

years, observed the testing process at 

DHS and TSA facilities, and spoke with 

DHS, TSA, and industry officials.

What GAO Recommends

TSA should finalize its third party 

testing strategy before implementation

and conduct and document a 

comprehensive assessment of testing 

data to identify key factors contributing 

to any acquisition inefficiencies. DHS 

concurred with GAO’s 

recommendations.

What GAO Found

The Transportation Security Administration’s (TSA) test and evaluation process 

has enabled TSA and Department of Homeland Security (DHS) officials to 

identify passenger and baggage screening technologies that will meet mission 

needs, but technology failures during testing have contributed to inefficiencies in 

the acquisition process. Consistent with departmental guidance and acquisition 

best practices, TSA’s test and evaluation process provides information regarding 

the ability of technologies to meet mission needs before agency officials decide 

whether to begin full production, saving the agency from investing in potentially 

expensive yet ineffective equipment. From June 2010 to July 2015, half of the 22 

systems that TSA tested successfully completed qualification and operational 

testing. TSA procured all but 1 of the 11 successful systems. Technologies that 

entered the test and evaluation process and were immature required significant 

modifications and retesting.

Number of Passenger and Baggage Screening Systems (Including Upgrades) Completing TSA 

Test and Evaluation Phases from June 2010 to July 2015 

TSA has taken steps to improve its test and evaluation process by helping 

ensure technologies are mature before entering testing, but it is too soon to tell 

whether these actions will address all of the factors that contribute to acquisition 

inefficiencies. A key action TSA is taking involves developing a third party testing 

strategy, through which a third party will help ensure systems are mature prior to 

entering TSA’s test and evaluation process. TSA plans to implement its approach 

in 2016, but it has yet to finalize key aspects of the strategy. For example, TSA 

has not identified whether there are a sufficient number of eligible third party 

testers or established a mechanism to oversee that testing. Without a finalized 

strategy, TSA risks unintended consequences, such as increasing acquisition 

costs. Further, TSA has not conducted or documented a comprehensive 

assessment of testing data and thus may be missing opportunities to identify 

additional areas for improvements to its acquisition process. An assessment of 

this data, such as costs incurred, could help TSA guide future reforms to the test 

and evaluation process to help ensure they address factors contributing to any 

acquisition inefficiencies.

View GAO-16-117. For more information, 

contact Michele Mackin at (202) 512-4841 or 

mackinm@gao.gov.
Page i GAO-16-117 TSA Acquisitions

Letter 1

Background 3

TSA Has Identified Technologies to Meet Mission Needs, but 

Failures during Testing Have Contributed to Acquisition 

Inefficiencies 10

Too Soon to Tell Whether TSA’s Actions to Ensure Technology 

Maturity Will Address All Factors Contributing to Acquisition 

Inefficiencies 17

Conclusions 25

Recommendations for Executive Action 25

Agency Comments and Our Evaluation 26

Appendix I Scope and Methodology 28

Appendix II Comments from the Department of Homeland Security 31

Appendix III GAO Contact and Staff Acknowledgments 35

Tables

Table 1: Overview of Key Transportation Security Administration 

(TSA) Passenger and Baggage Screening Technologies 3

Table 2: Test and Evaluation Roles and Responsibilities 7

Figures

Figure 1: Number of Passenger and Baggage Screening Systems 

(Including Upgrades) Completing Transportation Security 

Administration Test and Evaluation Phases from June 

2010 to July 2015 12

Figure 2: Transportation Security Administration’s Use of Third 

Party Testing Following a Technology Failure during 

Testing 19

Contents
Page ii GAO-16-117 TSA Acquisitions

Abbreviations

DHS Department of Homeland Security

EBSP Electronic Baggage Screening Program

PSP Passenger Screening Program

OSC Office of Security Capabilities 

TSA Transportation Security Administration

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-16-117 TSA Acquisitions

441 G St. N.W.

Washington, DC 20548

December 17, 2015

Congressional Committees

Commercial aircraft have long been a target of terrorists, who have 

carried out attacks with explosives smuggled onboard in checked 

baggage as well as with passenger-carried contraband and explosives. 

The 1988 bombing of a U.S. aircraft over Lockerbie, Scotland as well as 

the terrorist attacks of September 11, 2001, illustrate the real threat posed 

by terrorists against aircraft. The threat continues today as new and 

innovative methods—such as the use of artfully concealed homemade 

explosives—proliferate. The Transportation Security Administration 

(TSA), a component of the Department of Homeland Security (DHS), is 

responsible for overseeing security operations at the nation’s roughly 440 

commercial airports as part of its mission to protect the nation’s civil 

aviation system. TSA screens individuals, their carry-on luggage, and 

their checked baggage to deter, detect, and prevent carriage of any 

prohibited items, such as explosives and contraband, on board 

commercial aircraft. To carry out these activities, the agency relies to a 

large extent on security-related screening technologies, such as 

explosives detection systems and advanced imaging technology devices.

In our past work, we have found that TSA encountered challenges in 

effectively acquiring and deploying passenger and baggage screening 

technologies and had not consistently implemented DHS policy and best 

practices for procurement.1 In light of these concerns, among other 

issues, Congress enacted the Transportation Security Acquisition Reform 

Act in December 2014, which contained a provision for GAO to assess 

TSA’s test and evaluation activities for security-related technologies. This 

report assesses the extent to which (1) TSA’s test and evaluation process 

helps TSA meet mission needs through the acquisition of passenger and 

baggage screening technologies, and (2) TSA’s planned actions to 

improve the test and evaluation process address factors contributing to 

inefficiencies, if any, in acquiring those technologies.

 1

GAO, Advanced Imaging Technology: TSA Needs Additional Information before 

Procuring Next-Generation Systems, GAO-14-357 (Washington, D.C.: Mar. 31, 2014); in 

January 2012, we issued a classified report on TSA’s procurement and deployment of 

advanced imaging technology at airport checkpoints. 

Letter
Page 2 GAO-16-117 TSA Acquisitions

To assess the extent to which TSA’s test and evaluation process helps 

the agency meet mission needs, we reviewed DHS and TSA acquisition 

documentation for passenger and baggage screening technologies tested 

since June 2010. This documentation included, for example, letters of 

assessment and acquisition decision memorandums. We conducted 

interviews with TSA officials to determine the influence of test and 

evaluation results on TSA’s decisions about which technologies to 

procure and any reasons for repeated testing, which affected TSA’s ability 

to procure the technologies. We also met with relevant DHS officials 

regarding their roles and responsibilities related to TSA’s test and 

evaluation process, which included site visits to the two primary testing 

facilities for TSA’s security-related technologies—the TSA Systems 

Integration Facility in Arlington, Virginia and the DHS Transportation 

Security Laboratory in Atlantic City, New Jersey. To identify the extent to 

which TSA’s planned actions to improve the test and evaluation process 

address factors contributing to acquisition inefficiencies, if any, we 

reviewed TSA’s documentation related to the planned actions. We also 

met with representatives from two industry groups with experience related 

to TSA’s test and evaluation process to obtain industry views on any 

challenges in the test and evaluation process for screening technologies, 

areas for potential improvement, and TSA’s planned actions to improve 

the process. Finally, we conducted interviews with TSA testing officials 

regarding TSA’s existing and planned mechanisms for tracking 

technologies throughout the test and evaluation process, and any efforts 

to assess testing data by looking at, for example, timeframes for 

completing testing and costs incurred. This report focuses on the test and 

evaluation process TSA uses as it acquires screening systems and the 

extent to which it procures the systems that it tests, and does not discuss 

the ability of systems to meet mission needs once deployed in the field. 

Additional details about our scope and methodology are discussed in 

appendix I.

We conducted this performance audit from April 2015 to December 2015 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.
Page 3 GAO-16-117 TSA Acquisitions

TSA uses security-related technologies to help secure approximately 1.8 

million passengers, 1.2 million checked bags, and 3 million carry-on bags 

on 25,000 flights at roughly 440 federally regulated airports every day. As 

of August 2015, TSA had deployed about 15,000 units of security-related 

technology to airports nationwide and anticipates spending a significant 

portion of its $3.6 billion security capability budget on technologies such 

as these over the next 5 years. Within TSA, the Office of Security 

Capabilities (OSC) provides security-related technology solutions through 

two major DHS acquisition programs, the Passenger Screening Program 

(PSP) and the Electronic Baggage Screening Program (EBSP):

• PSP technologies, such as advanced imaging technology and bottled 

liquid scanners, work in combination at airport checkpoints to screen 

passengers and their carry-on baggage for threats.

• EBSP technologies, such as explosives trace detectors and 

explosives detection systems, help ensure that TSA screens 100 

percent of checked baggage for explosives, as mandated by law.2

For each technology type, vendors develop their own specific version or 

system. See table 1 for an overview of key PSP and EBSP technology 

types.

Table 1: Overview of Key Transportation Security Administration (TSA) Passenger and Baggage Screening Technologies

Technology

TSA acquisition 

program Description

Number deployed 

nationwide as of 

August 2015

Number TSA plans 

to purchase fiscal 

year 2016-2020

Advanced Imaging 

Technology 

Passenger 

Screening Program 

(PSP)

TSA uses advanced imaging 

technology to screen passengers for 

threats without physical contact. This 

technology detects a wide range of 

metallic and nonmetallic threats on 

passengers and highlights the 

location of potential threats on a 

generic passenger outline for further 

assessment by transportation 

security officers.

772 245

 2

49 U.S.C. § 44901.

Background

TSA Acquires SecurityRelated Technologies to 

Screen Passengers and 

Baggage for Threats
Page 4 GAO-16-117 TSA Acquisitions

Technology

TSA acquisition 

program Description

Number deployed 

nationwide as of 

August 2015

Number TSA plans 

to purchase fiscal 

year 2016-2020

Advanced Technology XRay PSP TSA uses advanced technology xrays to screen passengers’ carry-on 

baggage. These x-rays use detection 

algorithm software to identify 

potential organic threats such as 

explosives in carry-on baggage and 

provide a horizontal and vertical 

image of each screened item to the 

security officer. 

2,162 986

Bottled Liquid Scanner PSP TSA uses bottled liquid scanners to 

differentiate dangerous from 

nonthreatening liquids carried by 

passengers through checkpoint 

screening. 

1,579 428

Credential Authentication 

Technology 

PSP The purpose of this technology is to 

inspect the legitimacy of passenger 

credentials and confirm passengers’

flight status. Credential authentication 

technology is still going through 

TSA’s test and evaluation process 

and has not yet been deployed to 

airports. 

—- to be determined

Walk-Through Metal 

Detector 

PSP TSA uses walk through metal 

detectors, also known as enhanced 

metal detectors, to rapidly screen 

passengers while maintaining 

compliance with security 

requirements. These systems detect 

magnetic and non-magnetic 

materials.

1,389 1,073 

Explosives Trace Detector 

system

PSP and Electronic 

Baggage Screening 

Program (EBSP)

TSA uses explosives trace detector 

systems to detect explosive 

compounds on airline passengers as 

well as on their carry-on and checked 

baggage.

5,397 1,013

Explosives Detection 

System 

EBSP TSA uses explosives detection 

systems to screen checked baggage. 

These systems detect the presence 

of explosives and improvised 

explosive device components in 

checked baggage. Depending on the 

airport, the systems may be 

standalone or integrated with an 

airport’s baggage handling systems. 

1,717 279

Source: TSA. | GAO-16-117

According to TSA, EBSP and PSP are transitioning into a sustainment 

mode, meaning they are largely not procuring new types of screening 
Page 5 GAO-16-117 TSA Acquisitions

technologies and are instead focused on the recapitalization of over 2,400 

systems that are reaching their end-of-life over the next 5 years.

In our past work, we found key challenges related to TSA’s efforts to 

acquire technologies. For example, in March 2014, we found that TSA’s 

performance assessments of advanced imaging technology systems with 

automatic target recognition capability did not account for all factors 

affecting the effectiveness of the system, such as how well the system 

would perform with human operators, because the assessments were 

conducted in a laboratory.3 We recommended that TSA establish 

protocols to facilitate capturing operational data on secondary passenger 

screening at the checkpoint to determine the extent to which rates of false 

alarms for various advanced imaging technology systems affect 

operational costs once the systems with automatic target recognition 

capability are networked. TSA concurred with this recommendation. In its 

comments to our report, TSA stated that it will monitor, update, and report 

the results of its efforts to capture operational data and evaluate its 

associated impacts to operational costs. In January 2012, we found that 

TSA did not fully follow DHS acquisition policies when acquiring a 

different version of advanced imaging technology, which resulted in DHS 

approving nationwide deployment without full knowledge of TSA’s revised 

specifications.4 Among other things, we found that TSA had changed key 

performance parameters during the acquisition process without formally 

informing DHS acquisition officials, in violation of DHS policy. We 

recommended that TSA develop a road map describing when vendors will 

meet milestones for further developing advanced imaging technology and 

that TSA brief Congress as part of the budget process. Findings from this 

report resulted in a reduction in planned advanced imaging technology 

purchases amounting to approximately $1.4 billion.

DHS policies and processes for managing its major acquisition programs, 

such as TSA’s PSP and EBSP, are primarily set forth in DHS Acquisition 

Management Directive 102-01 and DHS Instruction Manual 102-01-001, 

Acquisition Management Instruction/Guidebook. Within TSA, the PSP and 

EBSP program management offices are responsible for planning and 

 3

GAO-14-357.

4

In January 2012, we issued a classified report on TSA’s procurement and deployment of 

advanced imaging technology at airport checkpoints. 

DHS Acquisition Process
Page 6 GAO-16-117 TSA Acquisitions

executing the acquisition programs within the cost, schedule, and 

performance parameters in their program baselines, which are approved 

by DHS. DHS’s Deputy Secretary or Under Secretary for Management 

serve as the decision authority for PSP and EBSP acquisitions since both 

are large acquisition programs with life cycle cost estimates of over $1 

billion.5 The decision authority reviews programs at five predetermined 

acquisition decision events to assess whether the programs are ready to 

proceed through each of the four acquisition life cycle phases.

During the first phase of the acquisition life cycle—the “need” phase—

TSA develops a mission need statement for a potential solution to an 

identified problem, such as threats concealed on passengers and in their 

carry-on items. If the acquisition decision authority concludes that the 

need is of sufficiently high priority to DHS, the program moves to the 

second phase known as the “analyze/select” phase. The TSA program 

management offices—such as PSP and EBSP—communicate this need 

to potential vendors through requests for information, requests for 

proposals, and broad agency announcements in search of potential 

solutions, such as innovative screening technologies. Program managers 

select the best technology option and, if the acquisition decision authority 

approves the proposed technology, the program moves to phase three. 

The third phase is the “obtain” phase in which TSA develops, tests, and 

evaluates systems—vendors’ versions of the selected technology. With 

acquisition decision authority approval, programs enter the final phase, 

the “produce/deploy/support” phase, and TSA can proceed with 

procurement and full deployment of tested and qualified systems.

The goal of test and evaluation is to make sure that a product works as 

intended before it is provided to end-users, such as TSA’s transportation 

security officers. According to DHS policy, the primary purpose of test and 

evaluation is to provide timely, accurate information to managers, 

decision makers, and other stakeholders to reduce programmatic, 

financial, schedule, and performance risk. TSA’s test and evaluation 

process is guided by DHS and TSA policies. DHS issued its test and 

evaluation policy in May 2009 and TSA issued its policy in July 2010.6

 5

As of August 2015, PSP’s life cycle cost estimate was $2.9 billion and EBSP’s was $20.3 

billion.

6

TSA issued an updated version of its test and evaluation policy in August 2015. 

TSA Test and Evaluation 

Process
Page 7 GAO-16-117 TSA Acquisitions

Within OSC, the Operations Support Division is responsible for testing 

and evaluating screening technologies in coordination with DHS’s 

Transportation Security Laboratory. Table 2 provides an overview of DHS 

and TSA test and evaluation roles and responsibilities.

Table 2: Test and Evaluation Roles and Responsibilities 

Transportation Security Administration 

(TSA)

• Develops Test and Evaluation Master Plans, which are top level planning documents 

for all test and evaluation events

• Reviews vendor-submitted independent test data and analysis—called qualification 

data packages—to ensure that technologies are ready to enter qualification testing

• Conducts operational test readiness reviews to confirm technology readiness for 

operational testing

• Plans and performs operational testing in airports

• Completes system evaluation reports following testing

• Performs acceptance testing on technologies after production and installation to 

ensure consistency of the manufacturing process, system configuration, and 

functionality 

Department of Homeland Security (DHS) 

Science and Technology Directorate, 

Director of Operational Test and 

Evaluation

• Approves TSA’s Test and Evaluation Master Plans and operational test plans

• Participates in operational test readiness reviews and observes operational testing

• Reviews TSA’s system evaluation reports and issues letters of assessment 

evaluating the testing results

DHS Transportation Security Laboratory • Performs research and development of technologies for detecting and mitigating 

threats to transportation security

• Along with TSA, reviews qualification data packages from vendors prior to formal 

qualification testing

• Plans and performs qualification testing on technologies that detect explosives and 

hand weapons

TSA Systems Integration Facility • Plans and performs developmental testing of some technologies and qualification 

testing of all screening technologies to verify technology conformance with technical 

requirements

• Performs regression testing to assess technology modifications made to address 

failures during qualification and operational testing

• Conducts additional test and support activities including small scale tests, proofs of 

concepts, and operational procedure assessments

Source: DHS and TSA. | GAO-16-117

As noted in the table, DHS and TSA both have responsibilities for 

evaluating vendors’ qualification data packages and determining whether 

to accept systems for testing. Once accepted, the systems undergo 

TSA’s three phase pre-acquisition test and evaluation process to verify 

requirements. Responsibility for verifying requirements is assigned to 

testers for each of the three phases: (1) qualification and certification 

testing at the Transportation Security Laboratory, (2) qualification testing 
Page 8 GAO-16-117 TSA Acquisitions

at the TSA Systems Integration Facility, and (3) operational testing at 

selected airports.

1. Qualification and certification testing at the Transportation 

Security Laboratory. Qualification testing is the formal verification 

and validation of a system’s performance and must provide 

confidence that the system will satisfy desired capabilities in an 

operational environment. Qualification testing is typically conducted at 

two facilities, each serving distinct functions. Testing conducted at the 

DHS Science and Technology Directorate’s Transportation Security 

Laboratory in Atlantic City, New Jersey is focused on qualifying or 

certifying that a system meets the probability of detection of all 

categories of explosives while meeting TSA’s designated false alarm 

requirement. Typically, systems complete detection testing at the 

Transportation Security Laboratory prior to undergoing the balance of 

qualification testing at the TSA Systems Integration Facility.

2. Qualification testing at the TSA Systems Integration Facility. TSA 

conducts additional qualification testing at the TSA Systems 

Integration Facility in Arlington, Virginia to verify system performance 

against defined functional requirements, such as system capacities, 

human factors, physical characteristics, user safety, and system 

reliability, availability, and maintainability. This testing may include 

transportation security officers who exercise standard operating 

procedures to provide feedback regarding the system’s functions in a 

controlled environment. Systems must pass qualification testing and 

undergo an operational test readiness review before they can proceed 

to operational testing.

3. Operational testing at select airports. TSA conducts operational 

testing at select airports to evaluate a system’s operational 

effectiveness and suitability in a realistic environment and to ensure 

that the airport infrastructure is ready to accept the system. According 

to TSA, it selects test sites, in coordination with various stakeholders,

using criteria intended to facilitate an unbiased evaluation of the 

systems. TSA injects mock threats during operational testing to 

assess performance of the entire system, including users and 

standard operating procedures. Operational tests focus on 

demonstrating that operational requirements have been met and that 

all critical operational issues have been resolved.

Following operational testing, TSA prepares a system evaluation report, 

which provides effectiveness and suitability determinations along with 

potential system improvement recommendations. The DHS Director of 

Operational Test and Evaluation reviews the report and prepares a letter 
Page 9 GAO-16-117 TSA Acquisitions

of assessment. The letter assesses the system evaluation report and the 

adequacy of TSA’s operational test and concludes whether the system is 

operationally suitable and effective for procurement. TSA’s program 

management offices place approved systems on qualified products lists, 

which contain systems that have successfully completed the test and 

evaluation process and have been approved by DHS. Once a system is 

on a qualified products list, which TSA maintains for most technology 

types, such as explosives detection systems, the vendor can participate 

in TSA’s procurement process.7 However, placement on a qualified 

products list does not guarantee that the vendor will receive a 

procurement contract.

TSA conducts acceptance testing on systems at vendors’ factories 

following production and again once installed at airports to ensure 

consistency of the manufacturing process, system configuration, and 

functionality. Post deployment, TSA’s Office of Inspection and the DHS 

Office of Inspector General conduct covert tests in airports to identify 

vulnerabilities in screening processes. TSA and DHS test and evaluate

systems prior to procurement, but covert testing has demonstrated that 

systems, when integrated into TSA’s screening process, do not always 

work as intended once deployed. TSA’s test and evaluation activities are 

funded by the program offices, and TSA’s life cycle cost estimate for 

EBSP reflects average annual testing costs of approximately $24 million. 

PSP officials estimate that the program averages $10 million in annual 

testing costs.

TSA’s security screening must adapt to meet evolving threats since 

potential terrorists use their knowledge of aviation security measures 

when planning aviation-related attacks. New explosives threats can result 

in heightened detection standards, which may require software upgrades 

to existing systems or the development of new technologies. DHS and 

TSA partner with industry to build, test, field, and sustain security-related 

technologies. For example, DHS makes available testing facilities and 

expertise at the Transportation Security Laboratory that vendors can use, 

by their choice, to further the development and evaluation of their 

systems. TSA’s August 2015 Strategic Five-Year Technology Investment 

 7

TSA does not maintain a qualified products list for second generation advanced imaging 

technology. As of August 2015 it had only qualified one vendor’s system for use.

DHS and TSA Partner with 

Industry to Develop and 

Deploy Screening 

Technologies in a Dynamic 

Threat Environment
Page 10 GAO-16-117 TSA Acquisitions

Plan for Aviation Security noted the market for security-related 

technologies is quite limited, with only a handful of vendors, which creates 

a significant challenge for small businesses. According to TSA, 

substantial funding and time are required by industry to develop, qualify, 

and produce TSA’s screening technologies, which represents a significant 

barrier to entry. For example, the procurement cost of an explosives 

detection system can exceed over half a million dollars. TSA’s 

approximately 15,000 systems were manufactured by 13 different 

vendors, with 5 vendors accounting for approximately 81 percent of 

deployed systems.

Consistent with departmental guidance and acquisition best practices, 

TSA’s test and evaluation process supports its acquisition decisions by 

providing information regarding the ability of passenger and baggage 

screening technologies to meet mission needs prior to full production 

decisions. From June 2010 to July 2015, only half of the 22 systems that 

TSA and DHS tested successfully passed qualification and operational 

testing and were therefore deemed effective and suitable for deployment. 

TSA procured all but one of the successful systems. Technology failures 

during testing, as a result of vendors’ immature technologies entering the 

test and evaluation process, often required significant fixes and have 

contributed to inefficiencies in TSA acquiring the technologies for use in 

airports.

TSA’s test and evaluation process is a critical means of providing DHS 

and TSA officials with accurate information about security-related 

technologies to support acquisition decisions. We found that the process 

is consistent with DHS and TSA policies and guidance and helps TSA 

ensure that the technologies it acquires fulfill mission needs upon 

deployment. For example, TSA has integrated end users, such as 

transportation security officers into its test and evaluation process, 

consistent with TSA test and evaluation policy. Also, TSA officials said 

they have designated program office liaisons to improve communication 

between testing officials and the TSA program offices responsible for 

procuring the screening technologies to help ensure program offices have 

the testing information they need to inform the procurement decision, in 

line with DHS test and evaluation policy. In addition, TSA establishes 

system evaluation teams for each system undergoing test and evaluation; 

the teams consist of the TSA and DHS officials that plan for and evaluate 

system effectiveness and suitability throughout the testing process. 

Further, TSA’s test and evaluation process is designed to identify and 

TSA Has Identified 

Technologies to Meet 

Mission Needs, but 

Failures during 

Testing Have 

Contributed to 

Acquisition 

Inefficiencies

Test and Evaluation 

Process Is Consistent with 

Guidance and Best 

Practices and Identifies 

Technologies That Meet 

Mission Needs
Page 11 GAO-16-117 TSA Acquisitions

evaluate existing technologies that can be adjusted or repurposed to 

meet the agency’s needs. This goal is in line with department guidance 

underlining the importance of pursuing technologies, specifically 

commercial off-the-shelf technologies that do not require a substantial 

investment of time and money to ensure they are effective once procured.

We previously found that the validation of product knowledge early in the 

acquisition process and before key investments are made is consistent 

with best practices by commercial firms.8 Specifically, leading commercial 

firms strive to detect as many problems as possible during testing, which 

in turn leads to easier and less expensive improvements to products 

down the road. Leading commercial firms highlighted the importance of 

perceiving problems during testing, not as failures, but as knowledge 

which can be used to improve a product. Consistent with this practice, 

while vendors’ proposed systems may fail TSA’s test and evaluation 

process, these failures can help the government identify needed fixes to 

better ensure that they will be effective and suitable prior to being 

deployed to airports nationwide. Further, we previously found that cost 

overruns and underperformance of technologies are exacerbated when 

problems discovered during testing are not resolved. For example, costly 

redesigns and retrofits could be required to achieve satisfactory 

performance of units already deployed in the field. By conducting 

operational testing on systems prior to procurement, TSA is working to 

prevent such issues.

Of the 22 PSP and EBSP systems TSA and DHS tested from June 2010 

to July 2015, 11 successfully completed qualification and operational 

testing, and TSA procured all but one of the 11. An additional 8 systems 

were tested during this period and testing remains ongoing. In addition, 

during this period one vendor withdrew its system from the testing 

process. These 9 systems are not depicted in figure 1 below, which 

shows the number of systems that made it through each stage of TSA’s 

test and evaluation process during this period.

 8

GAO, Best Practices: A More Constructive Test Approach is Key to Better Weapon 

System Outcomes, GAO/NSIAD-00-199 (Washington, D.C.: July 31, 2000).

Only Half of Screening 

Systems Tested in the Last 

5 Years Passed Testing 

and TSA Procured Almost 

All of the Successful 

Systems
Page 12 GAO-16-117 TSA Acquisitions

Figure 1: Number of Passenger and Baggage Screening Systems (Including 

Upgrades) Completing Transportation Security Administration Test and Evaluation 

Phases from June 2010 to July 2015

Note: Of the 12 systems that did not proceed to the procurement stage, TSA could decide to restart 

testing for 6 systems if vendors submit revised qualification data packages to re-enter the testing 

process.

Additional detail on the outcomes of vendors’ proposed systems follows.

• Four additional systems were proposed by vendors during the 5-year 

period, but did not proceed to qualification testing because TSA did 

not accept the vendors’ qualification data packages. TSA officials told 
Page 13 GAO-16-117 TSA Acquisitions

us they return nearly all qualification data packages to vendors at 

least once for additional clarification and documentation.

• Four systems did not pass qualification testing and therefore did not 

proceed to operational testing. For example, one proposed passenger 

screening system did not meet 15 of TSA’s 165 functional 

requirements. In this case, TSA identified 6 of the 15 failures as 

issues that could adversely affect an operational or mission essential 

capability and for which there is no known work-around solution. 

Deficiencies for this system included issues with safety, information 

security, maintenance, and system dimensions, among others.

• Seven systems either did not pass operational testing or had 

operational testing halted by TSA during this 5-year period. 

Operational testing is the first time a system is subject to realistic 

operational demands and used by a variety of transportation security 

officers; thus, problems with some systems are generally not detected 

until this phase of testing.

• Eleven systems successfully passed qualification and operational 

testing; TSA procured 10 of them. Of these, TSA placed 9 on its 

qualified products lists, meaning that they were eligible for

procurement. TSA procured the 10th system, second generation 

advanced imaging technology, independent of a qualified products 

list.9

TSA officials stated that test and evaluation results are a primary input 

into their acquisition decisions, but that they also consider factors such as 

mission, threats, and cost. A senior TSA acquisition official explained that 

TSA could decide to procure systems that have minor issues during 

testing and perform additional testing after the system is in airports to 

ensure that the problems are resolved. Similarly, TSA may choose not to 

procure systems that successfully complete TSA’s test and evaluation 

process. Placement on TSA’s qualified products list allows a vendor to 

participate in TSA’s procurement process, but it does not guarantee a 

procurement contract. However, TSA has awarded a contract to buy all 9 

systems it tested and placed on its qualified products list since June 

2010.

 9

TSA procured second generation advanced imaging technology through a low rate initial 

production approach, which allowed TSA to purchase systems for testing without a 

guarantee of purchasing future systems based on vendor proposals that state that the 

vendor can meet the requirements.
Page 14 GAO-16-117 TSA Acquisitions

Industry officials noted that there is an expectation among vendors that if 

they meet the requirements as outlined by TSA they will at least be able 

to compete for procurement. We found that, from June 2010 to July 2015, 

one system—a portable explosives trace detector system—passed 

operational testing but was not procured. This system was intended to 

enhance the ability of transportation security officers to randomly screen 

passengers’ hands and their accessible property for traces of explosives 

residue. In July 2012, following approximately three and a half months of 

testing, TSA found that this vendor’s system met requirements, including 

probability of detection of required detection threats, and it was found to 

be operationally effective and operationally suitable with limitations. 

However, according to senior TSA officials, a new threat subsequently 

emerged. The TSA Acquisition Review Board, which reviewed the 

potential acquisition in advance of the DHS acquisition decision authority, 

decided to wait for a system that met TSA’s new detection requirements; 

thus, TSA issued an acquisition decision memorandum in November 

2012 deferring the procurement. A senior TSA acquisition official stated 

that this decision was a positive action because the system did not meet 

TSA’s mission needs and therefore should not have been procured at that 

time.

TSA officials emphasized that immature technologies submitted by 

vendors are a key driver of testing failures and therefore delays in TSA’s 

ability to buy screening systems for use in airports. Because immature 

technologies often experience multiple failures during testing and require 

multiple retests, testing costs more and takes longer than originally 

anticipated. After a testing related failure, vendors will usually modify their 

systems to address deficiencies and then re-submit their systems for 

additional TSA testing. According to TSA leadership, the security-related 

technologies industry is still maturing, since it primarily developed after 

the terrorist attacks of September 11, 2001. As a result, TSA has worked 

extensively with industry to help develop systems that will meet TSA’s 

mission needs. However, TSA’s extensive work with vendors has 

contributed to inefficiencies in purchasing screening systems. TSA’s 

Assistant Administrator for OSC, who serves as the TSA Chief 

Technology Officer, stated that TSA wants to reduce its role in system 

development in order to increase efficiency and certainty in the test and 

evaluation process so that it can be a true purchaser of these systems.

We found that 4 of the 11 systems that successfully passed TSA’s testing 

process since June 2010 required at least two formal rounds of 

qualification or operational testing. Testing officials noted that most of the 

Immature Technologies 

Have Contributed to 

Acquisition Inefficiencies
Page 15 GAO-16-117 TSA Acquisitions

systems tested during this period required even further testing in addition 

to these formal rounds. For example, TSA or DHS also conducted 

regression testing to verify that corrections made to the previously failed 

systems did not adversely affect system performance, according to these 

officials. TSA testing officials stated that nearly all systems need some 

modification by the vendor during testing. Industry representatives 

involved in testing these systems also told us that systems are not always 

mature when they enter TSA’s test and evaluation process, and that they 

can require significant modifications and retesting before they are ready 

to be bought and deployed to airports.

TSA officials provided us with examples of three explosives detection 

systems that required multiple retests, which resulted in acquisition 

delays of several years. TSA also ended up spending over $3 million in 

additional costs incurred in retesting to ensure the systems were effective 

and suitable. All three systems had met explosives detection certification 

and false alarm rate criteria during qualification testing at the 

Transportation Security Laboratory, but failed to meet additional 

requirements during qualification testing at the TSA Systems Integration 

Facility and operational testing at selected airports. Some details of these 

delays and additional costs are below.

• One explosives detection system experienced failures during testing 

that led to a 19 month delay to the scheduled full rate production 

decision and an additional $1.2 million in testing costs. During 

operational testing in summer 2011, TSA found that the system was 

effective with minor limitations, but not suitable because it failed to 

meet TSA’s reliability requirements and negatively impacted the ability 

of transportation security officers to detect threats at a certain level. 

The vendor made significant modifications to the system in fall 2012, 

and following additional testing DHS approved full rate production of 

the system in August 2014.

• The second system experienced a 39 month delay to its scheduled 

full rate production decision and incurred an additional $1.1 million in 

testing costs as a result of multiple rounds of retests. The system 

began testing in fall 2010 and after follow-on operational testing in fall 

of 2012, TSA found the system to be effective with limitations and not 

suitable due to, among other things, issues with resetting or restarting 

the system after bag jam failures and reliability. The system 

underwent testing by a third party entity in late 2014 to verify that 

modifications made to the system improved its performance, and DHS 

approved full rate production in May 2015.
Page 16 GAO-16-117 TSA Acquisitions

• At the time of our engagement, the third explosives detection system’s 

scheduled full rate production decision had not yet occurred, but it had 

already been delayed by more than 60 months, resulting in an 

additional $1.2 million in testing costs. TSA began qualification testing 

of the system in fall 2010, and, after operational testing in fall 2011, 

TSA found the system to be effective with major limitations and 

suitable with minor limitations. TSA encountered issues with bag 

tracking and reliability, among other things. As of summer 2015, after 

five rounds of retests, the vendor was pursuing a third party test of the 

system.

According to TSA and DHS testing officials, the actual single rounds of 

qualification and operational testing timeframes are not long, but it takes 

vendors a significant amount of time to correct issues identified during 

testing, which results in a delay to the start of TSA’s next round of testing. 

We found in June 2014 that the difficulties in getting vendors systems to 

meet the higher level detection for advanced imaging technology required 

the vendors to go back and conduct remedial developmental testing 

activities. We also found that the qualification testing environment is not 

conducive to remediating deficiencies that require extensive additional 

development work because feedback to vendors during this phase is 

limited.10 TSA testing officials stated they believe that vendors submit 

immature technologies and use the testing process to inform further 

system development, but TSA expects systems to be fully mature at the 

start of testing.

 10In June 2014, we issued a sensitive, but unclassified report on TSA’s developmental 

testing phase of first and second generation advanced imaging technology.
Page 17 GAO-16-117 TSA Acquisitions

TSA has acknowledged the need to better ensure technology maturity at 

the start of testing to improve the efficiency of its test and evaluation 

process. In addition to other efforts, a key action TSA is taking to achieve 

this goal is developing a third party testing strategy—through which a 

third party tester will help ensure systems are mature prior to entering 

TSA’s test and evaluation process. TSA plans to implement this strategy 

in 2016, but it is too soon to tell whether the strategy will address all of the 

factors that contribute to acquisition inefficiencies because TSA has yet to 

finalize its key aspects. For example, TSA has not identified whether 

there are a sufficient number of eligible third party testers or established a 

mechanism to oversee the testing they will perform. Further, TSA did not 

conduct a comprehensive assessment of testing data prior to developing 

its third party testing strategy and may be missing opportunities to identify 

additional areas for improvement in its acquisition process.

As discussed earlier in the report, TSA’s test and evaluation process is 

consistent with policies and guidance outlined by DHS and TSA as well 

as acquisition best practices; however, TSA has acknowledged the need 

to improve the efficiency of the process by better ensuring technology 

maturity at the start of testing. For example, the Office of Security 

Capabilities’ most recent strategic plan as well as TSA’s overall 2015 

Strategic Technology Investment Plan both note that the test and 

evaluation process needs to be more efficient and agile in responding to 

emerging threats.

TSA has recently initiated two specific reforms to improve the efficiency of 

the test and evaluation process: increased transparency with vendors and 

a third party testing strategy. First, to increase transparency, TSA officials 

told us that they are sharing test plans with vendors to better prepare 

them for testing. While industry officials agreed that TSA has become 

more transparent, they said that the number of test plans that TSA has 

shared thus far have been limited. TSA also created a Test and 

Evaluation Process Guide to inform stakeholders—particularly industry—

of the test and evaluation process, including the test and evaluation 

phases and roles/responsibilities of TSA, DHS, and industry. TSA also 

holds periodic industry days to update vendors on planned procurements 

and changes to acquisition processes, among other things. However, 

TSA test and evaluation guidance necessarily limits the extent to which 

testing officials can share information with industry throughout the testing 

process. TSA’s Director of Test and Evaluation stressed that, to maintain 

the integrity of the test process, they do not intend to provide vendors with 

detailed information that could be used to “game” the tests. For example, 

Too Soon to Tell 

Whether TSA’s 

Actions to Ensure 

Technology Maturity 

Will Address All 

Factors Contributing 

to Acquisition 

Inefficiencies

TSA Is Taking Actions to

Better Ensure Technology 

Maturity at the Start of 

Testing
Page 18 GAO-16-117 TSA Acquisitions

according to officials from the Transportation Security Laboratory’s 

Independent Test and Evaluation division, the certification and 

qualification testing it conducts—unlike developmental testing assistance 

made available by the laboratory to vendors—provides, by design, limited 

feedback between testers and the vendors, to ensure the integrity of the 

test and evaluation process.

Second, TSA is developing a third party testing strategy, which it has 

partially implemented for technologies that have already entered the test 

and evaluation process. According to interim guidance effective in July 

2014, a vendor experiencing a significant failure during testing is required 

to fund and undergo third party testing and provide results to TSA, 

demonstrating that the system has met the previously failed requirements 

before the system is allowed to resume TSA’s testing process. Officials 

told us that the guidance exempts detection-related testing. At this time, 

according to TSA and DHS officials, third party testing cannot replace the 

explosives detection testing conducted by the Transportation Security 

Laboratory because no other testing facility has the same capabilities. 

Since TSA issued the guidance, one vendor has utilized a third party 

tester to verify that an operational availability problem discovered during 

follow-on operational testing was resolved. TSA is in the process of

implementing the primary piece of the strategy, which establishes

additional third party testing requirements that vendors must meet before 

entering the test and evaluation process. Figure 2 provides an overview 

of TSA’s use of third party testing following a technology failure during 

testing.
Page 19 GAO-16-117 TSA Acquisitions

Figure 2: Transportation Security Administration’s Use of Third Party Testing Following a Technology Failure during Testing
Page 20 GAO-16-117 TSA Acquisitions

TSA is developing a third party testing strategy and plans to implement it 

for select technologies in 2016. The strategy is a key TSA effort to 

improve technological maturity and ensure readiness for testing in order 

to reduce the number of technology failures during testing. A key aspect 

of this strategy is to strengthen requirements for vendors’ qualification 

data packages, which are required to enter the test and evaluation 

process. Currently, TSA requires some third party testing to confirm 

certain requirements in these data packages, such as compliance with 

emissions and radiation standards. But for other requirements, at present 

vendors simply can attest that they have been met rather than provide 

independent verification. The strategy could increase the number of 

requirements that must be independently verified in qualification data 

packages for some technologies, though TSA has not yet worked out the 

details.

TSA has taken some steps to develop the strategy. For example, TSA is 

working in coordination with the National Institute of Standards and 

Technology to develop requirements for third party testers in accordance 

with international standards for test laboratories. TSA has communicated 

the developing strategy to key stakeholders, including DHS and industry, 

through meetings with DHS officials, industry days, and a request for 

information for potential third party testers. Also, TSA is taking steps to 

prioritize the requirements that should be independently verified in vendor 

qualification data packages.

However, TSA has yet to finalize key aspects of the strategy to ensure 

that it will be successful before it is formally required next year.

• TSA officials are unsure of how many third party testers may be able 

to provide the necessary services to vendors. Without knowledge of 

the number of third party testers that can provide the necessary 

services, it is unclear whether there will be enough testers to satisfy 

TSA Is Implementing a 

Third Party Testing 

Strategy but Has Not 

Finalized Its Approach or 

Conducted a 

Comprehensive 

Assessment of Testing 

Data

TSA Has Yet to Finalize Key 

Aspects of the Third Party 

Testing Strategy
Page 21 GAO-16-117 TSA Acquisitions

the demand for this service. Further, if the number of testers is limited, 

it may increase the cost of testing services.

• TSA has not yet established a process for approving or monitoring 

third party testers, or established how often they will need to be 

recertified. TSA officials stated they may choose to have an 

independent party approve third party testers after initial 

implementation of the strategy. At the time of our review, TSA was 

finalizing the third party test application process for potential third 

party testers to apply for TSA approval. TSA officials estimated that 

this process would be completed by the end of March 2016.

• TSA has not established a mechanism to oversee third party testing. 

DHS and TSA officials stated that the extent of DHS’s role is 

dependent on how the third party testing strategy is implemented.

• TSA officials believe that the strategy will reduce the number of 

rounds of retests and decrease the time it takes for TSA to test 

systems. However, officials stated that there is no guarantee that third 

party testing will shorten acquisition timelines.

• TSA officials are unsure whether the third party testing strategy will 

save overall acquisition costs, which they have highlighted as a 

potential benefit. Specifically, while third party testing will shift the 

costs of retesting from TSA to the vendors, industry officials told us it 

is probable that vendors will reflect these additional costs in their 

pricing. TSA officials responsible for planning and implementing the 

third party testing strategy told us they had not assessed the potential 

impacts of vendors paying for testing on a system’s costs, or the 

possibility that third party testing costs could be a barrier to entering 

the market for new vendors.

As we established in prior work, components of sound planning include, 

among other items, identifying: problems and causes; resources, 

investments, and risks; roles, responsibilities, and coordination; and 

integration among and with other entities.11 If TSA does not finalize key 

aspects of its strategy prior to implementing further third party testing 

requirements for vendors to enter testing, it may cause unintended 

consequences, such as increasing acquisition costs or creating a barrier 

to market entry for new vendors, such as small businesses.

 11GAO, Social Security Disability: Additional Performance Measures and Better Cost 

Estimates Could Help Improve SSA’s Efforts to Eliminate Its Hearings Backlog, 

GAO-09-398 (Washington, D.C.: Sept. 9, 2009). 
Page 22 GAO-16-117 TSA Acquisitions

At the time of our review, TSA had not conducted a comprehensive 

assessment of testing data, such as timeframes for completing testing 

and costs incurred, because it lacked a mechanism to track and 

consolidate testing data across all technologies, such as testing delays, 

costs, timeframes, and results. Thus, TSA does not have any 

documented assessment supporting the decision to implement the third 

party testing strategy and was unable to provide us with testing 

timeframes for each of the 22 systems tested in the past five years. TSA 

tracks testing delays and costs in separate systems and provides actual 

testing timeframes in the test reports for each system. However, after we 

raised this point during the course of our review, TSA officials developed

a master testing tracker to more comprehensively track testing data. The 

master tracker consolidates testing data to generate a holistic view of 

specific technologies by summarizing testing timeframes, delays, costs, 

and progression through testing, among other things. TSA testing officials 

stated that the master tracker will help them to support TSA leadership by 

consolidating data for each system in one place, thereby eliminating the 

need to go through individual test reports to gather specific testing 

information, such as testing timeframes.

As noted above, in OSC’s Strategic Plan for 2013 to 2016, it expressed a 

commitment to risk-informed, data driven analysis and decision-making. 

While the master testing tracker TSA plans to develop is a positive first 

step towards more informed decision-making, officials have not 

established a plan for assessing the information collected from the 

tracker. Conducting a comprehensive assessment of these testing data is 

the critical next step. TSA has previously failed to assess information it 

collected with respect to third party testing. Specifically, while developing 

its third party testing strategy TSA issued a request for information in 

December 2012 about potential third party testers and their capabilities. 

However, senior TSA testing officials stated that TSA did not evaluate any 

of the responses that it received from potential third party testers because 

TSA intended the request to be a mechanism to provide vendors with 

information. According to Standards for Internal Control in the Federal 

Government, ongoing monitoring should occur in the course of normal 

operations.12 In addition, we previously found that agencies can use 

performance information to identify problems in existing programs, to try 

 12GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1

(Washington, D.C.: November 1999).

TSA Has Not Conducted a 

Comprehensive Assessment of 

Testing Data
Page 23 GAO-16-117 TSA Acquisitions

to identify the causes of problems, and/or to develop corrective actions. 

The benefit of collecting performance information is only fully realized 

when this information is actually used by agencies to make decisions 

oriented toward improving results.13

TSA’s actions to address the repeated technology failures during testing 

which have led to acquisition inefficiencies—in large part through its third 

party testing strategy—focus on improving technological maturity. 

However, TSA and industry officials we spoke with identified additional 

issues that may be contributing to the inefficiencies, which third party 

testing may not address. Specifically, TSA and industry officials 

highlighted issues pertaining to the program management offices’ 

development of acquisition schedules. Some industry officials we met 

with stated that the testing timeframes in acquisition schedules, 

specifically for explosives detection systems, are unrealistic. For example, 

TSA has communicated to vendors that testing for explosives detection 

systems will take around three years; however, according to an industry 

official this testing typically takes four or more years. Testing officials 

stated that they provide the program offices with time estimates for how 

long a single round of testing should take, but that it is ultimately the 

program office’s decision regarding how many rounds of testing to include 

in the schedule. TSA program officials acknowledged that they typically 

set optimistic schedules, assuming that technologies will not require 

multiple rounds of retesting. However, testing officials have stated it is 

rare for technologies to pass testing the first time.

In addition, industry officials identified issues with how requirements have 

been defined and interpreted in the past. For example, one industry 

official stated that there is a disconnect among TSA, DHS, and vendors 

about how operational and functional requirements will be interpreted and 

evaluated during the testing process. This official stated that when TSA 

tested a system submitted by the company that the official represents, the 

vendor and TSA interpreted a requirement about operational availability 

differently, specifically with regards to whether the requirement applied to 

the system on its own or to the system as integrated into the airport 

infrastructure. The vendor failed testing and was required to make system 

modifications prior to TSA’s qualification of the system.

 13GAO, Managing for Results: Enhancing Agency Use of Performance Information for 

Management Decision Making, GAO-05-927 (Washington, D.C.: Sept. 9, 2005).
Page 24 GAO-16-117 TSA Acquisitions

TSA testing officials also noted problems with how TSA has defined, 

interpreted, and shared requirements. For example, TSA developed two 

sets of requirements for its 2010 competitive procurement for explosives 

detection systems—requirements that systems must meet and 

requirements that TSA would like the systems to meet, which are referred 

to as minimum “shall” and non-minimum “shall” requirements, 

respectively. However, key stakeholders have disagreed on how these 

requirements should be interpreted as systems go through the testing 

process. In the case of one explosives detection system, the program 

office disagreed with how testing officials evaluated the test results, 

noting that the non-minimum “shall” requirements should not have been 

considered in the evaluation of effectiveness or suitability. Testing officials 

stated that systems needed to meet some of the non-minimum “shall”

requirements in order to fulfill minimum “shall” requirements and therefore 

those should have been identified as minimum “shalls.” In addition, TSA 

did not release the operational requirements document for explosives 

detection systems to vendors prior to testing, and, as a result, vendors did 

not know what to expect during operational testing, according to testing 

officials.

TSA is taking steps to address some of these issues. For example, 

testing officials stated that TSA plans to have one set of requirements for 

the next competitive procurement of explosives detection systems. In 

addition, TSA’s Mission Analysis Division is focused on improving the 

requirements development process through a more rigorous and 

structured method that involves the participation of relevant stakeholders, 

such as industry, DHS, and end-users, who TSA officials stated have not 

historically been involved in the process. In March 2015, industry officials 

identified additional actions that TSA could take to improve the 

transparency of requirements. Specifically, industry officials provided 

formal recommendations to TSA regarding how to further improve the 

transparency of requirements, such as developing a limited set of key 

requirements, which are critical to executing TSA’s mission, sharing 

requirements documents in draft form prior to final release, and defining 

and prioritizing requirements for each phase of the testing process, 

among other things.

Without conducting and documenting an assessment of testing data 

available to date, such as testing timeframes, costs incurred, and testing 

delays across all technologies and sharing it with key stakeholders, it is 

too soon to tell to what extent TSA’s reforms will address factors 

contributing to any acquisition inefficiencies. Specifically, TSA may be 

missing opportunities to identify other factors, in addition to technology 
Page 25 GAO-16-117 TSA Acquisitions

immaturity, that are outside the purview of testing officials. An overall 

assessment of testing data could inform potential areas for improvement 

in the acquisition process, such as incomplete or unclear requirements 

and unrealistic acquisition schedules.

For over a decade, TSA has secured the nation’s civil aviation system 

amid continued threats. Due to the significant challenge the agency faces 

in balancing security concerns with efficient passenger movement, it is 

important that TSA procure and deploy effective passenger and baggage 

screening technologies. As the security-related technology industry 

evolves to meet new and changing threats, TSA’s test and evaluation 

processes are also evolving. Since 2010, though, only half the systems 

tested have successfully passed testing and been qualified by TSA for 

procurement. Technology failures during testing have resulted in multiple 

retests and inefficiencies in acquiring systems that could potentially help 

TSA execute its mission.

TSA has taken steps that could improve the maturity of technologies put 

forth by industry and reduce the burden on TSA’s own testing resources. 

TSA plans to implement its third party testing strategy for selected 

technologies in 2016. However, because TSA has not finalized certain 

aspects of this strategy—for example, determining how many potential 

third party testers are qualified to assist TSA, and how TSA will oversee 

this component of testing—there is a risk that the strategy will not be as 

effective as envisioned. Unintended consequences, such as increased 

acquisition costs, could result. While the third party testing strategy is 

likely to help improve system readiness for entering the test process, TSA 

has not conducted a comprehensive assessment of testing data for 

baggage and passenger screening systems, and therefore cannot be 

certain whether the steps currently planned will address other factors 

contributing to acquisition inefficiencies. Given TSA’s emphasis on 

improving its acquisition and test and evaluation processes, an 

assessment of TSA’s testing data would help guide future TSA reforms.

We recommend that the Administrator of TSA take the following two 

actions:

1. To help ensure that actions taken to improve the test and evaluation 

process address identified challenges, finalize all aspects of the third 

party testing strategy before implementing further third party testing 

requirements for vendors to enter testing.

Conclusions

Recommendations for 

Executive Action
Page 26 GAO-16-117 TSA Acquisitions

2. To help ensure that the reforms TSA has underway are informed by 

existing information, conduct and document a comprehensive 

assessment of testing data available to date, such as timeframes for 

completing testing, costs incurred, and testing delays across all 

technology areas to identify key factors contributing to any acquisition 

inefficiencies and potential areas for reform.

We provided a draft of this product to DHS for comment. In its written

comments, reproduced in appendix II, DHS concurred with both of our 

recommendations and provided plans of action and estimated completion 

dates for them. In response to our first recommendation—that TSA 

finalize all aspects of the third party testing strategy before implementing 

further third party testing requirements for vendors to enter testing—DHS 

stated that TSA plans to begin implementing the third party test program 

in a phased approach by December 31, 2016. TSA’s planned actions 

include, among other things, finalizing the third party tester application 

process, communicating the new approach to industry, and developing 

third party testing requirements. We believe that these are positive steps 

to finalize the strategy, but that TSA should also consider the potential 

consequences of implementing the strategy, such as the cost impacts to 

TSA and vendors, which will help to ensure the effectiveness of the 

strategy.

DHS also provided technical comments that we incorporated into the 

report as appropriate.

We are sending copies of this report to the Secretary of Homeland 

Security, the TSA Administrator, and the appropriate congressional 

committees. In addition, the report is available at no charge on the GAO 

website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact 

me at (202) 512-4841 or mackinm@gao.gov. Contact points for our 

Offices of Congressional Relations and Public Affairs may be found on 

the last page of this report. GAO staff who made significant contributions 

to this report are listed in appendix III.

Michele Mackin

Director

Acquisition and Sourcing Management

Agency Comments 

and Our Evaluation
Page 27 GAO-16-117 TSA Acquisitions

List of Committees

The Honorable John Thune

Chairman

The Honorable Bill Nelson

Ranking Member

Committee on Commerce, Science, and Transportation

United States Senate

The Honorable Michael McCaul

Chairman

The Honorable Bennie G. Thompson

Ranking Member

Committee on Homeland Security

House of Representatives
Appendix I: Scope and Methodology

Page 28 GAO-16-117 TSA Acquisitions

The Transportation Security Acquisition Reform Act included a provision 

for GAO to examine the Transportation Security Administration’s (TSA) 

test and evaluation process for security-related technologies. Specifically, 

this report examines the extent to which (1) TSA’s test and evaluation 

process helps TSA meet mission needs through the acquisition of 

passenger and baggage screening technologies, and (2) TSA’s planned 

actions to improve the test and evaluation process address factors 

contributing to inefficiencies, if any, in acquiring those technologies.

For our first objective, we reviewed Department of Homeland Security 

(DHS) and TSA acquisition and testing policies and guidance to 

determine the role of test and evaluation in TSA’s acquisition process and 

compared them to acquisition best practices. We also analyzed testing 

documentation provided to us by TSA for the 22 Passenger Screening 

Program (PSP) and Electronic Baggage Screening Program (EBSP) 

systems TSA has tested since June 2010, such as test and evaluation 

plans, test reports, system evaluation reports, data packages for entry 

into testing, and information regarding the number of retests during 

testing to determine the number of systems that passed each stage of 

TSA’s testing process. In addition, we reviewed TSA and DHS acquisition 

documentation for passenger and baggage screening technologies, 

including requirements documents, schedules, letters of assessment, 

acquisition decision memorandums, contract award notices, and 

information regarding the number of each technology deployed in airports 

nationwide to determine the number of systems that TSA has tested, 

qualified for use, and procured since June 2010. Additionally, we 

conducted interviews with TSA officials from the Office of Security 

Capabilities (OSC), the Office of Acquisitions, PSP, EBSP, and the 

Mission Analysis Division, regarding the acquisition and test and 

evaluation processes, the influence of test and evaluation results on 

TSA’s decisions about which technologies to procure, and reasons for 

any acquisition inefficiencies. We also met with DHS officials from the 

Science and Technology Directorate and the Office of Program 

Accountability and Risk Management regarding their roles and 

responsibilities related to TSA’s test and evaluation process. Further, we 

attended a joint industry and TSA workshop hosted by the Airport 

Consultants Council focused on TSA’s acquisition process for securityrelated technologies, including its test and evaluation process. Finally, we 

visited the two primary testing facilities for TSA’s security-related 

technologies—the TSA Systems Integration Facility, in Arlington, Virginia 

and DHS’s testing facility, the Transportation Security Laboratory, in 

Atlantic City, New Jersey to interview testing officials and observe the 

Appendix I: Scope and Methodology
Appendix I: Scope and Methodology

Page 29 GAO-16-117 TSA Acquisitions

testing process. We chose these locations because they are TSA’s two 

primary testing facilities for security-related technologies.

For our second objective, we reviewed TSA strategic planning 

documents, including TSA’s August 2015 Strategic Five-Year Technology 

Investment Plan for Aviation Security, OSC’s Strategic Plan for 2013 to 

2016, and OSC’s May 2014 Transportation Security Strategic Capability 

Investment Plan, to identify TSA’s stated challenges with the test and 

evaluation process and its plans for improvement. We also reviewed 

TSA’s interim third party testing guidance, implementation timeline, and 

briefings to industry as well as testing examples, which served as TSA’s 

rationale for implementing the third party testing strategy. In addition, we 

conducted interviews with TSA and DHS testing officials, including testers 

and evaluators, regarding challenges of TSA’s acquisition and test and 

evaluation processes as well as ongoing and planned efforts to improve 

testing efficiency and transparency. We also met with representatives 

from two industry groups with experience related to TSA’s test and 

evaluation process to obtain industry views on any challenges in the test 

and evaluation process for screening technologies, areas for potential 

improvement, and their perspectives on TSA’s third party testing strategy. 

We met with representatives from the Airport Consultants Council’s 

Security Manufacturers Coalition, who represented three of TSA’s largest 

vendors receiving 8 of PSP and EBSP’s 10 largest contracts to date. We 

also met with officials from the Homeland Security and Defense Business 

Council, who represented companies providing test and evaluation 

support services to DHS and TSA. Additionally, we conducted an 

interview with the DHS Director of Operational Test and Evaluation 

regarding planned efforts to expand DHS oversight of component-level 

testing, including TSA, from operational to developmental testing and its 

involvement in the development of TSA’s third party testing strategy. 

Further, we met with officials from the National Institute of Standards and 

Technology involved in the development of TSA’s third party testing 

strategy to determine TSA’s coordination with stakeholders in the 

development and implementation of the strategy. Finally, we conducted 

interviews with TSA testing officials regarding TSA’s existing and planned 

mechanisms for tracking systems throughout the test and evaluation 

process and any efforts to assess testing data, such as timeframes for 

completing testing and costs incurred and compared them to Standards 

for Internal Control in the Federal Government.

We conducted this performance audit from April 2015 to December 2015 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 
Appendix I: Scope and Methodology

Page 30 GAO-16-117 TSA Acquisitions

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.
Appendix II: Comments from the Department 

of Homeland Security

Page 31 GAO-16-117 TSA Acquisitions

Appendix II: Comments from the Department 

of Homeland Security
Appendix II: Comments from the Department 

of Homeland Security

Page 32 GAO-16-117 TSA Acquisitions
Appendix II: Comments from the Department 

of Homeland Security

Page 33 GAO-16-117 TSA Acquisitions
Appendix II: Comments from the Department 

of Homeland Security

Page 34 GAO-16-117 TSA Acquisitions
Appendix III: GAO Contact and Staff 

Acknowledgments

Page 35 GAO-16-117 TSA Acquisitions

Michele Mackin, (202) 512-4841 or mackinm@gao.gov

In addition to the contact named above, Katherine Trimble (Assistant 

Director), Charlie Shivers III (Analyst-in-Charge), Peter Anderson, Molly 

Callaghan, William Carrigg, Kristine Hassinger, Mark Hoover, Michael 

Kaeser, Jean McSween, Lindsay Taylor, and Ozzy Trevino made 

significant contributions to this report.

Appendix III: GAO Contact and Staff 

Acknowledgments

GAO Contact

Staff 

Acknowledgments

(100095)
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (http://www.gao.gov). Each weekday 

afternoon, GAO posts on its website newly released reports, testimony, 

and correspondence. To have GAO e-mail you a list of newly posted 

products, go to http://www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.

Subscribe to our RSS Feeds or E-mail Updates. 

Listen to our Podcasts and read The Watchblog.

Visit GAO on the web at www.gao.gov.

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Please Print on Recycled Paper.