Checked Baggage Screening: TSA Has Deployed Optimal Systems at the Majority of TSA-Regulated Airports, but Could Strengthen Cost Estimates
Released: 2012-04-27T00:00:00

Letter

Background

With the passage of ATSA in November 2001, TSA assumed from the Federal Aviation Administration (FAA) responsibility for securing the nation’s civil aviation system. In accordance with ATSA, TSA is responsible for the procurement, installation, and maintenance of explosive detection systems, including EDS and ETD, used to screen checked baggage for explosives (see figs. 1 and 2) at TSA-regulated airports. EDS machines identify suspicious items or anomalies in checked baggage that could indicate the presence of explosives or detonation devices. At airports with EDS, EDS machines are generally employed for primary screening of checked baggage while ETD machines are used for secondary screening to help resolve questions raised by EDS screening. At airports without EDS, ETD machines are used as the primary method for screening checked baggage.

TSA deploys EDS machines in stand-alone and in-line configurations. In a stand-alone configuration, checked baggage is manually loaded and unloaded by screeners (see fig. 1). In contrast, an in-line configuration integrates EDS machines with a baggage handling system—a conveyor system that transports and sorts baggage from the ticket counter through the baggage screening system. (See fig. 3, which shows an in-line configuration with three EDS machines.)

Generally, an in-line checked baggage inspection system employs three levels of screening (see fig. 4). EDS machines perform automated (Level 1) screening. If the EDS machine is unable to clear a bag, it sends an alarm to a screener who performs a secondary (Level 2) inspection known as On-Screen Resolution by reviewing an image of the contents of the bag via computer monitor. If the screener cannot resolve the alarm using on-screen resolution tools, the bag goes to the Checked Baggage Resolution Area (Level 3) where another screener will perform manual inspection of the bag assisted by an ETD machine.

TSA officials stated that deployment of an integrated, centralized in-line system of EDS machines can enhance security, increase screening efficiencies, and lower screening costs by, among other things, reducing the number of screeners needed to conduct baggage screening and reducing work-related injuries caused by lifting heavy bags. Installing an in-line system can require modification of an airport terminal, including removal of the existing system, installation of a new baggage handling system and EDS machines, and the use of an interim solution to screen checked baggage while the in-line system is built. TSA estimates that depending on the size and complexity of an in-line project, installing an in- line system can take one to four years at larger (category X and I) airports.

In 2005, we reported that although TSA made substantial progress in installing EDS machines, the agency had not conducted a systematic, prospective analysis to determine which airports could achieve long-term savings and improve efficiencies and security by installing in-line systems or, where in-line systems may not be economically justifiable, by making greater use of stand-alone EDS rather than relying on the labor-intensive and less efficient ETD screening processes. We recommended that TSA systematically evaluate baggage screening needs at airports, including identifying and prioritizing the airports where the benefits—such as cost savings of screening operations and improved security—of replacing stand-alone baggage screening machines with in-line systems are likely to exceed the costs of the systems. TSA concurred and in response released its Strategic Planning Framework in February 2006, which identified and prioritized airports based on an analysis of several factors, including security risk and the amount of estimated cost savings.

In March 2011, we reported that by continuing to replace or modify older baggage screening systems with more efficient solutions, including in-line screening systems, TSA could continue to eliminate baggage screener TSA agreed that the deployment of more efficient systems positions. offers potential personnel cost savings to the federal government.

GAO-11-318SP. airports, will be operated at the levels in established requirements and we recommended that TSA develop a reliable schedule for the EBSP. DHS concurred with these recommendations and has initiated actions to implement them.

In February 2012, we reported that we continue to believe that TSA might achieve savings in screening personnel costs by continuing to replace or modify older checked baggage screening systems with more efficient solutions, including in-line screening systems, to the extent possible. TSA reported that since the issuance of GAO’s 2011 report, it had replaced 60 stand-alone checked baggage screening machines with more efficient in-line screening systems.

Since fiscal year 2006, about $6.8 billion has been made available to TSA for activities related to checked baggage screening (see table 1), making the EBSP one of DHS’s largest acquisition programs.

Following the terrorist attacks of September 11, 2001, and the enactment of ATSA, airports relied upon various sources of funding to support security-related capital improvement projects. For example, as enacted, ATSA authorized the use of Airport Improvement Program (AIP) funds for projects to support the installation of explosive detection systems. Subsequently enacted statutes, such as the Vision 100—Century of Aviation Reauthorization Act (Vision 100), however, either limited or precluded the use of AIP to fund projects related to the installation of in- line systems. TSA, which is solely responsible for procuring and deploying equipment to screen checked baggage for explosives, also provides funding in support of related facility modifications. The Consolidated Appropriations Resolution, 2003, first authorized the use of LOIs by TSA for airport facility modification projects related to the installation of in-line baggage screening systems. Although not a binding commitment of federal funding, LOIs are agreements providing that TSA will reimburse airports or airlines for a specified percentage of an eligible project’s cost, subject to the availability of appropriations. This in turn enables an airport to proceed with a project because the airport and any investors are aware that the agreed-upon percentage of allowable costs will likely be reimbursed. The airport or airline is responsible for its share of the total funding needed to complete the project and generally must be capable of funding the project in its entirety.

From fiscal years 2003 through 2007, TSA entered into 8 LOI agreements covering 9 airports. Pursuant to the law then in effect, these LOI agreements provided for a 75 percent federal cost share of allowable project costs. Beginning in fiscal year 2008 and in accordance with Vision 100, any LOI entered into by TSA was to reflect a 90 percent federal cost share. Since 2008, TSA has entered into 4 more LOIs at a 90 percent federal cost share. As of December 13, 2011, TSA reported that its net cumulative obligations for all 12 LOIs were $1.46 billion. TSA also uses OTAs to support airport facility modification projects related to the installation of checked baggage screening equipment. TSA describes OTAs, which have become the primary administrative vehicles through which TSA financially supports such projects, generally as single-year reimbursable agreements (in contrast to the multiyear LOI agreements). According to TSA, OTAs take many forms and are generally not required to comply with federal laws and regulations that apply to contracts or cooperative agreements, such as the Federal Acquisitions Regulations, thus enabling the parties to negotiate provisions that are mutually agreeable. According to TSA, the federal cost share applied to OTAs may be negotiated, but since fiscal year 2008 TSA has generally followed the federal cost share applicable to LOIs. As of the end of fiscal year 2011, TSA had used at least 150 OTAs to support airport facility modifications related to the installation of in-line systems. According to TSA, these OTAs have reflected federal cost shares ranging from 50 to100 percent. As of December 13, 2011, TSA reported that its net cumulative obligations for all OTAs were $1.74 billion.

TSA Has Made Progress in Deploying Optimal Checked Baggage Screening

TSA Has Collaborated with Airports to Install Optimal Baggage Screening Systems at 76 Percent of TSA-Regulated Airports

TSA reports that 337 of 446 airports (76 percent) the agency regulates for security have optimal baggage screening systems. The remaining 109 of 446 airports (24 percent) do not yet have optimal baggage screening systems in all screening areas. To be considered an airport with an optimal baggage screening system, as TSA considers it and as we define it for purposes of this report, an airport must have completed installation and activation of the in-line or stand-alone systems that best fits the airport’s screening needs without relying on temporary stand-alone systems.may have a mix of explosive detection systems (EDS or ETD) and configurations (in-line or stand-alone systems), depending on the airport’s needs. TSA officials told us that they plan to deploy equipment for an additional 201 in-line systems in the future, which will include the Thus, an airport with an optimal baggage screening system purchase of an estimated 685 new EDS machines for installation at airports that are not screening with optimal configurations.

TSA aims to complete its efforts to deploy optimal screening systems by using EDS machines as the primary means for screening checked baggage at all category X, I, II, and III airports while continuing to use ETD machines as the primary means at category IV airports. Additionally, TSA plans to deploy EDS machines in in-line configurations at all category X and I airports and in-line or stand-alone configurations at category II and III airports. At each of the 10 airports we visited, we observed distinct checked baggage screening needs, based on an airport’s terminal configuration and the number of passenger boardings. We discussed an airport’s willingness and financial ability to pay for facility modifications required to install in-line systems. For each of these airports, officials and engineers provided documentation (for example, drawings and blueprints) on distinct facility modification projects to accommodate baggage screening system upgrades. According to airport officials, TSA and airport officials work together to determine the most appropriate baggage screening configuration based on an airport’s needs. Of the 337 airports where baggage screening systems are optimal and no longer using temporary solutions, 55 airports use EDS in-line systems exclusively for their primary checked baggage screening needs, while 92 airports use EDS stand-alone machines only, and 167 use ETD systems exclusively. The remaining 23 airports have a mix of systems.

See figure 5 for additional details on airport screening configurations. Also, see appendix II for more details on the status of efforts to optimize checked baggage screening systems.

Of the 337 airports with optimal baggage screening systems, larger airports were less likely to have completed optimal solutions than smaller airports. Specifically, 36 percent (10 of 28) of the category X airports and 49 percent (28 of 57) of the category I airports were considered to have optimal solutions, whereas 60 percent (46 of 77) of the category II airports, 76 percent (96 of 127) of the category III airports, and all of the 157 of the category IV airports were considered to be optimally configured, as shown in figure 6.

According to TSA and airport officials, this is because the larger airports generally need to install more complex in-line systems, which are more time and resource intensive to install and often require a significant amount of airport infrastructure modification and construction, while the smaller airports, particularly the category IV airports that rely on the smaller ETD machines, require far less time and resources to install these systems. Moreover, TSA officials stated that the in-line systems that best meet the screening needs of larger airports take longer to plan and build because of the added complexity and scale of the upgrades and the coordination required among multiple stakeholders.

TSA anticipates that in the next 5 years about 60 percent (1,153 of 1,933) of the EDS machines will reach the end of their useful life of about 10 years and will need to be replaced, as shown in figure 7. As a result, to ensure that 100 percent of checked bags continue to be screened as required by ATSA, TSA revised its focus from replacing stand-alone EDS in airport lobbies with in-line systems to replacing its aging fleet of EDS and ETD machines, a process it calls recapitalization. However, TSA reported that it will continue to collaborate with airports or airlines to install optimal in-line systems if it coincides with efforts to recapitalize aging EDS machines or if the existing in-line systems do not meet current TSA standards. In August 2011, TSA issued its EDS and ETD Recapitalization and Optimization Plan, which establishes the method and criteria for prioritizing when and how EBSP recapitalization and optimization will occur. The plan notes that many in-line recapitalization projects will include an optimization component. For example, a number of early in- line screening systems are likely to require optimization, among other things, to improve performance, increase efficiency, and reduce operating costs. At one airport we visited with an early in-line system, we observed a baggage handling system that needed to be replaced because it had sharp curves and steep grades that led to an excessive number of errors and jams. The airport was involved in TSA’s recapitalization pilot program, and airport officials anticipated TSA supporting the optimization of this system as part of recapitalization.

Reducing the Federal Cost Share for Eligible Airport Facility Modification Projects Could Reduce TSA’s Costs

Reducing the Federal Cost Share

Consistent with current law, TSA enters into reimbursable agreements through which it generally funds 90 percent of the cost of an eligible airport facility modification project to support the installation of an optimal system, with an airport or airline funding the remaining 10 percent of the project’s cost. If the federal cost share for airport facility modification projects is reduced, TSA may be able to use available funding to install a greater number of optimal solutions than currently anticipated. From fiscal year 2003 through fiscal year 2007, in accordance with law then in effect, TSA entered into LOIs at a 75 percent federal cost share. Looking forward, we used TSA’s projections of airport modification costs for each fiscal year, 2012 through 2030, as presented in its latest LCCE, and estimated the amount of expenditures for airport modifications that could be shifted from TSA to airports if the federal government’s cost share were reduced from the current 90 percent to 75 percent. As discussed later in this report, we found that TSA’s LCCE data are of questionable reliability for a precise estimate. However, the data can serve to provide a rough indication of how much TSA could save if the cost share were adjusted. Thus, we estimate that if the federal cost share for such projects returned to the 75 percent TSA applied to many of the reimbursable agreements it entered into prior to fiscal year 2008, rather than the current federal share of 90 percent, TSA’s anticipated expenditures for these modifications would be roughly $300 million less through fiscal year 2030.

TSA had previously determined that assigning costs among industry stakeholders and the nation as a whole is difficult because operational improvements to the baggage handling systems and national security benefits are difficult to quantify. This, in turn, makes it difficult to develop a cost share formula that would allow TSA to allocate costs in proportion to benefits. Consistent with the Intelligence Reform and Terrorism Prevention Act of 2004, TSA commissioned the 2006 Baggage Screening Investment Study Working Group to prepare a report for the Aviation Security Advisory Committee, which examined what an appropriate federal government/airport cost share should be for the installation of checked baggage screening equipment. The working group, which consisted of over 60 members representing, among others, TSA, FAA, airports, airlines, designers of baggage handling systems, and financial institutions, were unable to develop a consensus on an appropriate cost share formula, in large part because of the difficulties of measuring benefits, differing views on the federal responsibility for funding capital investments related to baggage screening, and the competing demands on the federal budget. As a result, potential cost share options were not submitted to the Congress as part of DHS’s fiscal year 2006 budget submission.

Airport Officials Acknowledge Benefits from In-line Systems but Oppose Increasing Their Cost Share

Representatives of all 10 airports we visited told us that they benefit from the installation of integrated, in-line baggage screening systems. Specifically, officials from 9 of the 10 airports cited the reduction of passenger congestion in airport terminals because stand-alone EDS machines were removed from the lobby or ticketing areas, officials from half of the airports noted that in-line systems reduce the number of lost or stolen bags by creating a streamlined process for moving checked baggage directly from where baggage is checked by the passenger and airline to the aircraft, and officials from 3 of 10 airports noted that in-line systems facilitate airport growth.

However, for various reasons, officials representing 8 of the 10 airports opposed a reduction in the federal cost share that would increase airports’ share of modifications costs.the airports stated the following four concerns: Specifically, officials from half of

Assuming a larger share of airport modification costs would pose hardships because of current fiscal or funding constraints.

Airports incur additional (that is, nonallowable) costs that are necessary to building an in-line system, but which TSA will not reimburse. Examples of the nonallowable costs the airports cited include the costs of designing an in-line system and constructing rooms in which screeners manually screen bags that have not previously been cleared. As a result, officials from 5 of the 10 airports we spoke with told us that after necessary, but nonallowable, costs were included, the airports were already paying for more than 10 percent of the modification costs associated with in-line systems.

Airports have a backlog of capital projects or would rather fund projects that will produce additional revenue, such as parking garages or larger areas for concessions, than projects that are related to TSA’s security responsibilities.

TSA will be the primary beneficiary of in-line baggage screening systems because the integration and consolidation of these systems will enable TSA to reduce the number of baggage screeners and provide TSA with other operational efficiencies.

TSA’s Cost Estimation Processes Do Not Fully Comply with Best Practices, and TSA and DHS Are Working Together to Establish an Acquisition Program Baseline

TSA Estimates That the EBSP Will Cost Close to $50 Billion through 2030

TSA’s August 25, 2011, life cycle cost estimate identified a total program cost for EBSP of $49.2 billion through fiscal year 2030. The $49.2 billion includes $2.65 billion for EBSP program operation and management; $11.03 billion for capital costs, including, among other things, recapitalization and facility modifications for optimization; $14.89 billion for operations and maintenance of equipment; $18.42 billion for screener salaries; and $2.22 billion for research and development and other miscellaneous related costs. Additionally, TSA officials reported that the program is expected to provide life cycle cost savings of $537 million. However, we found that the cost estimates are of questionable reliability for a precise estimate. TSA continues to revise its life cycle cost estimates. For example, its August 2011 EBSP life cycle cost estimate report stated that new requirements, including recapitalization and upgrading the efficiency of early in-line systems, will likely lead to a gap between anticipated program needs and anticipated funding during fiscal years 2012 to 2017, totaling up to $436 million. However, in December 2011 TSA officials told us the DHS Acquisition Review Board had requested that TSA revise the EBSP funding plans and projections to more accurately reflect current budget constraints and reduced funding available for the program. According to TSA officials, they plan to complete the revised EBSP planning estimates and funding projections to help eliminate the potential funding gap before the next Acquisition Review Board meeting in May 2012. EBSP senior program officials explained that TSA will address the potential funding gap by (1) controlling the costs associated with engineering initiatives and improvements in technology performance, (2) delaying funding of some new in-line systems and recapitalization projects, and (3) extending the useful life of equipment beyond 10 years in cases where replacement could be delayed.

TSA’s Life Cycle Cost Estimates Are Not Fully Consistent with Best Practices for Reliable and Credible Estimates

Although TSA’s methods for developing its LCCE reflect features of best practices, its methods do not fully adhere to these practices. As highlighted in our past work, a high-quality, reliable cost estimation process provides a sound basis for making accurate and well-informed decisions about resource investments, budgets, assessments of progress, and accountability for results and thus is critical to the success of a program. According to the Office of Management and Budget (OMB), federal agencies must maintain current and well-documented estimates of program costs, and these estimates must encompass the program’s full life cycle.increased risk of making poorly informed investment decisions, securing Without such an estimate, agencies are at insufficient resources to effectively execute defined program plans and schedules, and experiencing program cost and schedule overruns and performance shortfalls. As highlighted in our Cost Estimating and Assessment Guide, a reliable cost estimate has four characteristics—it is comprehensive, well documented, accurate, and credible. We reviewed TSA’s cost estimation procedures for the EBSP and assessed the extent to which the agency met the four characteristics, as shown in table 2.

Our assessment showed that TSA’s EBSP estimates partially met three characteristics and minimally met one characteristic of a reliable cost estimate. Specifically, TSA’s cost estimate was as follows:

Partially comprehensive because the estimate defines the program, reflects the current schedule, is technically reasonable, includes assumptions identified by a team of personnel and engineers, and provides risks related to detection standards. However, the cost estimate is not considered fully comprehensive because it does not incorporate costs associated with all security threats, lacks a detailed product-oriented work breakdown structure that covers the entire scope of work, and lacks a single technical baseline. Without fully accounting for life cycle costs, management may have difficulty successfully planning program resource requirements and making wise decisions. Further, the program lacks a defined end date. A reasonable criterion is that the estimate capture at least 10 years of costs beyond the planned full operational capability date—the date at which optimal systems are fully deployed and operating at all locations. However, we cannot determine whether the time frame is sufficient because we have not received documentation to support the program’s official, planned full operational capability date. According to TSA, the EBSP does not have a defined end date for procurement because maintaining compliance with the 100 percent screening mandate established by ATSA requires TSA to continuously procure and replace equipment as it reaches the end of its useful life. TSA also believes that it is following DHS acquisition guidance outlined under the acquisition decision memorandum dated January 13, 2005, for estimating threshold dates. Nevertheless, the EBSP still lacks a defined, official full operational capability date, without which we can neither determine whether the time frame used in the LCCE is sufficient nor verify that the life cycle cost estimate is fully comprehensive.

Partially documented because TSA used relevant data to help develop the estimate. For example, TSA’s estimated price for the equipment is based on existing contracts for EBSP equipment purchases, maintenance costs, LOI agreements, and OTAs. TSA also provided narratives, briefings, and documents to describe the program requirements, purpose, technical characteristics, and acquisition strategy, and explained how calculations were performed. However, TSA did not adequately document many assumptions or methodologies underlying its cost model to the extent that would allow someone unfamiliar with the cost estimate, using only the available documentation, to easily re-create the estimate. For example, equipment purchase and hardware upgrade costs were based in part on estimates from engineers and contract specialists rather than historical or analogous data. Unless ground rules and assumptions are clearly documented, the cost estimate will not have a basis for areas of potential risk to be resolved. In addition, TSA also provided little or no evidence that the assumptions and methodologies underlying the cost estimate were approved by management.officials agreed that additional documentation could improve the outside reviewers’ ability to evaluate the estimate. According to TSA, the EBSP plans to follow DHS guidance to implement software dedicated to the estimation, documentation, and reporting of costs during all phases of the EBSP program life cycle to help address documentation concerns.

Partially accurate because while the estimate is properly adjusted for inflation, differences between planned and actual costs are not fully documented, explained, or reviewed. In addition, we cannot determine whether the cost estimate is unbiased—that is, neither overly conservative nor overly optimistic—because the program did not perform an uncertainty analysis that meets best practices. While TSA agreed that costs should be documented, it could not explain why the differences between planned and actual costs were not being fully documented, explained, or reviewed.

Minimally credible because while TSA identified changes in cost for each scenario against the baseline and developed a limited risk analysis, TSA did not complete other relevant activities to ensure that the estimate accounts for bias and uncertainty. For example, the agency did not (1) document in detail the assumptions and parameters associated with its sensitivity analysis, such as detailed calculations on how each parameter was varied between its minimum and maximum values; (2) conduct a fully objective uncertainty analysis that derives the point estimate percentile rather than assumes it; (3) cross-check major cost elements to see whether results are similar; and (4) use an independent cost estimate to validate the cost estimate. Given the important role that an independent cost estimate provides in developing an objective and unbiased assessment of whether the program estimate can be achieved, developing and using an independent cost estimate would provide decision makers with insight into a program’s potential costs and reduce the risk of underfunding a program. TSA officials told us they did not perform a complete uncertainty analysis because it was too costly and time consuming. TSA officials concurred that an independent cost estimate was not done for the EBSP and agreed that completing an independent estimate would be helpful.

Regarding the credibility of TSA’s estimates, our past work has shown that program cost estimates that are independently validated help improve the confidence that the estimate is credible, are needed for making timely and informed budget decisions, and help reduce the likelihood of unanticipated program cost growth. Our prior reviews of several DHS programs, including the EBSP, have also shown that if cost estimates are not validated in accordance with DHS acquisition management directive requirements at the start of an acquisition program, it is difficult to assess whether a program is being deployed within planned budgets. Further, DHS’s Acquisition Management Directive requires major program cost estimates to be validated early in the decision-making process, before programs can receive authorization for acquisition contracts at the DHS Acquisition Review Board meeting. However, we found that since 2008 the DHS Acquisition Review Board allowed the EBSP to proceed with acquisition contracts before the LCCE was independently validated by DHS, which is inconsistent with DHS policy.

In May 2010, the DHS Cost Analysis Division reviewed the EBSP life cycle cost estimate and found that it needed more comprehensive data and that its accuracy could not be determined. As of December 2011, the estimate had not been independently validated by DHS. According to TSA, DHS guidance that required validation of cost estimates was not in place until November 2008. The interim DHS directive for LCCEs and validation was established in November 2008, and the Cost Analysis Division was designated the authority responsible for independent cost estimates on January 24, 2010. However, the DHS Acquisition Review Board did not request a validated LCCE until February 25, 2011. TSA officials also stated that since the program’s budget circumstances changed over time, the cost estimate needed to be revised to reflect a constrained budget for fiscal year 2013 and other program changes. For example, TSA officials stated that new requirements for the EBSP, particularly the shift in focus to recapitalization of the aging EDS and ETD fleet and upgrades to in-line baggage screening systems and threat detection levels, contributed to continued program cost growth and delayed efforts to validate the latest LCCEs. As a result, the federal government’s portion of the cost estimates for the EBSP has increased from approximately $20.5 billion in fiscal year 2010 to $25.4 billion in fiscal year 2011—a 24 percent increase.DHS on the validation of the cost estimate for the next Acquisition Review Board meeting scheduled for May 2012.

TSA is currently working with According to our 2009 Cost Estimating and Assessment Guide, endorsed by the OMB and DHS, cost estimates are integral to determining and communicating a realistic view of likely cost outcomes that can be used to plan the work necessary to develop, produce, and support a program. Taking steps to ensure that its cost estimates for the EBSP conform to cost estimating best practices will help provide TSA with a sound basis for understanding how the program can be sustained in future years.

TSA and DHS Are Working to Develop an EBSP Acquisition Program Baseline

Another foundation for making informed budget decisions is the acquisition program baseline, which is to document a program’s critical cost elements, including acquisition costs and life cycle costs. According to DHS’s acquisition guidance, the program baseline is the contract between the program and departmental oversight officials and must be established at program start to document the program’s expected cost, deployment schedule, and technical performance. Establishing such a baseline at program start is important for defining the program’s scope, assessing whether all life cycle costs are properly calculated, and measuring how well the program is meeting its goals. As we have previously reported, establishing realistic original baseline estimates is important for minimizing the risks of poorly defined requirements and achieving better program outcomes. By tracking and measuring actual program performance against this formal baseline, management can be alerted to potential problems, such as cost growth or changing requirements, and has the ability to take early corrective action.

However, since the inception of the program more than eight years ago, the EBSP has not had a DHS approved acquisition program baseline and DHS did not require TSA to complete an acquisition program baseline until November 2008. An approved baseline will provide DHS with additional assurances that TSA’s approach is appropriate and that the capabilities being pursued are worth the expected costs. DHS officials told us that several reorganizations of DHS offices responsible for approving the baseline and a lack of functional expertise within the agency contributed to further delays in approving the EBSP acquisition program baseline. According to TSA officials, they have twice submitted an acquisition program baseline to DHS for approval. In November 2009 and February 2011, TSA requested approval of a program baseline, but according to DHS officials TSA did not have a fully developed life cycle cost estimate. In November 2011, DHS told TSA that it needed to revise the life cycle cost estimates as well as its procurement and deployment schedules to reflect budget constraints. DHS officials told us that they could not approve the acquisition program baseline as written because TSA’s estimates were significantly over budget. TSA officials stated that TSA is currently working with DHS to amend the draft program baseline for approval. TSA officials stated that they plan to resubmit the revised acquisition program baseline before the next Acquisition Review Board meeting in May 2012. Establishing and approving a program baseline, as DHS and TSA currently plan to do for the EBSP, could help DHS assess the program’s progress in meeting its goals and achieve better program outcomes.

Conclusions

TSA’s EBSP is aimed at increasing airport screening efficiencies and addressing the continuing threat of explosives concealed in checked baggage, at a total estimated cost to the federal government and the private sector of close to $50 billion through fiscal year 2030. Given the size of the federal investment, it is vital that TSA ensures effective stewardship over these resources and conveys useful information to the Congress about the scope and cost of the program. However, the limitations we identified in TSA’s EBSP cost estimates raise questions about their reliability. Taking steps to ensure that its cost estimates meet the four characteristics for high-quality and reliable cost estimates would provide TSA with increased assurance about the reliability of the estimated total cost of the program and better position it to account for all resources and associated costs required to develop, implement, and sustain the EBSP.

Recommendation for Executive Action

In order to strengthen the credibility, comprehensiveness, and reliability of TSA’s cost estimates and related savings estimates for the EBSP, we recommend that the Administrator of TSA ensure that its life cycle cost estimates conform to cost estimating best practices.

Agency Comments and Our Evaluation

On March 30, 2012, we provided a draft of this report to DHS for its review and comment. DHS provided written comments on April 19, 2012, which are reprinted in appendix IV. In its written comments, DHS concurred with our recommendation that TSA ensure that its life cycle cost estimates conform to cost estimating best practices and discussed efforts under way to address it. DHS further acknowledged the importance of producing life cycle cost estimates that are comprehensive, well documented, accurate, and credible so that they can be used to support DHS funding and budget decisions. DHS also noted that after conducting an internal review, TSA is implementing a management directive that applies DHS guidance and the best practices from the GAO Cost Estimating and Assessment Guide. As part of this effort, TSA is (1) establishing a working group and executive board to review program cost estimates to validate whether the estimates are credible and affordable, (2) requiring all life cycle cost estimates to be approved by DHS to ensure consistency and quality across TSA programs, (3) purchasing and training its employees on specialized cost estimating software, and (4) initiating hiring actions to hire additional cost estimating personnel. TSA believes that this will institutionalize cost estimating best practices within the organization and ultimately allow TSA and the Department to make better-informed investment decisions. These are positive steps; however, additional time will be needed to assess whether they have been fully and consistently implemented in accordance with GAO best practices.

We are sending copies of this report to the Secretary of Homeland Security, the Assistant Secretary of the Transportation Security Administration, and appropriate congressional committees. In addition, this report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-8777 or lords@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

We examined the Department of Homeland Security’s (DHS) Transportation Security Administration’s (TSA) operation of the Electronic Baggage Screening Program (EBSP) to assess the program’s current status, alternative cost sharing options, and cost estimates. Specifically, we addressed the following questions:

What is the status of TSA’s efforts to install optimal checked baggage screening systems in collaboration with airports?

How would reducing the current federal cost share for eligible airport modification projects from 90 percent to its previous level of 75 percent affect the amount that TSA pays for these modifications, and what benefits, if any, do airports report receiving from in-line baggage screening systems?

To what extent are TSA’s cost estimation procedures consistent with best practices and is TSA’s acquisition baseline consistent with DHS guidance?

To determine the status of TSA’s efforts to install optimal checked baggage screening systems, we obtained data as of December 2011 and January 2012 from TSA, such as the current number of airports with at least one in-line system, and the number of airports with optimal systems. We also obtained data on the number of airports configured exclusively with in-line screening, the number of airports configured with a mix in-line and stand-alone explosives detection systems (EDS), and the number of airports using only stand-alone EDS for the same time period. We also collected data on the overall number of operational in-line systems and EDS and explosives trace detection (ETD) machines as of December 2011. We reviewed documentation from TSA’s EBSP, including the EBSP strategic plans for fiscal years 2006, 2008, and 2009, and the 2011 EDS and ETD Recapitalization and Optimization Plan.

We assessed the reliability of the various data TSA provided about airports, including the number of TSA-regulated airports (by category) and the numbers of airports with the different configurations of baggage screening systems (in-line or stand-alone), and the investment and budget expenditure dollar values in the letters of intent and other transaction agreements by questioning cognizant TSA officials and obtaining extensive documentation about these various data. We found these data to be sufficiently reliable for the purposes of our report.

To determine how reducing the federal cost share from the current 90 percent to the previous federal cost share of 75 percent for eligible airport modification projects may affect the amount that TSA pays for these modifications, we calculated estimates based on TSA’s August 2011 projections of how much airport modifications will cost in the future. These projections represent TSA’s best estimate for how much it will spend on airport modifications for in-line systems each year from fiscal years 2012 through 2030. We also reviewed the reliability of the cost estimate by evaluating how well TSA followed best practices detailed in the GAO Cost Estimating and Assessment Guide (see below). While TSA’s process for estimating costs only partially meets the characteristics of a reliable cost estimate, the data can serve to provide a rough indication of how much could be saved by reducing the federal cost share for optimization. To assess how the installation of in-line baggage handling systems may benefit the airports that receive them, we visited a nonrandom sample of 10 airports. We chose these airports based on the size of airport, type of checked baggage screening systems installed, and status of airport facility modification completion. We discussed cost share with officials at each airport representing the airport authority, tenant airlines, and TSA’s Federal Security Director. In addition, we interviewed officials from the largest industry associations that represent airport executives, airports and airlines (the American Association of Airport Executives, the Airports Council International North America, and the Air Transport Association). We also interviewed an official from of the Association for Airline Passenger Rights. In addition, we discussed potential benefits of optimization with aviation security experts. These results cannot be generalized to the entire industry, but did provide broader perspectives on the issues and costs associated with the EBSP.

To assess the extent to which TSA’s methods for estimating costs for EBSP are consistent with best practices and its acquisition program baseline is consistent with DHS guidance, we analyzed TSA’s most recent life cycle cost estimate and recapitalization report finalized in August 2011. Specifically, we used best practices in the GAO Cost Estimating and Assessment Guide to evaluate TSA’s estimating methodologies, assumptions, and results to assess whether the official cost estimates were comprehensive (i.e., includes all costs), accurate, well documented, and credible. Our Cost Estimating and Assessment Guide considers an estimate to be comprehensive if its level of detail ensures that all pertinent costs are included and no costs are double counted; accurate if it is not overly conservative, is based on an assessment of the most likely costs, and is adjusted properly for inflation; well documented if the estimate can be easily repeated or updated and can be traced to original sources through auditing; and credible if the estimate has been cross-checked with an independent cost estimate and a level of uncertainty associated with the estimate has been identified.

We also interviewed the TSA EBSP office’s cost estimating team and its consultants to obtain a detailed understanding of their methodology, the cost model, and data. In doing so, we interviewed cognizant program officials, including the Program Manager and cost analysis team, regarding their respective roles, responsibilities, and actions in developing the cost estimate, reviewing it, or both.

We examined data reliability of the cost estimate by doing the following:

Obtaining cost estimates and reviewing how each major element was calculated with an emphasis on the basis for the estimate and strength and quality of the supporting documentation.

Verifying that the parameters used to create each estimate were valid and applicable by comparing to available cost estimating references, posing questions to the cost estimators for clarification, and relying on other technical sources for cross-checking.

Verifying that calculations were correct for each major element.

Verifying that escalation was properly applied and elements rolled up accurately to the overall program cost estimate.

We reviewed TSA’s EBSP cost estimates to determine whether the characteristic was (1) not met if the agency provided no evidence that satisfied any portion of the criterion, (2) minimally met if the agency provided evidence that satisfied less than one-half of the criterion, (3) partially met if the agency provided evidence that satisfied about one- half of the criterion, (4) substantially met if the agency provided evidence that satisfied more than one-half of the criterion, and (5) met if the agency provided complete evidence that satisfied the entire criterion. One analyst assigned a value ranging from 1 to 5 indicating the extent to which the agencies met each best practice and averaged the values for the practices that were associated with each characteristic. A second analyst independently verified the results. We also interviewed program officials from TSA and DHS responsible for each cost estimate about the estimate’s derivation. In doing so, we independently assessed the cost estimates for the current EBSP, as provided to us in August 2011, against our best practices.

To understand how TSA is working to make better informed budget decisions and complying with DHS guidance to develop an acquisition program baseline, we reviewed DHS guidance on acquisitions and documents related to TSA’s efforts to coordinate with DHS on developing an acquisition program baseline for EBSP, which DHS considers the contract between the program and departmental oversight officials to document the program’s expected cost, deployment schedule, and technical performance. We also reviewed EBSP Acquisition Review Board decisions and relevant acquisition decision memos during the period 2005 through 2011. Additionally, we interviewed TSA and DHS officials, including officials in the TSA Chief Financial Officer’s office and DHS’s Program Accountability and Risk Management Office to identify what procedures have been put in place to approve the acquisition program baseline.

To gain a better understanding of issues across all of our objectives, including the development of optimal systems, status of implementation, funding challenges, impact of a change in the cost share formula at the airport level, and cost estimation, we conducted site visits to California, New York, Massachusetts, Washington, D.C., and Florida to interview local airport officials, regional TSA officials, and airline representatives. To get a range of airports for our site visits, we made our selections based on the size of airport, type of checked baggage screening systems installed, and status of airport facility modification completion. We also considered recommendations from TSA and industry association officials about which airports to visit. Because we selected a nonprobability sample of airports, the information we obtained from these interviews and visits cannot be generalized to all airports. However, we believe that observations obtained from these visits provided us with a greater understanding of the airport officials’ perspectives. On these site visits, we interviewed airport, airline, and TSA officials responsible for financing, operating, and installing the checked baggage systems within their respective airports.

We conducted this performance audit from October 2010 through April 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Status of Implementation Efforts for TSA’s EBSP

TSA, through its EBSP, has deployed EDS and ETD machines in a variety of in-line and stand-alone configurations at airports to streamline airports and TSA operations, reduce screening costs, and enhance security. The following three tables provide information on the status of checked baggage screening systems. Table 3 highlights the different system configurations for airports that have optimal checked baggage systems by airport category. Table 4 shows the number of TSA-regulated airports with at least one in-line system. Table 5 provides the numbers of in-line systems and stand-alone EDS and ETD machines at TSA- regulated airports.

Appendix III: Our Evaluation of EBSP’s Cost Estimate Process

In determining that TSA’s processes for developing EBSP cost estimates do not fully comply with best practices, we evaluated TSA’s cost estimation methods against our 2009 Cost Estimating and Assessment Guide. (See table 6.) We applied the following scale across the four categories of best practices:

Not met: TSA provided no evidence that satisfies any portion of the criterion.

Minimally met: TSA provided evidence that satisfies less than one-half of the criterion.

Partially met: TSA provided evidence that satisfies about one-half of the criterion.

Substantially met: TSA provided evidence that satisfies more than one-half of the criterion.

Met: TSA provided complete evidence that satisfies the entire criterion.

Appendix IV: Comments from the Department of Homeland Security

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Glenn Davis, Assistant Director, and Daniel Rodriguez, Analyst-in-Charge, managed this assignment. Wendy Dye, Daren Sweeney, and Yee Wong made major contributions to the planning and all other aspects of the work. David Alexander and Richard Hung assisted with design, methodology, and data reliability. Chuck Bausell and Jack Wang assisted with economic analysis. Jason Lee, Stacey Steele, and Karen Richey assisted with life cycle cost estimate analysis. Nathan Tranquilli assisted with acquisition and contracting issues. Linda Miller provided assistance in report preparation. Thomas Lombardi provided legal support.