Aviation Security: Cost Estimates Related to TSA Funding of Checked Baggage Screening Systems at Los Angeles and Ontario Airports
Released: 2007-03-30T00:00:00

Letter

Summary of Findings

A key reason for the difference between the 2003 total project cost estimate and the revised 2005 estimate to install in-line baggage screening systems at LAX and ONT was that the 2003 estimate was developed at an early stage in the design process and was therefore based on preliminary data and assumptions that were subject to change. Consequently, the estimate did not adequately foresee some of the costs of retrofitting new systems into existing buildings or allow for sufficient space for the EDS machines, baggage inspection rooms, and conveyor belts. LAWA officials stated that they were under a tight timeframe to apply for the LOI because TSA had told them that federal funding was limited and that 17 other airports were competing for the funding. The 2003 total project cost estimate used concepts and construction estimates developed in about 12 weeks by Boeing, TSA’s contractor. LAWA relied on designs and estimates developed by Boeing and its subcontractors to determine the total project cost estimate because the company had expertise in integrating EDS equipment into airports. According to TSA and LAWA officials, both TSA and Los Angeles signed the LOI/MOA knowing the preliminary nature of the cost estimate.

According to construction industry guidance, an estimate’s accuracy depends on the quality of information known about the project at the time the estimate is prepared. The 2003 estimate was made at the “concept development” stage where the final project cost can be expected to range from 50 percent under to 100 percent over the estimated cost, according to this guidance. The 2005 revised estimate was made at the “design development” stage where the range of the final project cost estimate can be expected to be more accurate—from 20 percent under to 30 percent over the estimated cost.

In December 2003, LAWA presented TSA with a summary of inadequacies it had found in the original Boeing concept and the associated potential cost and scheduling impacts. LAWA then began an engineering study to update the in-line system concepts at LAX and ONT, the results of which it presented to TSA in September 2004. TSA reviewed these updated concepts and determined that they would meet its performance requirements; however, TSA’s review did not address cost issues. LAWA used these updated concepts to develop its 2005 estimate, which was based on more definitive information about terminal design requirements than the 2003 estimate. According to LAWA, new construction and excavation included in the 2005 designs increased the estimated costs. Among the design changes, LAWA determined that the placement of EDS machines in the 2003 concepts was infeasible in five of nine of the LAX terminals and both ONT terminals. In addition, the 2005 estimate included 20 additional baggage inspection rooms, 9 rooms for on-screen resolution of EDS alarms, and 10 computer rooms at LAX and ONT terminals. The 2005 estimate also included over $11 million in computer networking costs and costs associated with on-screen resolution of EDS alarms, which the 2003 estimate did not foresee. TSA also highlighted two additional factors that caused differences between the two estimates—cost increases due to the delay in beginning construction of the project and the escalation of construction costs between 2003 and 2005. LAWA also determined that TSA’s contractor and subcontractor made a mathematical error in the 2003 concept development estimate: construction costs were only included for one of the two baggage screening facilities and neither of the connected tunnels at ONT. TSA officials told us in January 2007 they were not able to substantiate this error.

Further, according to LAWA, system redesigns were required because TSA’s guidance on in-line baggage screening systems changed between the 2003 and 2005 estimates, leading to higher estimates. Because few in-line systems were in use at the time of the September 2003 LOI/MOA, only limited information on the capabilities of the in-line EDS machines, including actual bags screened per hour and false alarm rates, was available for modeling the systems. In June 2006, TSA produced the Recommended Security Guidelines for Airport Planning, Design and Construction to guide future construction of in-line checked baggage screening systems based on its past experiences. TSA also expects to release more detailed guidelines for in-line system planning and design in a few months.

The LOI/MOA affords TSA flexibility to amend the agreement to account for changed circumstances. However, under the terms agreed to in the LOI/MOA, TSA has no obligation to amend the LOI/MOA or to reimburse the City of Los Angeles for any additional costs beyond those agreed to in the LOI/MOA, and TSA officials have stated that the agency does not have plans for such reimbursement.

Scope and Methodology

To review key factors that contributed to the differences between the 2003 and 2005 cost estimates, we reviewed TSA and LAWA documents used in developing the cost estimates, including design plans, reports, briefings, and emails. We interviewed officials from TSA and LAWA, as well as TSA contractors and other relevant officials who participated in the cost- estimation process to learn about the factors that contributed to the increased estimate of the cost of in-line checked baggage screening systems at LAX and ONT. We visited LAX and ONT to obtain a first-hand perspective of the modifications needed to install the in-line EDS systems at both airports. Additionally, we examined industry guidance on estimating costs for construction projects. We did not independently verify the 2003 or 2005 cost estimates.

We conducted our work in accordance with generally accepted government auditing standards from October 2006 through March 2007.

Agency Comments and Our Evaluation

We provided a draft of this report to LAWA and the Department of Homeland Security (DHS) for review and comment. LAWA provided written comments which we have included in their entirety in appendix III. DHS provided no written comments. TSA provided e-mail comments. In addition, LAWA and TSA provided technical comments concerning facts in the report which we incorporated as appropriate.

In its March 8, 2007, comments, LAWA wrote that it believes the draft did not paint an accurate or complete picture of the facts. In general, LAWA raised three points: (1) TSA has the authority to revise the LOI to reflect accurate cost figures and explicitly anticipated doing so during the LOI/MOA development process, (2) the report fails to assign specific responsibility for initial designs and any errors, as directed by the Report of the Senate Appropriations Committee, and (3) the report does not recognize that LAWA responded to TSA urgency in completing the agreements and, as a result, used preliminary design and cost estimates as the basis for entering into the LOI/MOA.

We do not agree with LAWA’s comments. With respect to its first point, it is true that the LOI/MOA agreements afford TSA flexibility to amend the agreements to account for changed circumstances. As stated in our report, however, under the terms of the LOI/MOA and in accordance with the law, TSA is under no obligation to amend the LOI/MOA or to reimburse LAWA for any costs beyond those agreed upon in the LOI/MOA. To date, as noted in our report, TSA has not indicated any intent to amend the LOI/MOA agreements to provide LAWA with additional funding for this project.

LAWA states that it did not believe it would be held financially responsible for increases in eligible and allowable costs due to reasons beyond its control. When subsequent estimates revealed that the project costs would exceed the LOI/MOA-estimated amount, LAWA requested an amendment to the LOI to receive a 75 percent federal reimbursement of the $485 million revised estimate.

LAWA also commented that a senior TSA official provided written assurances that the agency “would have the opportunity to cover an increase in costs due to design changes” and referenced an April 2004 e-mail from a TSA official to LAWA in support of this assertion. LAWA noted that it relied on this and other assurances from TSA, “reinforced in various discussions,” at the time it concluded the LOI/MOA process.

While the April 2004 e-mail cited above did note that LAWA would not “be held to estimates that do not prove to be right on the mark,” this particular statement was made at least seven months after TSA and LAWA had concluded the LOI/MOA process and entered into the agreements. Furthermore, the MOA clearly provides that the agreement signed by both parties constitutes the “complete integration of all understandings between the parties.” More generally, it provides that any prior, contemporaneous, or subsequent changes, whether written or oral, have no force or effect, and that any changes or modifications to the MOA must be in writing, signed by the TSA Contracting Officer, and duly executed by the City of Los Angeles to have such force or effect. Neither LAWA nor TSA presented any documentation suggesting that steps prescribed in the MOA had been (or were anticipated to be) taken to amend the LOI/MOA with respect to the reimbursable amount.

LAWA also stated that the report does not reflect the extensive and protracted discussions LAWA had with TSA, leaving the impression that LAWA simply presented a new set of design concepts to TSA in September 2004. Our objective in this report, as agreed with congressional offices, was to identify the key factors that contributed to the differences between the 2003 and the 2005 cost estimates. As such, we reported that LAWA reported the findings of its engineering study in September 2004 and that TSA approved the concepts. We believe this statement sufficiently demonstrates the agreement between TSA and LAWA on the revised designs.

In its second point, LAWA suggests that the GAO report avoids assigning responsibility to TSA or its contractors and, as a result, “failed to answer the Senate Committee’s direction to provide a detailed explanation of the reasons for any differences the original estimate, including identification of and the party responsible for any material mistakes, omissions, and infeasible design concepts in the original estimate.”

The objective of the report, as agreed with the appropriate congressional offices in accordance with our Congressional Protocols, was to identify factors contributing to differences between the estimates. To the extent appropriate, we identified the roles and responsibilities of the various parties. Specifically, we noted that Boeing produced the conceptual designs that served as the basis for the 2003 estimate agreed to in the LOI/MOA. We noted that these designs had been developed at an early stage of the design process, which assumes costs that can differ greatly from final project costs. The report also states that LAWA had determined that Boeing had made a mathematical error in the Ontario estimate. Further, we determined that both TSA and LAWA had signed the LOI/MOA knowing of the preliminary nature of the cost estimate. Finally, as referenced in our scope and methodology and as agreed with the congressional offices, in identifying factors associated with the estimates, we did not independently verify the 2003 or 2005 cost estimates.

In its third point, LAWA suggests the report does not recognize that LAWA responded to TSA urgency in completing the agreements and, as a result, used preliminary design and cost estimates as the basis for entering into the LOI/MOA.

The report notes that “LAWA officials stated that they were under a tight timeframe because TSA had told them that federal funding was limited and 17 other airports were competing for the funding.” We identified this as a factor associated with the preliminary nature of the 2003 estimate. LAWA also commented that it was essentially required to accept the Boeing design and cost estimates. The report states that TSA and LAWA used the Boeing estimate to provide the basis for the estimate agreed to in the September 2003 LOI/MOA. Neither TSA nor LAWA provided evidence suggesting that TSA had required LAWA to accept the Boeing design and cost estimates.

In its e-mail comments, TSA stated that concept development for in-line solutions at all of the airports with LOIs was a collaborative effort between TSA and the respective airport entity. TSA further stated that most of the LOI estimates were developed early in the concept development phase. We incorporated a comment into the report to acknowledge that TSA viewed the development of the concepts as a partnership; however, we did not review concept development at other airports.

In its comments, TSA also states that there were only two changes in its guidance between 2003 and 2005. The first was the addition of the use of On-Screen Alarm Resolution Protocol during the alarm resolution process, which TSA acknowledged required redesign and associated cost increases. The second was the increase in the baggage throughput number per EDS which led to the deletion of 13 EDS machines from the quantity estimated in the 2003 concepts. According to TSA, this would have a significant impact on lowering the overall project cost, which would be supported by decreasing required space, baggage handling system infrastructure (generally up to $4 million per machine on average) and associated electrical, mechanical, data and other infrastructure.

In its comments, TSA described two additional factors that caused differences between the two estimates— cost increases due to the delay in beginning construction of the project and the escalation of construction costs between 2003 and 2005. We incorporated this comment into the report.

Finally, TSA stated in its comments that it had reviewed the mathematical error LAWA determined Boeing made in the 2003 concept for ONT, and had not been able to validate that the error had been made. In our report we acknowledge that TSA was not able to substantiate the error.

We will send copies of this report to the Secretary of Homeland Security and the Assistant Secretary, Transportation Security Administration, and interested congressional committees. We will send a copy of the report to LAWA and will also make copies available to others upon request. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov. If you have any questions or need additional information, please contact me at (202) 512- 2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are acknowledged in appendix I.

Appendix I: GAO Contact and Staff Acknowledgments

Acknowledgments

In addition to the contact named above, Charles Bausell, Kevin Copping, Kimberly Cutright, Glenn Davis, Terrell Dorn, Maria Edelstein, Richard Hung, Julian King, Brian Lipman, Thomas Lombardi, Amanda Miller, and Linda Miller made key contributions to this report.

Appendix II: Briefing Slides

Review of Cost Estimates for Installing

Introduction

Objectives

Senate Report 109-273 directs GAO to review the

differences between the original 2003 cost estimate and LAWA’s revised 2005 cost estimate.

As agreed with committee offices, GAO identified the key factors that contributed to the differences in the two estimates.

To address the objective, we reviewed key TSA and LAWA documents used in developing the cost estimates, including design plans, reports, briefings, and emails; interviewed TSA, LAWA, and other officials who participated in the cost- estimation process; visited LAX and ONT to obtain a first hand perspective of the modifications needed to install the in-line EDS Systems at both airports; and reviewed industry guidance on estimating costs for construction projects.

We did not independently verify the 2003 or 2005 cost estimates for installing the in-line EDS systems or the reasonableness/adequacy of the designs.

We conducted our work in accordance with generally accepted government auditing standards from October 2006 through March 2007.

Background

In-line EDS systems are advantageous because they can: Increase the efficiency of airport, airline, and TSA operations.

Lower costs by reducing the number of transportation security officers required to screen checked baggage. out of airport lobbies where it causes overcrowding, creating a potential target for terrorists.

In-line EDS systems are integrated with airports’ baggage- handling conveyor systems.

Background (cont.)

TSA and LAWA initiated discussions about in-line solutions for LAX and ONT in January 2003 in order to replace stand-alone EDS machines in the airports’ lobbies.

LAWA used concepts and construction estimates developed by TSA’s contractor, Boeing, for the LAX and ONT in-line baggage screening systems because of Boeing’s and its subcontractors’ expertise in integrating explosive detection systems into airports.

TSA reported that concepts for LAX and ONT were developed through a partnership between LAWA and the agency.

Boeing’s 2003 cost estimate consisted of $260 million for LAX and $24 million for ONT. LAWA and TSA agreed to add another $57 million (or 20 percent) for estimated administrative expenses for a total project cost of $341 million.

Background (cont.)

In December 2003, LAWA presented TSA with a summary of inadequacies it had found in the original concepts, and the associated potential cost and scheduling impacts.

In April 2005, LAWA submitted a revised cost estimate to TSA based on TSA approved concepts from a September 2004 LAWA engineering study.

This revised cost estimate included a request that TSA amend the LOI/MOA to increase the federal reimbursement by about $122 million, a sum that would raise the total federal reimbursement to about $378 million.

As of February 2007, TSA has obligated $256 million in accordance with the schedule set forth in the LOI/MOA for Los Angeles. As of October 2006, TSA had reimbursed LAWA for about $26 million in expenses from the total amount obligated.

LAWA officials stated that they were under a tight timeframe to apply for the LOI because TSA told them that federal funding was limited and that 17 other airports were competing for the funding.

TSA’s Boeing contracting team spent about 12 weeks from January to April 2003 to develop initial drawings (known as concept development drawings) and construction cost estimates. Between April and September 2003, TSA and LAWA negotiated the addition of administrative and contingency costs to the estimate.

TSA’s in-line system design guidance was evolving while the estimate was being developed, requiring frequent changes to the conceptual drawings.

Accuracy of Cost Estimates Depends on Completeness and Maturity of Information

The accuracy of an estimate depends on the quality of information

known about the project at the time the estimate is being prepared.

The 2003 estimate was made at the “concept development” stage where the final project cost can be expected to range from 50 percent under to 100 percent over the estimated cost based on construction industry guidance.5 The 2005 estimate was made at the “design development” stage where the final project cost can be expected to range from 20 percent under to 30 percent over the estimated cost. (See fig.1.)

Construction Industry Institute, Improving Early Estimates Research Team, Improving Early Estimates: Best Practices Guide, (September 1998.)

The Use of Preliminary Information Limited the Precision of the 2003 Estimate

Due to the preliminary information on which it was based, the 2003 estimate

did not foresee some of the challenges of retrofitting new systems into existing buildings and did not include space for various screening operations.

LAWA determined that the 2003 concepts for each terminal did not allow for sufficient space for the EDS machines, conveyor belts, and the construction of baggage inspection rooms, rooms for computer monitors for on-screen resolution of EDS alarms and rooms for computer routers and servers.

LAWA determined that the placement of EDS machines in the 2003 concepts needed to be changed in five of the nine terminals at LAX and both ONT terminals.

The 2005 Estimate Was Based on More Definitive Information

The revised 2005 estimate was based on concepts approved by TSA in 2004 which included more definitive information about terminal design requirements than the 2003 estimate.

The 2005 estimate designs included new construction and excavation, leading to cost estimate increases.

The 2005 estimate included the incorporation of 20 additional baggage inspection rooms, 9 on-screen resolution rooms and 10 computer rooms at LAX and ONT terminals.

The 2005 estimate included over $11 million in networking costs and costs associated with on-screen resolution of EDS alarms.

TSA also highlighted two additional factors that impacted the estimates—cost increases due to the delay in beginning construction of the project and the escalation of construction costs between 2003 and 2005.

In 2004, LAWA determined that Boeing had made a mathematical error in the 2003 concept development estimate: construction costs were only included for one of the two baggage screening facilities and neither connecting tunnel at ONT. In January 2007, TSA officials told us that they were not able to substantiate the mathematical error in the 2003 concept development estimate.

Changes in TSA Guidance and Evolving Technologies Contributed to Rise in Estimate

According to LAWA, changes in TSA guidance on in-line system

designs after the LOI/MOA was signed necessitated system redesigns.

Because few in-line systems were in use at the time of the LOI/MOA, limited information on the capabilities of the in-line EDS machines, including actual bags screened per hour and false alarm rates, was available for modeling the systems.

In June 2006, TSA produced Recommended Security Guidelines for Airport Planning, Design and Construction to guide future construction of in-line checked baggage screening systems. TSA also expects to release more detailed guidelines for in-line system planning and design in a few months.

Concluding Observations

The rise in the estimate between 2003 and 2005 was primarily

related to the fact that the 2003 estimate was developed at an early stage in the design process and was therefore based on preliminary data and assumptions that were subject to change.

According to TSA and LAWA officials, both TSA and Los Angeles signed the LOI/MOA knowing the preliminary nature of the cost estimate.

The LOI/ MOA affords TSA flexibility to amend the agreements to account for changed circumstances. However, under the terms agreed to in the LOI/MOA, TSA has no obligation to amend the LOI/MOA or to reimburse Los Angeles for any additional costs beyond those agreed to in the LOI/MOA and does not have plans to do so.

Time Line of Key Events from the 2003 and 2005 Cost Estimates

LAWA presents TSA summary of inadequacies in 2003 estimate.

TSA approves design assumptions for LAX and ONT.

LAWA submits revised cost estimate to TSA and requests amendment to LOI/MOA for additional funding.

LAWA requests TSA to amend the LOI/MOA for a time extension and additional funding.

Appendix III: Comments from Los Angeles World Airports