a

GAO

United States General Accounting Office

Report to Congressional Committees

May 2004 INFORMATION 

TECHNOLOGY

Early Releases of 

Customs Trade System 

Operating, but Pattern 

of Cost and Schedule 

Problems Needs to Be 

Addressed

GAO-04-719
www.gao.gov/cgi-bin/getrpt?GAO-04-719

To view the full product, including the scope 

and methodology, click on the link above. 

For more information, contact Randolph C. 

Hite at (202) 512-3439 or hiter@gao.gov. 

Highlights of GAO-04-719, a report to the

Subcommittees on Homeland Security, 

Senate and House Committees on 

Appropriations 

May 2004

INFORMATION TECHNOLOGY 

Early Releases of Customs Trade System 

Operating, but Pattern of Cost and 

Schedule Problems Needs to Be 

Addressed 

The DHS fiscal year 2004 ACE expenditure plan provides for certain 

activities, including system implementation infrastructure and support, 

operations and maintenance, and the definition and design of two future 

releases. This plan, including related program documentation and program 

officials’ statements, largely satisfies the legislative conditions imposed by 

the Congress. Some of the recommendations that GAO has previously made 

to strengthen ACE management have been addressed, and DHS has 

committed to addressing those that remain. However, much remains to be 

done before these recommendations are fully implemented. Specifically, 

progress on overcoming human capital challenges has been slow. 

GAO made several observations about ACE. Specifically, the first two ACE 

releases, which have been deployed, are operating largely as intended. 

However, achieving this initial operating capability has been difficult. DHS 

has established a pattern of borrowing resources from future releases to 

address problems of near-term releases, which has adversely affected the 

cost, schedule, and capability commitments of ongoing releases. In 

particular, the delay in completing the second ACE release has introduced a 

pattern of increased reliance on concurrent activities to meet the dictated 

schedule, continued release schedule delays, and cost overruns for the 

ongoing releases, as the figure below indicates. This domino effect is 

continuing into Release 3 and beyond. This pattern is not likely to change 

unless the degree of concurrence among activities within and between 

releases is better controlled, which will require that the reasons for release 

quality problems that led to the concurrent activity be addressed. Until the 

reasons for this pattern are pinpointed and corrected, in part through 

implementation of GAO’s unaddressed recommendations, DHS will not be 

positioned to deliver promised capabilities on time and within budget, or 

produce mission value commensurate with investment cost. 

Cost Overruns and Schedule Delays, Releases 1–4 

The Department of Homeland 

Security’s (DHS) Bureau of 

Customs and Border Protection 

(CBP) is conducting a multiyear, 

multibillion-dollar acquisition of a 

new trade processing system 

planned to support the movement 

of legitimate imports and exports 

and strengthen border security. By 

congressional mandate, 

expenditure plans for this system, 

called the Automated Commercial 

Environment (ACE), must meet 

certain conditions, including GAO 

review. This study addresses the 

extent to which the latest plan, for 

fiscal year 2004, satisfies these 

conditions, provides information 

about DHS’s efforts to implement 

GAO’s recommendations for 

improving ACE management, and 

makes observations about ACE. 

To assist DHS in managing ACE 

and increasing the chances that 

future releases will deliver 

promised capabilities on time and 

within budget, GAO is making 

recommendations to the Secretary 

of Homeland Security aimed at 

addressing recurring cost and 

schedule problems. DHS concurred 

with GAO’s recommendations and 

described specific actions that it is 

taking to respond to each. 
Page i GAO-04-719 Customs Modernization

Contents

Letter 1

Compliance with Legislative Conditions 2

Status of Open Recommendations 2

Observations on Management of ACE 5

Conclusions 7

Recommendations for Executive Action 7

Agency Comments and Our Evaluation 8

Appendixes

Appendix I: IT: Early Releases of Customs Trade System Operating, but 

Pattern of Cost and Schedule Problems Needs to be 

Addressed 10

Appendix II: Comments from the U.S. Department of Homeland 

Security 103

Appendix III: GAO Contact and Staff Acknowledgments 107

GAO Contact 107

Staff Acknowledgments 107
Contents

Page ii GAO-04-719 Customs Modernization

Abbreviations

ACE Automated Commercial Environment

CBP Customs and Border Protection

CBPMO Customs and Border Protection Modernization Office

DHS Department of Homeland Security

IDIQ indefinite delivery/indefinite quantity

ITDS International Trade Data System

IV&V independent verification and validation

I

2

V2 integrated independent verification and validation 

OIG Office of Inspector General

OMB Office of Management and Budget

PTR program trouble report

SA-CMM Software Acquisition Capability Maturity Model

SAT system acceptance test

SEI Software Engineering Institute

SIT system integration test

SWIT software integration test

UAT user acceptance test

US-VISIT United States Visitor and Immigrant Status Indicator 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately.
Page 1 GAO-04-719 Customs Modernization

United States General Accounting Office

Washington, D.C. 20548

A

May 14, 2004 Letr

The Honorable Thad Cochran

Chairman

The Honorable Robert C. Byrd

Ranking Minority Member

Subcommittee on Homeland Security

Committee on Appropriations

United States Senate

The Honorable Harold Rogers

Chairman

The Honorable Martin Olav Sabo

Ranking Minority Member

Subcommittee on Homeland Security

Committee on Appropriations

House of Representatives

In January 2004, U.S. Customs and Border Protection (CBP) within the 

Department of Homeland Security (DHS) submitted to the Congress its 

fiscal year 2004 expenditure plan, seeking release of $318.7 million for its 

Automated Commercial Environment (ACE) program. ACE is to be CBP’s 

new trade system. The program’s goals include facilitating the movement of 

legitimate trade through more effective trade account management and 

strengthening border security by identifying import and export 

transactions that have an elevated risk of posing a threat to the United 

States. As required by CBP’s fiscal year 2004 appropriations,1

 we reviewed 

the expenditure plan. Our objectives were to (1) determine whether the 

ACE fiscal year 2004 expenditure plan satisfies certain legislative 

conditions, (2) determine the status of our open ACE recommendations, 

and (3) provide any other observations about the expenditure plan and 

DHS’s management of the ACE program.

On March 8, 2004, we briefed your offices on the results of this review. This 

report transmits the results of our work. The full briefing, including our 

scope and methodology, is reprinted as appendix I.

1

P.L. 108-90 (October 1, 2003).
Page 2 GAO-04-719 Customs Modernization

Compliance with 

Legislative Conditions

The expenditure plan satisfied or partially satisfied the conditions specified 

in DHS’s appropriations act. Specifically, the plan, including related 

program documentation and program officials’ statements, satisfied or 

provided for satisfying all key aspects of (1) meeting the capital planning 

and investment control review requirements of the Office of Management 

and Budget (OMB), (2) complying with the DHS enterprise architecture,2

and (3) review and approval by DHS and OMB. The plan partially satisfied 

the condition that specifies compliance with the acquisition rules, 

requirements, guidelines, and systems acquisition management practices of 

the federal government.3

Status of Open 

Recommendations

CBP is making progress in addressing our open recommendations. Each 

recommendation, along with the status of CBP’s actions to address it, is 

summarized below.

• Before building each ACE release (i.e., beginning detailed design and 

development), certify to Customs’ House and Senate appropriations 

subcommittees that the enterprise architecture has been sufficiently 

extended to provide the requisite enterprise design content, and has 

been updated to ensure consistency and integration across business 

areas. 

With respect to the fiscal year 2004 expenditure plan, CBP has completed 

action to implement this recommendation by using a process developed by 

the former Customs Service to certify ACE releases against the former 

Customs enterprise architecture. Further, DHS is developing a process to 

ensure alignment of ACE with the departmentwide enterprise architecture 

it is developing to replace DHS component agency and bureau 

architectures.

2

An enterprise architecture is an institutional blueprint for guiding and constraining 

investments in business process change and systems.

3

In June 2003, the Department of the Treasury’s Office of Inspector General (OIG) issued a 

report on the ACE program’s contract, concluding that the former Customs Service, now 

CBP, did not fully comply with Federal Acquisition Regulation requirements in the 

solicitation and award of the contract. CBP disagrees with the Treasury OIG conclusion. To 

resolve the disagreement, DHS asked us to render a formal decision. We are currently 

reviewing the matter.
Page 3 GAO-04-719 Customs Modernization

• Develop and implement a rigorous and analytically verifiable costestimating program that embodied the tenets of effective estimating as 

defined in the Software Engineering Institute’s (SEI) institutional 

and project-specific estimating guidance.4

CBP is in the process of implementing this recommendation. It has 

developed and begun to implement a cost-estimating program that 

embodies SEI’s models. For example, CBP hired a contractor to define and 

implement a cost-estimating program and develop independent cost 

estimates. Additionally, it has defined and documented processes for 

estimating expenditure plan costs, and tasked contractors with ensuring 

that expenditure plan estimates are evaluated against SEI criteria for 

validating software cost and schedule estimates. 

• Immediately develop and implement a human capital management 

strategy that provides both near- and long-term solutions to program 

office human capital capacity limitations, and report quarterly to the 

appropriations committees on the progress of efforts to do so. These 

efforts should include defining the office’s skill and capability needs 

in terms that will allow the program office to attract qualified 

individuals and that will provide sufficient rewards and training, 

linked to performance, to promote their retention. 

CBP is in the process of implementing this recommendation, and it has 

reported on these actions to the Congress. In particular, the program office 

developed and began implementing a human capital management plan that 

called for addressing several areas, including filling vacant positions. 

However, the program office has continued to experience difficulty in 

filling key positions and has begun implementing a new staffing plan 

intended to address DHS’s concern that the program office has insufficient 

government program management staff.

4

SEI’s institutional and project-specific estimating guidelines are defined in Robert E. Park, 

Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of 

Software Organizations, CMU/SEI-95-SR-005 (Pittsburgh, Pa.: Carnegie Mellon University 

Software Engineering Institute, 1995) and A Manager’s Checklist for Validating Software 

Cost and Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburg, Pa: 1995), respectively.
Page 4 GAO-04-719 Customs Modernization

• Develop and implement process controls for SEI’s Software 

Acquisition Capability Maturity Model (SA-CMM5

) level 2 key process 

areas and for the level 3 acquisition risk management key process 

area; develop and implement each of the missing SA-CMM key 

practices for these key process areas, and until this is accomplished, 

report to the appropriations committees quarterly on the progress of 

efforts to do so. 

CBP has implemented this recommendation. In November 2003, SEI 

assigned the program a level 2 rating, meaning that the program had 

established basic acquisition management processes. Further, CBP had 

addressed the two weaknesses SEI identified in the level 3 acquisition risk 

management area, and has reported to the Congress on the status of its 

acquisition management improvement activities. 

• Establish an independent verification and validation (IV&V) 

function to assist CBP in overseeing contractor efforts, such as 

testing.

6

CBP is in the process of implementing this recommendation. The program 

office has established an IV&V function and has designated its IV&V 

contractor. However, program documentation describes roles for the IV&V 

contractor that are so integral to ACE development, testing, and 

deployment, as well as to ACE program management, that they raise 

questions about how the contractor could also perform independent ACErelated assessments. CBP officials stated that they plan to address this 

situation.

• Have future ACE expenditure plans specifically address any proposals 

or plans, whether tentative or approved, for extending and using ACE 

infrastructure to support other homeland security applications, 

including any impact on ACE of such proposals and plans. 

5

Capability Maturity Model (CMM) is a service mark of Carnegie Mellon University, and 

CMM is registered in the U.S. Patent and Trademark Office. The SA-CMM identifies key 

process areas that are necessary to effectively manage software-intensive system 

acquisitions. Level 2 is the second level of the SA-CMM’s five-level scale; achieving this level 

means that an organization has the software acquisition rigor and discipline to repeat 

project successes. 

6

IV&V involves having an independent organization conduct unbiased reviews of 

management processes, products, and results with the goal of verifying and validating that 

these meet stated requirements and standards.
Page 5 GAO-04-719 Customs Modernization

CBP plans to implement this recommendation. Program officials 

acknowledged the potential for ACE infrastructure to support other DHS 

system applications and the potential for integrating ACE data and 

applications with those systems. According to program officials, to begin 

preparing for this potential, they have focused initially on discussing 

collaboration opportunities with DHS’s United States Visitor and Immigrant 

Status Indicator Technology (US-VISIT) program.7

Observations on 

Management of ACE

We recognize accomplishments to date, including the deployment of the 

first 2 of 10 planned ACE releases, and address the need for rigorous and 

disciplined program management practices relating to managing ACE’s 

costs, schedule, and requirements, as well as the degree of concurrent 

system development activities. An overview of the observations follows: 

• Release 1 and 2 testing revealed a sufficient volume and significance 

of system defects to affect schedule commitments. Release 1 testing 

revealed system defects, which were corrected about 3 months after the 

testing was concluded. Release 2 testing revealed more system defects 

than expected; addressing these defects required more time than 

originally planned for Release 2 testing.

• Requirements planned for Releases 1 and 2 have been deferred to later 

releases. Releases 1 and 2 were originally planned as a single release 

that would satisfy 465 requirements. Subsequently, 103 of these 

requirements, which included capabilities related to document 

management, data recovery, and security, were deferred to Releases 3 

and 4, and 14 were deferred to later releases, for a total of 117 deferred 

requirements.

• Release 2 delays have set in motion a pattern of increased reliance on 

concurrent activities, continued release delays, and cost overruns. The 

time required to resolve defects resulted in overlap of Release 2 test 

phases. Delays in completing Release 2 increased its overlap with 

Release 3 and have contributed to Release 3 delays. Likewise, Release 3 

delays will increase overlap with Release 4 and could cause Release 4 

delays. 

7

US-VISIT is a governmentwide program to collect, maintain, and share information on 

foreign nationals in order to enhance national security and facilitate legitimate trade and 

travel while adhering to U.S. privacy laws and policies.
Page 6 GAO-04-719 Customs Modernization

Finally, the increased concurrency and schedule delays resulted in 

Releases 1 through 4 costing more than planned.8

• Releases 1 and 2 are largely operating according to adjusted 

requirements, defect density has stabilized, and the system is mostly 

up and running when needed. Defects prevalent in Releases 1 and 2 

have stabilized, but nontrivial defects remain. Releases 1 and 2, 

however, are mostly available when needed. 

• Following problems with Release 1, steps were taken to avoid future 

problems, but the success of these actions is unclear. As a result of 

Release 1 cost overruns and schedule delays, CBP directed its 

contractor to develop and implement a plan to improve program 

performance. The objective of this plan was to demonstrate serious, 

tangible, and measurable actions to resolve program issues. However, 

program officials could not point to metrics that demonstrated 

improved performance, and the cost to develop and implement the 

corrective action plan is unknown. 

• Initial releases represent a small fraction of the total ACE software. 

Releases 1, 2, 3, and 4 account for only about 14 percent of ACE 

software and are to be designed, developed, and deployed within about 

3 years. In contrast, the remaining six ACE releases constitute 86 

percent of ACE software but are to be designed, developed, and 

deployed within about 3-1/2 years. CBP officials stated that they are 

currently studying the extent to which off-the-shelf software can be 

employed in the remaining six releases.

• The reliability of key cost estimates in the expenditure plan is unclear.

The cost estimates in the expenditure plan varied from independent cost 

estimates, and the derivation of the expenditure plan estimates only 

partially satisfied SEI guidance for assessing the effectiveness of 

processes used to derive cost estimates. 

8

Our March 8, 2004, briefing to the staffs of the Subcommittees on Homeland Security, 

Senate and House Committees on Appropriations, reported a $23.3 million overrun for 

Releases 1 and 2 in addition to a $36.2 million overrun for Releases 3 and 4. Subsequently, 

CBP’s ACE development contractor revised the estimated overrun for Releases 3 and 4 to 

$46 million.
Page 7 GAO-04-719 Customs Modernization

Conclusions The fiscal year 2004 ACE expenditure plan, along with program 

documentation and officials’ commitments, largely satisfies the legislative 

conditions imposed by the Congress. Further, many of the 

recommendations that we have made to strengthen ACE management have 

been addressed, and CBP leadership has committed to addressing those 

that remain. However, much must still be done before these 

recommendations are fully implemented. Particularly, progress has been 

slow on overcoming human capital challenges.

To CBP’s credit, the first two ACE releases are operating largely as 

intended. Achieving this initial operating capability, however, has not 

occurred without difficulty, such as borrowing resources from future 

releases to overcome problems on near-term releases; similar difficulties 

that could affect cost, schedule, and capability commitments are being 

experienced on ongoing releases. This pattern is not likely to change unless 

the degree of concurrency among activities within and between releases is 

better controlled, and the underlying reasons for introducing this 

concurrency are addressed. While time and resources were invested in 

management improvements over a year ago to address root causes, the 

absence of meaningful measures for determining whether these 

investments will be successful, coupled with limited progress on some of 

our previous recommendations, makes CBP’s chances of delivering future 

ACE release capabilities on time and within budget uncertain at best. 

Recommendations for 

Executive Action

To assist CBP in managing ACE and increasing the chances that future 

releases will deliver promised capabilities on time and within budget, we 

recommend that the Secretary of Homeland Security, through the Under 

Secretary for Border and Transportation Security and the CBP 

Commissioner, direct the CBP Chief Information Officer to take the 

following actions: 

• Ensure the independence of its IV&V agent.

• Ensure that future expenditure plans are based on cost estimates that 

are reconciled with independent cost estimates.

• Reconsider the ACE acquisition schedule and cost estimates in light of 

early release problems, including these early releases’ cascading effects 

on future releases and their relatively small size compared with later 
Page 8 GAO-04-719 Customs Modernization

releases and in light of the need to avoid the past levels of concurrency 

among activities within and between releases.

• Define measures, and collect and use associated metrics, for 

determining whether prior and future program management 

improvements are successful.

• Report quarterly to the House and Senate Committees on 

Appropriations on efforts to address the above, as well as on our 

previous but unaddressed recommendations.

Agency Comments and 

Our Evaluation

In written comments on a draft of this report signed by the director, DHS 

Bankcard Programs and GAO/OIG Liaison, DHS concurred with our 

recommendations and stated actions that it is taking or plans to take to 

implement each. DHS also provided recommendations for disclosing 

certain sensitive information included in our draft report. We modified this 

report in accordance with DHS’s recommendations. DHS’s comments are 

reprinted in appendix II. 

We are sending copies of this report to the Chairmen and Ranking Minority 

Members of other Senate and House committees and subcommittees that 

have authorization and oversight responsibilities for homeland security. We 

are also sending copies to the Secretary of Homeland Security, the Under 

Secretary for Border and Transportation Security, the CBP Commissioner, 

and the Director of OMB. In addition, the report will be available at no 

charge on the GAO Web site at http://www.gao.gov. 
Page 9 GAO-04-719 Customs Modernization

Should you or your offices have any questions on matters discussed in this 

report, please contact me at (202) 512-3439 or by e-mail at hiter@gao.gov. 

Other contacts and key contributors to this report are listed in appendix III.

Randolph C. Hite

Director, Information Technology Architecture 

and Systems Issues
Page 10 GAO-04-719 Customs Modernization

Appendix I

IT: Early Releases of Customs Trade System Apendxies

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed ApendxiI

Information Technology: Early Releases of Customs 

Trade System Operating, but Pattern of Cost and 

Schedule Problems Needs to Be Addressed

Briefing to the Staffs of the

Subcommittees on Homeland Security,

Senate and House Committees on Appropriations

March 8, 2004

This briefing has been modified to exclude sensitive material that the Bureau of Customs and Border

Protection deemed to be acquisition-sensitive.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 11 GAO-04-719 Customs Modernization

2

Briefing Overview

Introduction

Objectives

Results in Brief

Background

Results

• Legislative Conditions

• Status of Recommendations

• Observations

Conclusions

Recommendations

Agency Comments

Attachment 1: Scope and Methodology
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 12 GAO-04-719 Customs Modernization

3

Introduction

The Department of Homeland Security (DHS) Bureau of Customs and Border

Protection (CBP), formed from the former U.S. Customs Service and other entities 

with border protection responsibility, is acquiring a new trade processing system, 

known as the Automated Commercial Environment (ACE). The goals of ACE are to

• facilitate the movement of legitimate trade through more effective trade 

account management;

• strengthen border security by identifying import/export transactions that have

an elevated risk of posing a threat to the United States or of violating a trade

law or regulation; and

• provide a single system interface between the trade community1 and the 

federal government,2 known as the International Trade Data System (ITDS),

and thereby reduce the data reporting burden placed on the trade community

while also providing federal agencies with the data and various capabilities to 

support their respective international trade and transportation missions.

1 Members of the trade community include importers and exporters, brokers and trade advisors, and carriers.

2 Federal agencies include CBP and other federal agencies responsible for managing international trade and transportation processes.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 13 GAO-04-719 Customs Modernization

4

Introduction

The Department of Homeland Security Appropriations Act, 2004,1 limits obligating

any funds for ACE until DHS submits for approval to the House and Senate

Committees on Appropriations a plan for expenditure that

1. meets the capital planning and investment control review requirements

established by the Office of Management and Budget (OMB), including

Circular A-11, part 3;2

2. complies with CBP’s enterprise architecture;

3. complies with the acquisition rules, requirements, guidelines, and systems 

acquisition management practices of the federal government;

4. is reviewed and approved by the CBP Investment Review Board,3 DHS, and 

OMB; and 

5. is reviewed by GAO.

1 Pub. L. 108-90 (Oct. 1, 2003).

2 OMB Circular A-11 establishes policy for planning, budgeting, acquisition, and management of federal capital assets.

3 The purpose of the IRB is to integrate capital planning and investment control, budgeting, acquisition, and management of

investments. It is also to ensure that spending on investments directly supports and furthers the mission and that this spending

provides optimal benefits and capabilities to stakeholders and customers.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 14 GAO-04-719 Customs Modernization

5

Introduction

In the Department of Homeland Security Appropriations Act, the Congress 

appropriated approximately $318.7 million for the ACE program.1

DHS submitted its fiscal year 2004 expenditure plan for $318.7 million on January 

21, 2004, to its House and Senate Appropriations Subcommittees on Homeland

Security.

1 Pub. L. 108-90 (Oct. 1, 2003).
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 15 GAO-04-719 Customs Modernization

6

Objectives

Objectives

As agreed, our objectives were to

• determine whether the ACE fiscal year 2004 expenditure plan satisfies the

legislative conditions,

• determine the status of our open recommendations made on ACE, and

• provide any other observations about the expenditure plan and DHS’s

management of the ACE program.

We conducted our work at CBP headquarters and contractor facilities in the 

Washington, D.C., metropolitan area from January through March 2004 in 

accordance with generally accepted government auditing standards. Details of our 

scope and methodology are given in attachment 1.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 16 GAO-04-719 Customs Modernization

7

Results in brief: Objective 1

Legislative Conditions

Fiscal year 2004 ACE expenditure plan satisfaction of legislative conditions

Legislative conditions Status

1. Meets the capital planning and investment control review requirements

established by OMB, including OMB Circular A-11, part 7.a

Satisfiedb

2. Complies with DHS’s enterprise architecture.c Satisfied

3. Complies with the acquisition rules, requirements, guidelines, and systems

acquisition management practices of the federal government.

Partially satisfiedd

4. Is reviewed and approved by the CBP Investment Review Board, DHS, and

OMB.

Satisfied

5. Is reviewed by GAO. Satisfied

Source: GAO. 

a

Capital investment and control requirements are now found in OMB Circular A-11, part 7, rather than part 3.

b

Satisfied or provides for satisfying every aspect of the condition that we reviewed.

c

As agreed with the staffs of the appropriations subcommittees, we assessed compliance with the DHS enterprise

architecture in lieu of the CBP enterprise architecture.

d

Satisfied or provides for satisfying many, but not all, key aspects of the condition that we reviewed.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 17 GAO-04-719 Customs Modernization

8

Results in brief: Objective 2

Open Recommendations

Status of actions to implement our open recommendations

GAO recommendations Status

Before building each ACE release, certify to CBP’s appropriations

subcommittees that the enterprise architecture has been sufficiently extended

and updated. 

Completea

Develop and implement a rigorous and analytically verifiable cost estimating

program.

In progressb

Immediately develop and implement a human capital management strategy

that provides both near and long-term solutions; develop and implement

missing human capital practices, and report quarterly to Congress on

progress.

In progress

Source: GAO.

a

Actions have been taken to fully implement the recommendation.

b

Actions are under way to implement the recommendation.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 18 GAO-04-719 Customs Modernization

9

Results in brief: Objective 2

Open Recommendations

Status of actions to implement our open recommendations

GAO recommendations Status

Develop and implement process controls for Software Engineering Institute

Software Acquisition Capability Maturity Modela level 2 key process areas and

for the level 3 acquisition risk management key process area; develop and

implement missing practices, and report quarterly to Congress on progress.

Complete

Establish an independent verification and validation function to assist CBP in

overseeing contractor efforts, such as testing.b

In progress

Have future ACE expenditure plans specifically address any proposals or

plans for extending and using ACE infrastructure to support other homeland

security applications.

Plannedc

Source: GAO.

a

The Software Acquisition Capability Maturity Model® developed by Carnegie Mellon University’s Software Engineering

Institute (SEI) defines five levels of maturity that provide a roadmap for continuously improving an organization’s acquisition

management processes.

b

Independent Verification and Validation involves having an independent organization conduct unbiased reviews of test

management processes, products, and results with the goal of verifying and validating that these meet stated requirements

and standards.

c

Actions are planned to implement the recommendation.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 19 GAO-04-719 Customs Modernization

10

Results in brief: Objective 3

Observations

Observations:

• Release 1 and 2 testing revealed a sufficient volume and significance of 

system defects to impact schedule commitments.

• Requirements planned for Releases 1 and 2 have been deferred.

• Release 2 delays have set in motion a pattern of increased reliance on 

concurrent activities, continued release delays, and cost overruns.

• Releases 1 and 2 are largely operating according to adjusted requirements;

defect density has stabilized and system is mostly up and running when

needed.

• Following Release 1 problems, steps were taken to avoid future problems, but 

success of actions is unclear.

• Initial releases represent a small fraction of the total ACE software.

• Reliability of key cost estimates in expenditure plan is unclear.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 20 GAO-04-719 Customs Modernization

11

Results in brief: Objective 3

Observations

To better ensure that future ACE releases deliver promised capability within budget

and on time, we are making recommendations to the DHS Secretary to address a 

recurring pattern of cost and schedule problems.

In their comments on a draft of this briefing, DHS, CBP, and ACE officials, including

the DHS Chief Information Officer and the CBP Acting Chief Information Officer, 

generally agreed with the briefing and stated that it was fair and balanced.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 21 GAO-04-719 Customs Modernization

12

Background

ACE-Related Business Functions 

ACE is to support CBP administration, enforcement, and trade compliance

business operations.

• Supported administration business operations include (1) processing and

recording of revenue generated from trade compliance, and (2) maintaining

the general ledger and accounting.

• Supported enforcement business operations include identifying import and

export transactions (i.e., cargo, conveyances, crew, and passengers) that 

have an elevated risk of threatening the United States or violating a trade law

or regulation.

• Supported trade compliance business operations include (1) establishing and

maintaining trade accounts; (2) processing and releasing cargo for import or

export; tracking and monitoring of conveyances, cargo, and individuals

involved in importing and exporting; (3) liquidating import entries including the 

processing of protests and decisions and the potential authorization of 

refunds; and (4) ruling on import and export legal, regulatory, and policy 

issues.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 22 GAO-04-719 Customs Modernization

13

Background

Description of ACE Technical Architecture

The ACE technical architecture is to consist of layers or tiers of computer 

technology:

• The Client Tier includes user workstations and external system interfaces.

• The Web & Interface Tier provides the mechanisms for the user workstations

and external systems to access ACE.

• The Enterprise Application Integration Tier provides the middleware for 

integrating and routing information between ACE software applications and

legacy systems.

• The Applications Tier includes software applications comprising commercial

products (e.g., SAP1), custom-developed software, and existing legacy

systems that provide the applications supporting business processes.

• The Data Tier includes the business data, storage resources, and access

services used by the ACE software applications and the legacy systems. 

Security and data privacy are embedded in all five layers.

1 SAP is a commercial enterprise resource planning software product that has multiple modules, each performing separate

but integrated business functions. CBP is using SAP as part of a joint project between its Office of Finance to support

financial and property management and the CBP Modernization Office (CBPMO) to support such ACE functions as customer

relationship management.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 23 GAO-04-719 Customs Modernization

14

Background

ACE Technical Architecture

Simplified view of ACE technical architecture
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 24 GAO-04-719 Customs Modernization

15

Background

Acquisition Strategy

CBP’s initial strategy provided for the acquisition of ACE in four increments over 4 

years. In September 2002, CBP modified this strategy to acquire and deploy the 

first two increments in two releases, with all increments and releases acquired over 

4 years. In October 2003, CBP changed plans to acquire ACE through the

acquisition and deployment of 10 releases over 6 years. 

The following figure maps the initial four increments to the 10 releases now

planned, 7 of which are called “releases” (e.g., Release 1, Release 2, ... Release

7) and 3 of which are called “selectivity” (e.g., Selectivity 1, Selectivity 2, and

Selectivity 3).
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 25 GAO-04-719 Customs Modernization

16

Background

Acquisition Strategy

Evolution of ACE from 4 increments to 10 releases
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 26 GAO-04-719 Customs Modernization

17

Background

Composite View of ACE Releases

The functionality associated with, status of, and plans for the 10 ACE releases are 

as follows. 

Release 1: Provide IT infrastructure—computer hardware and system software—to 

simultaneously support operating the subsequent system releases. This release

was deployed in October 2003 and is operating.

Release 2: Provide initial group of CBP national account managers1 and 41

importers access to account information, such as trade activity. This release was 

deployed in October 2003 and is operating.

Release 3: Provide additional account managers and importers, as well as brokers 

and carriers,2 access to account information; provide initial financial transaction

processing and CBP revenue collection capability, allowing importers and their

brokers to make monthly payments of duties and fees. This release is under

development and scheduled for deployment in August 2004.

1 CBP national account managers work with the largest importers to ensure their compliance with trade laws.

2 Brokers obtain licenses from CBP to conduct business on behalf of the importers by filling out paperwork and obtaining a

bond; carriers are individuals or organizations engaged in transporting goods for hire.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 27 GAO-04-719 Customs Modernization

18

Background

Composite View of ACE Releases

Release 4: Provide truck manifest1 processing and interfacing to legacy 

enforcement systems and transaction history data. This release is under

development and scheduled for deployment in February 2005.

Release 5: Provide legacy systems interface for airport, seaport, and rail crossing

inspectors for accessing supply chain, enforcement, and compliance data, as well 

as additional manifest functionality for selected federal government agencies

(including the Food and Drug Administration) and the trade community. This

release is scheduled for deployment in August 2005.

Selectivity 1: Identify shipments that pose a security risk using new and different

combinations of data than current legacy systems permit, as well as end-user

defined threat identification criteria. This release is scheduled for deployment in 

September 2005.

1 Manifests are lists of passengers or invoices of cargo for a vehicle, such as a truck, ship, or plane.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 28 GAO-04-719 Customs Modernization

19

Background

Composite View of ACE Releases

Release 6: Provide additional account management functionality for daily

statement creation, refund identification and generation, and expanded payment

acceptance; license, permit, and certificate granting, tracking, and revoking; and 

cargo, conveyance, and equipment tracking. This release is scheduled for 

deployment in December 2006.

Selectivity 2: Identify shipments that pose a security risk during authorized

movements. This release is scheduled for deployment in May 2006.

Release 7: Provide manifest processing for cargo moving between modes of

transportation; additional cargo and conveyance tracking and initial export 

processing functions; importer activity summary; and regulatory audit, protest, and 

drawback1 processing. This release is scheduled for deployment in June 2007.

Selectivity 3: Identify imports and exports (e.g., mail, courier, hand-carried items,

and shipments transitioning between modes of transportation) that pose a security 

risk. This release is scheduled for deployment in November 2006.

1 A refund of duties on an imported product subsequently exported or used to produce a product for export.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 29 GAO-04-719 Customs Modernization

20

Background

Composite View of ACE Releases

In addition to the expanding functionality planned from one release to the next, the 

number of users and locations is planned to grow with each release, as shown

below.

Note: Selectivity 1, 2, and 3 are planned to be accessible by 37,000 or more users via the CBP network and are not included in the above figures.

Source: CBP.

Number of Users

3300 14300

95800

677150

45000

266250

0

100000

200000

300000

400000

500000

600000

700000

800000

1 & 2 3 4 5 6 7

Release

Number of Ports

0 0

304 276

85 85

0

50

100

150

200

250

300

350

1 & 2 3 4 5 6 7

Release
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 30 GAO-04-719 Customs Modernization

21

Background

Contract Overview

CBP is acquiring and implementing ACE through an indefinite-delivery, indefinitequantity contract1 awarded on April 27, 2001, to IBM Global Services. IBM and its 

subcontractors are collectively called the e-Customs Partnership (eCP). Through a 

series of contract task orders, CBP plans to acquire and implement ACE.

The following table describes and provides the status of the executed eCP task 

orders.

1 An indefinite-delivery, indefinite-quantity contract provides for an indefinite quantity, within stated limits, of supplies or

services during a fixed period of time. The government schedules deliveries or performance by placing orders with the

contractor.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 31 GAO-04-719 Customs Modernization

22

Background

Contract Tasks

Status and description of eCP task orders

No. Name Start Status Description

001 Program

management

August 2001 Completed

July 2003 

Initial program and project management; continued by

task 009.

002 Enterprise

architecture and

engineering

August 2001 Completed

June 2003 

Initial enterprise architecture and system engineering;

continued by task 010.

003 Requirements

and planning

August 2001 Completed

July 2002 

Initial requirements development and program planning

effort; continued by tasks for specific

increments/releases.

004 Releases 1 and 2 February 2002 Completed

October 2003 

Design, development, testing, and deployment of

Releases 1 and 2 (initially intended to build Increment 1,

which was subsequently divided into four releases;

Releases 3 and 4 are covered by task 008)

005 Requirements

definition

February 2002 Ongoing Development of Release 5 project plan, documentation

of ACE business processes, and development of an

ACE implementation strategy.

006 Enterprise

process

improvement

February 2002 Completed

December 2003 

Enterprise process improvement integration.

Source: GAO analysis based on CBP data.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 32 GAO-04-719 Customs Modernization

23

Background

Contract Tasks

Status and description of eCP task orders (cont’d)

No. Name Start Status Description

007 International

Trade Data

System

January 2002 Ongoing Assistance for participating government agencies to define

requirements for an integrated ACE/ITDS system.

008 Releases 3 and 4 August 2002 Ongoing Design, development, testing, and deployment of Releases 3 

and 4.

009 Foundation

program

management

February 2003 Completed

October 2003

Follow-on to task 001 to continue program and project

management activities; continued by Task 016.

010 Foundation

architecture and

engineering

February 2003 Completed

December 2003

Follow-on to task 002 to continue enterprise architecture and

system engineering activities; continued by task 017.

011 Infrastructure and

facilities

August 2002 Completed

March 2003

Acquisition and setup of the necessary infrastructure and

facilities for the contractor to design, develop, and test

releases. Infrastructure and facilities work is now part of task

009.

012 Operations and

maintenance

April 2003 Ongoing Establishment of the infrastructure to operate and maintain

releases.

Source: GAO analysis based on CBP data.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 33 GAO-04-719 Customs Modernization

24

Background

Contract Tasks

Status and description of eCP task orders (cont’d)

No. Name Start Status Description

013 Legacy scripts

modernization

June 2003 Completed

November 2003

Conversion of scripts for interfacing desktop PC

applications (MS Word and Excel) and mainframe

computer applications.

014 Knowledgebased risk

management

September 2003 March 2004

planned

completion

Development, demonstration, and delivery of a

prototype to provide CBP insight into whether

knowledge-based risk management should be used in 

ACE.

015 Technology

prototypes

October 2003 July 2004

planned

completion

Development and demonstration of technology

prototypes to provide CBP insight into whether the

technologies should be used in ACE.

016 Foundation

program

management -

workforce

transformation

February 2004 Ongoing Program management and support to organizational

change management through activities such as impact

assessments, end user training, communication, and

outreach.

017 Business

analysis

January 2004 Ongoing Planning for integrated business framework and

evaluation of technology options for future releases.

Source: GAO analysis based on CBP data.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 34 GAO-04-719 Customs Modernization

25

Background

ACE Schedule

March 2004
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 35 GAO-04-719 Customs Modernization

26

Background

Chronology of ACE Expenditure Plans 

Since March 2001, five ACE expenditure plans have been submitted.1

• On March 26, 2001, CBP submitted to its appropriations committees the first

expenditure plan seeking $45 million for the modernization contract to sustain 

CBPMO operations, including contractor support. The appropriations

committees subsequently approved the use of $45 million, bringing the total

ACE funding to $50 million.

• On February 1, 2002, the second expenditure plan sought $206.9 million to 

sustain CBPMO operations; define, design, develop, and deploy Increment 1, 

Release 1 (now Releases 1 and 2); and identify requirements for Increment 2 

(now Release 5 and Selectivity 1 in addition to part of Release 6 and 

Selectivity 2). The appropriations committees subsequently approved the use 

of $188.6 million, bringing total ACE funding to $238.6 million.

1 In March 2001, appropriations committees approved the use of $5 million in stopgap funding to fund program management

office operations.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 36 GAO-04-719 Customs Modernization

27

Background

Chronology of Five ACE Expenditure Plans

• On May 24, 2002, the third expenditure plan sought $190.2 million to define,

design, develop, and implement Increment 1, Release 2 (now Releases 3 and 

4). The appropriations committees subsequently approved the use of $190.2

million, bringing the total ACE funding to $428.8 million.

• On November 22, 2002, the fourth expenditure plan sought $314 million to

operate and maintain Increment 1 (now Releases 1, 2, 3, and 4), design and

develop Increment 2, Release 1 (now Release 5 and Selectivity 1), and to 

define requirements and plan Increment 3 (now part of Releases 6 and 7 and 

Selectivity 2 and 3). The appropriations committees subsequently approved

the use of $306.4 million, bringing total ACE funding to $735.2 million.

• On January 21, 2004, CBP submitted its fifth expenditure plan, seeking

$318.7 million for ACE implementation infrastructure and support, ACE 

operations and maintenance, and definition and design of Release 6 and 

Selectivity 2.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 37 GAO-04-719 Customs Modernization

28

Background

Mapping of ACE Releases to Expenditure Plans

How expenditure plan1 requests have funded or are to fund life cycle phases of 

each ACE release

Source: GAO analysis based on CBP data.

1Expenditure Plan 1 funded program management and planning only.

Requirements development

Design

Development

Deployment

Operations and maintenance

Release Expenditure Plan 2 Expenditure Plan 3 Expenditure Plan 4 Expenditure Plan 5

Release 1

Release 2

Release 3

Release 4

Release 5

Selectivity 1

Release 6

Selectivity 2

Release 7

Selectivity 3
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 38 GAO-04-719 Customs Modernization

29

Background

Summary of Expenditure Plan Funding

Summary of the ACE fiscal year 2004 expenditure plan

Plan activity

Release 6 definition and design

Selectivity 2 definition and design

Implementation infrastructure and support

Foundation program management

Foundation architecture and engineering

Workforce transformation and training

Operations and maintenance

CBPMO costs

Federally Funded Research and Development Center

International Trade Data System

SAPb

Management reserve

Total Expenditure Plan Funding a $318.7

Source: GAO based on CBP data.

a

Millions of dollars.

b

SAP is a commercial enterprise resource planning software product that has multiple modules, each performing separate

but integrated business functions. CBP is using SAP as part of a joint project between its Office of Finance to support

financial and property management and the CBPMO to support such ACE functions as customer relationship management.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 39 GAO-04-719 Customs Modernization

30

Objective 1 Results

Legislative Conditions

The fiscal year 2004 ACE expenditure plan satisfied or partially satisfied each of 

the legislative conditions.

Condition 1. The plan, in conjunction with related program documentation and

program officials’ statements, satisfied the capital planning and investment control 

review requirements established by OMB, including Circular A-11, part 7, which 

establishes policy for planning, budgeting, acquisition, and management of federal 

capital assets. 

Details of our analysis are shown on the table that follows.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 40 GAO-04-719 Customs Modernization

31

Objective 1 Results

Legislative Condition 1

Examples of A-11 conditions Results of our analysis

1. Indicate whether the project was

approved by the IRB and 

reviewed by Chief Financial and

Procurement Officers

The plan was approved by the DHS IRB, which is

chaired by the Deputy Secretary and composed of

department executives, including DHS’s Chief Financial

and Procurement Officers.

2. Provide justification and

describe acquisition strategy

The plan provides a high-level justification for ACE.

Supporting documentation describes the acquisition

strategy for all ACE releases, including Release 6 and

Selectivity 2 activities identified in the plan.

3. Summarize life cycle costs and

cost/benefit analysis, including

the return on investment

CBP’s November 2003 cost/benefit analysis

summarizes current life cycle and return on investment

cost information. CBP reports that the overall ACE

benefit to cost ratio for CBP alone is 1.43a

 and 6.66b for

CBP, other government agencies, and the trade

community. a

The government benefit to cost ratio of 1.43, which is for CBP only, was calculated by dividing the present value benefits of

$3.97 billion by the present value cost of $2.78 billion. According to the cost/benefit analysis, benefits are generated from the

efficiency gains of ACE and the decommissioning of the legacy Automated Commercial System.

b

The national benefit to cost ratio of 6.66, which is for CBP, other government agencies, and the trade community, was

calculated by dividing the present value benefits of $18.55 billion by the present value cost of $2.79 billion. The ratio may not

be exact due to rounding.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 41 GAO-04-719 Customs Modernization

32

Objective 1 Results

Legislative Condition 1

Examples of A-11 conditions Results of our analysis

4. Address enterprise

architecture

Alignment of ACE to the enterprise architecture has yet to be

determined, but DHS and CBP officials stated that they plan to

address this issue.c

5. Address security and privacy The security of Release 2 was certified on August 22, 2003,

and accredited on September 16, 2003; CBP plans to certify

and accredit future releases. CBP is in the process of preparing

a privacy impact assessment for Release 3 and expects to

complete it by March 2004.

6. Address Section 508

complianced

Release 2 was deployed without requiring Section 508

compliance by the system development contractor. The

requirement for Section 508 compliance was not recognized

and was not reflected in the designs of Releases 2 and 3. CBP 

decided to partially address Section 508 compliance in Release

3 and defer complete compliance to future releases. CBP plans

to assess Release 3 with respect to Section 508 compliance,

and implement an action plan to ensure that noncompliant

areas are corrected in future releases.

Source: GAO.

c

For more details, see the results of our analysis for the second legislative condition.

d

Section 508 of the Rehabilitation Act (29 U.S.C. 794d), as amended by the Workforce Investment Act of 1998 (P.L. 105-220),

August 7, 1998, requires federal agencies to develop, procure, maintain, and use electronic information technology in a way that

ensures that the technology is accessible to people with disabilities.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 42 GAO-04-719 Customs Modernization

33

Objective 1 Results

Legislative Condition 2

Condition 2. The plan and related program officials’ statements provide for 

complying with the DHS enterprise architecture.

DHS released version 1 of the architecture in October 2003,1 and plans to issue 

version 2 in September 2004.

According to the DHS Chief Information Officer (CIO), DHS is developing a process 

to align its systems modernization efforts, such as ACE, to its enterprise 

architecture. Alignment of ACE to the enterprise architecture has yet to be 

determined, but DHS CIO and CBP officials stated that they plan to address this 

issue. Moreover, the fiscal year 2004 expenditure plan states that Release 6 and 

Selectivity 2 will be aligned with the DHS enterprise architecture.

1 Department of Homeland Security Enterprise Architecture Compendium Version 1.0 and Transitional Strategy.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 43 GAO-04-719 Customs Modernization

34

Objective 1 Results

Legislative Condition 3

Condition 3. The plan, in conjunction with related program documentation, partially

satisfies the condition of compliance with the acquisition rules, requirements,

guidelines, and systems acquisition management practices of the federal 

government.

The Software Acquisition Capability Maturity Model (SA-CMM), developed by 

Carnegie Mellon University’s Software Engineering Institute (SEI), is consistent 

with the acquisition guidelines and systems acquisition management practices of 

the federal government, and it provides a management framework that defines

processes for acquisition planning, solicitation, requirements development and 

management, project management, contract tracking and oversight, and

evaluation.

On November 5, 2003, SEI assessed ACE acquisition management against the 

SA-CMM and assigned a level 2 rating, indicating that CBPMO has instituted basic 

acquisition management processes and controls in the following areas: acquisition

planning, solicitation, requirements development and management, project

management, contract tracking and oversight, and evaluation.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 44 GAO-04-719 Customs Modernization

35

Objective 1 Results

Legislative Condition 3

In June 2003, the Department of the Treasury’s Office of Inspector General (OIG) 

issued a report on the ACE program’s contract, concluding that the former Customs 

Service, now CBP, did not fully comply with Federal Acquisition Regulation

requirements in the solicitation and award of its contract because the ACE contract 

is a multiyear contract and not an indefinite delivery /indefinite quantity (IDIQ) 

contract. Further, the Treasury OIG found that the ACE contract type, which it 

determined to be a multiyear contract, is not compatible with the program’s stated 

needs for a contract that can be extended to a total of 15 years because multiyear

contracts are limited to 5 years. Additionally, the Treasury OIG found that Customs 

combined multiyear contracting with IDIQ contracting practices. For example, it 

plans to use contract options to extend the initial 5-year performance period.

CBP disagrees with the Treasury OIG conclusion. 

To resolve the disagreement, DHS asked GAO to render a formal decision. We are 

currently reviewing the matter.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 45 GAO-04-719 Customs Modernization

36

Objective 1 Results

Legislative Condition 4

Satisfies condition 4. The plan satisfies the condition that it be reviewed and

approved by the CBP IRB, DHS, and OMB.

On July 15, 2003, the CBP IRB approved the plan. DHS subsequently approved

the plan on November 20, 2003, and OMB approved the plan on January 20, 2004.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 46 GAO-04-719 Customs Modernization

37

Objective 1 Results

Legislative Condition 5

Satisfies condition 5. The plan satisfies the condition that it be reviewed by GAO.

Our review was completed on March 4, 2004.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 47 GAO-04-719 Customs Modernization

38

Objective 2 Results

Open Recommendation 1

Open recommendation 1: Before building each ACE release (i.e., beginning

detailed design and development), certify to Customs’ House and Senate

appropriations subcommittees that the enterprise architecture has been sufficiently 

extended to provide the requisite enterprise design content and has been updated

to ensure consistency and integration across business areas.

Status: Complete.

In response to our recommendation made in March 2002, the U.S. Customs 

Service established a process for certifying to its appropriations committees that 

the Customs enterprise architecture had been sufficiently extended and updated

before detailed design and development of each ACE release was begun.

In March 2003, the U.S. Customs Service business operations supported by ACE 

became part of the DHS CBP. 
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 48 GAO-04-719 Customs Modernization

39

Objective 2 Results

Open Recommendation 1

In May 2003, CBP used the process developed by the former Customs Service to 

certify Releases 3 and 4 against the former Customs enterprise architecture. At this 

time, DHS also committed to developing a single, departmentwide enterprise

architecture to replace DHS component agency/bureau architectures. DHS

released version 1 of its enterprise architecture in October 2003,1 and plans to 

issue version 2 of its architecture in September 2004.

According to the DHS CIO, DHS is developing a process to ensure alignment of its 

systems modernization efforts, including ACE, to its enterprise architecture. While 

this alignment step for ACE has not occurred yet, the fiscal year 2004 expenditure

plan states that Release 6 and Selectivity 2 will be aligned with the DHS 

architecture.

1 Department of Homeland Security Enterprise Architecture Compendium Version 1.0 and Transitional Strategy.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 49 GAO-04-719 Customs Modernization

40

Objective 2 Results

Open Recommendation 2 

Open recommendation 2: Develop and implement a rigorous and analytically

verifiable cost estimating program that embodies the tenets of effective estimating 

as defined in SEI’s institutional and project-specific estimating models.1

Status: In progress.

The program office has developed and begun to implement a cost estimating 

program that embodies SEI’s models. For example, CBP hired a contractor to 

define and implement a cost estimating program. The contractor has responsibility

for development of ACE life cycle cost estimates and independent cost estimates to 

support budget planning, budget execution, and contract task order evaluation.

Additionally, the program office has defined and documented processes for 

estimating expenditure plan costs and management reserve in addition to 

preparing independent cost estimates. Further, CBP tasked its support and

modernization contractors with ensuring that the expenditure plan estimates are 

evaluated against SEI criteria for validating software cost and schedule estimates.

1SEI’s institutional and project-specific estimating guidelines are defined in Checklists and Criteria for Evaluating the Cost

and Schedule Estimating Capabilities of Software Organizations and A Manager’s Checklist for Validating Software Cost and

Schedule Estimates, respectively.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 50 GAO-04-719 Customs Modernization

41

Objective 2 Results

Open Recommendation 3

Open recommendation 3: Immediately develop and implement a human capital

management strategy that provides both near- and long-term solutions to program

office human capital capacity limitations, and report quarterly to the appropriations

committees on the progress of efforts to do so. These efforts should include

defining the office’s skill and capability needs in terms that will allow the program

office to attract qualified individuals and that will provide sufficient rewards and 

training, linked to performance, to promote their retention.

Status: In progress

The program office developed and began implementing a human capital

management plan. This plan called for addressing several areas, including

identification of required core competencies (i.e., defined knowledge and skill

requirements), assessing existing capabilities, developing a training program, and 

filling vacant positions.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 51 GAO-04-719 Customs Modernization

42

Objective 2 Results

Open Recommendation 3

In its June and October 2003 reports to its appropriations committees on progress

in addressing these areas, the program office indicated that it had defined core 

competencies, was assessing capabilities and developing an annual training plan,

and had filled three vacancies and developed a plan for filling its remaining

vacancies. However, the program office has continued to experience difficulty in 

filling key positions, including three responsible for organizational change

associated with adopting the business process changes embedded in ACE

functionality.

Currently, program officials told us that the capability assessment and the annual

training plan were completed and both are awaiting approval. They also told us that 

they have suspended development of the original plan for filling vacancies, and in 

January 2004 began implementing a new staffing plan that was developed in 

response to a DHS IRB statement that the program office has insufficient

government program management staff. We made this same observation, along 

with recommendations to address it, 2 years ago.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 52 GAO-04-719 Customs Modernization

43

Objective 2 Results

Open Recommendation 4

Open recommendation 4: Develop and implement process controls for SEI’s SACMM level 2 key process areas and for the level 3 acquisition risk management

key process area; develop and implement each of the missing SA-CMM key

practices for these key process areas, and until this is accomplished, report to the 

appropriations committees quarterly on the progress of efforts to do so.

Status: Complete

CBP has taken steps to develop and implement each of the key process areas for 

the SA-CMM level 2 key process areas and the level 3 acquisition risk 

management area. SEI assessed the acquisition management of Releases 1 

through 4 against the SA-CMM level 2 criteria. On November 5, 2003, SEI 

assigned the program a level 2 rating, meaning that the program has established

basic acquisition management processes. 
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 53 GAO-04-719 Customs Modernization

44

Objective 2 Results

Open Recommendation 4

SEI also conducted an unrated assessment of the program’s acquisition risk

management capability in May 2003. The assessment identified two weaknesses:

(1) risk management was not integrated into acquisition planning and (2) a defined 

organizational process was not used to produce the program’s acquisition risk 

management plan. CBP has addressed these weaknesses by updating its 

acquisition planning process to include risk management activities and by

documenting an acquisition risk management process. 

CBP’s October 2003 report to its appropriations committees reported on the status 

of these acquisition management improvement efforts. 
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 54 GAO-04-719 Customs Modernization

45

Objective 2 Results

Open Recommendation 5

Open recommendation 5: Establish an independent verification and validation

(IV&V) function to assist CBP in overseeing contractor efforts, such as testing.

Status: In progress

The program office has established an integrated independent verification and 

validation (I2V2) function, and it has designated its I2V2 contractor.

According to CBP’s I2V2 strategy, the contractor is to review key eCP deliverables

and products and provide assessments and reports at various life-cycle stages and

milestones. Further, the strategy recognizes the importance of maintaining

independence in the conduct of I2V2 and outlines (1) a reporting structure that 

allows the contractor to report its findings to CBP executives and (2) an approach

to assigning staff to I2V2 tasks that is to ensure contractor independence from 

program management pressures.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 55 GAO-04-719 Customs Modernization

46

Objective 2 Results

Open Recommendation 5

However, other program documentation, such as the fiscal year 2004 expenditure

plan, the contractor’s statement of objectives, and the modernization program work 

breakdown structure, describes roles for the I2V2 contractor that are so integral to 

ACE development, testing, and deployment, as well as ACE program management,

that they raise questions about how the contractor could also perform independent

ACE-related assessments. For example, they assign the I2V2 contractor

responsibility for

• supporting development of long-range and short-term strategic plans for the 

modernization program;

• developing the task orders that contractually specify eCP’s work; and

• developing incentives for improving eCP’s performance.

CBP officials stated that they recognize the importance of ensuring the 

independence of their verification and validation function and plan to address this 

situation.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 56 GAO-04-719 Customs Modernization

47

Objective 2 Results

Open Recommendation 6

Open recommendation 6: Have future ACE expenditure plans specifically

address any proposals or plans, whether tentative or approved, for extending and 

using ACE infrastructure to support other homeland security applications, including

any impact on ACE of such proposals and plans.

Status: Planned

Program officials acknowledge the potential for ACE infrastructure to support other 

DHS system applications and the potential for integrating ACE data and 

applications with those systems. To begin preparing for this potential, program

officials told us they have focused initially on discussing collaboration opportunities

with DHS’s United States Visitor and Immigrant Status Indicator Technology (USVISIT) system.1 To date, those discussions have included

• sharing lessons learned on leveraging contractors and

• coordinating enterprise architecture activities.

1US-VISIT is a governmentwide program to collect, maintain, and share information on foreign nationals for enhancing

national security and facilitating legitimate trade and travel, while adhering to U.S. privacy laws and polices.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 57 GAO-04-719 Customs Modernization

48

Objective 3 Results

Observation 1

Observation 1: Release 1 and 2 testing revealed a sufficient volume and 

significance of system defects to impact schedule commitments.

Development of each ACE release includes software integration, system 

integration, and system acceptance testing by eCP, followed by user acceptance

testing by CBP. Generally, the purpose of these tests is to identify defects or 

problems in either meeting defined system requirements or in satisfying system

user needs. A brief description of the four types of testing is provided below.

Test Description

Software integration test

(SWIT)

Verify that the software subsystems and modules perform as

intended.

System integration test

(SIT)

Verify that related system, subsystem, or module components

are capable of integrating and interfacing with each other.

System acceptance test

(SAT)

Verify that the developed system, subsystem, or module

operates in accordance with requirements.

User acceptance test

(UAT)

Verify that the functional scope of the release meets the

business functions for the users.

Source: eCP.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 58 GAO-04-719 Customs Modernization

49

Objective 3 Results

Observation 1

Release 1 user acceptance testing revealed problems.

According to CBP’s test results report, user acceptance testing (UAT) for Release 1 

was intended to ensure that the design of the users’ workstation screens, the 

capabilities for navigating among screens, and the functionality associated with

these screens satisfied user needs.1 Additionally, CBP’s stated goal of Release 1 

UAT was to collect feedback from users and identify defects and improvement

opportunities.

UAT began on January 8, 2003, and was completed on January 24, 2003. It 

consisted of users (1) completing 28 test cases and (2) responding to 10 questions

designed to solicit their satisfaction (on a scale of 1 to 5, with 1 being very poor and 

5 being excellent) about different aspects of Release 1, such as screen design,

quality, reliability, availability, response time, and training.

1 In GAO-03-406, we reported that the first ACE release had completed SAT.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 59 GAO-04-719 Customs Modernization

50

Objective 3 Results

Observation 1

UAT results show that the average user satisfaction rating for Release 1 was

slightly less than 3 out of 5, and that this resulted in 38 defects, referred to as 

program trouble reports (PTRs). Of these 38, 11 were judged to be either critical or 

severe. These and other categories of PTRs are described below.

In order to successfully complete UAT and make Release 1 operational, CBP 

reported that it resolved 17 PTRs and deferred to Release 2 the remaining 21.

About 3 months after UAT concluded, CBP reported that all 38 PTRs were

resolved.

Category Description

Critical Defect prevents or precludes the performance of an operational or mission-essential

capability, jeopardizes safety or security, or causes the system, application, process, or 

function to fail to respond or end abnormally.

Severe Defect prevents or precludes system from working as specified or produces an error that

degrades or impacts system performance or user functionality.

Moderate Defect prevents or precludes system from working as specified or produces an error that

degrades or impacts system performance or user functionality. An acceptable (reasonable

and effective) workaround is in place that rectifies the defect until a permanent fix can be 

made.

Minor Defect is inconsequential, cosmetic, or inconvenient but does not prevent users from using the 

system to accomplish their tasks.

Source: GAO based on eCP data.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 60 GAO-04-719 Customs Modernization

51

Objective 3 Results

Observation 1

Release 2 integration, acceptance, and user acceptance testing revealed problems

that required additional time to address.

System integration testing (SIT). As previously stated, SIT is intended to verify that 

related system, subsystem, or module components are capable of integrating and 

interfacing with each other. Successfully completing SIT requires that all test cases 

achieve a “test passed” status, meaning that the test procedures were successfully

executed and completed, and the test objectives were met. 

Release 2 SIT, which began on December 31, 2002, and was to end on April 4, 

2003, consisted of 46 test cases. Of these 46, 33 were reported to be successfully

completed. According to the test results, of the remaining 13 test cases, either they 

could not be executed for various reasons (such as a capability not being available

in the current release) or a test case was changed during attempts to execute it. 

The 46 test cases, according to eCP’s projection, were expected to disclose 389 

defects; however, they resulted in 745. 
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 61 GAO-04-719 Customs Modernization

52

Objective 3 Results

Observation 1

When SIT concluded in June 2003 (about 2.5 months later than planned), 49 PTRs 

remained open (6 severe, 36 moderate, and 7 minor). However, because none of 

the tested items failed the SIT and the defects identified were either resolved or 

deferred to SAT or a later release, the SIT test procedures were judged to be 

successfully executed.

As of January 2004, the program office reports that all the critical and severe PTRs 

have been resolved, and that 16 PTRs are still open, of which 11 are moderate and 

5 are minor. 
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 62 GAO-04-719 Customs Modernization

53

Objective 3 Results

Observation 1

System acceptance testing (SAT). According to CBP’s test results report, SAT is 

intended to show that a system performs according to its specified functional and 

performance requirements. To pass SAT, all test cases must pass without 

identification of critical or severe defects.

Release 2 SAT was to begin April 7, 2003, and conclude on April 28, 2003, but 

because of the large number of unexpected defects, SAT began on May 1, 2003, 

and ended on June 5, 2003, which is about a 1 month delay. Of the SAT test cases 

executed, all but 19 reportedly passed. Of these 19, one resulted in a critical PTR, 

which was closed within 2 days, and 18 produced moderate or minor PTRs. The 

program office reports that those 18 were either corrected or deferred to future

releases based on implementation of acceptable procedural workarounds.

As of January 2004, 11 of the deferred PTRs reportedly remain open. An example

of a deferred PTR is Release 2’s inability to identify duplicate accounts. According

to a system integration manager, this defect was not categorized as critical or 

severe because an acceptable workaround was available (i.e., establishing a 

manual process to identify possible duplicate accounts until Release 3 is ready).
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 63 GAO-04-719 Customs Modernization

54

Objective 3 Results

Observation 1

User acceptance testing (UAT). According to CBP’s test results report, UAT is 

intended to verify that the system meets the needs of the users.

Release 2 UAT began on July 10, 2003, and was to end on August 15, 2003. It 

consisted of CBP and the trade community users executing 19 and 23 test cases, 

respectively, and being asked to rate the release in several areas using a 1 to 5 

scale (with 1 indicating very dissatisfied and 5 indicating very satisfied). The test

areas addressed the account management functionality included in Release 2, 

such as creating a new account, managing account details, and creating reports.

The average user satisfaction rating was slightly more than 4 out of 5, which 

indicated user satisfaction, according to the test results report. 

UAT nevertheless resulted in the identification of 17 defects. As of January 2004, 

15 of the associated PTRs are reported to have been resolved, and the remaining

2 are categorized as moderate.

The shaded portion of the graph on the following page shows the number of PTR’s 

open (i.e., unresolved) each week during Release 2 testing.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 64 GAO-04-719 Customs Modernization

55

Objective 3 Results

Observation 1

Release 2 Unresolved Defects by Week

0

10

20

30

40

50

60

70

80

90

100

11/18/2002

12/2/2002

12/16/200212/30/2002

1/13/20031/27/20032/10/20032/24/20033/10/20033/24/2003

4/7/2003

4/21/2003

5/5/2003

5/19/2003

6/2/2003

6/16/20036/30/20037/14/20037/28/20038/11/20038/25/2003

9/8/2003

9/22/200310/6/2003

10/20/2003

11/3/2003

11/17/2003

12/1/2003

12/15/200312/29/2003

1/12/20041/26/2004

Severity 1 Severity 2 Severity 3

Planned (1/2003):

Actual:

SIT SAT UAT and Pilot Period

SIT SAT UAT and Pilot Period

10/17/2003

Number of PTRs

Weeks

Source: GAO based on CBP data
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 65 GAO-04-719 Customs Modernization

56

Objective 3 Results

Observation 2

Observation 2: Requirements planned for Releases 1 and 2 have been 

deferred.

As previously described, each ACE release is expected to contain specific

functionality and performance capabilities. These capabilities are defined in a 

system requirements document, which lists the overall requirements for the entire 

ACE system. The requirements specification set breaks these into more detailed

requirements and assigns them to releases. 

Releases 1 and 2 were originally planned as a single release that was to satisfy 

465 requirements. Subsequently, 103 of these requirements were deferred to 

Release 3 and 4, and 14 requirements were deferred to later releases, for a total of 

117 deferred requirements. Also, 25 of 465 requirements were deleted from the 

baseline. In total, 142 requirements were either deferred or deleted from the 

original baseline, representing about 31 percent of the original baseline of 465 

requirements.

At the same time, 36 new requirements were added to the baseline for Release 1 

and 2, meaning that Release 1 and 2 satisfied 359 total requirements. The next 

slide summarizes the requirements changes in Release 1 and 2.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 66 GAO-04-719 Customs Modernization

57

Objective 3 Results

Observation 2

Source: GAO analysis based on eCP data

Requirements Changes

Release 1 and 2

465

323

103

36

14

25

0

50

100

150

200

250

300

350

400

450

500

Original Actual Deferred/Deleted

Number of Requirements

Deleted (25)

Moved to Release 5

or later (14)

Moved to Releases 3

and 4 (103)

Added (36)

Original
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 67 GAO-04-719 Customs Modernization

58

Objective 3 Results

Observation 2

According to the requirements specification set for Release 3 and 4, the following

categories of capabilities and functionality were deferred from Releases 1 and 2 to 

Releases 3 and 4:

• document management, document retention, workflow, and data recovery;

• network response time; and

• security (remote access authentication, data security during network transfer,

audit trails).

According to program officials, the deferred security requirements must be 

implemented by Release 4.

Program change requests cite the reasons for the respective requirements deferral. 

Reasons include the following:

• Document management and retention requirements were deferred because

neither Release 1 nor 2 needed to be able to accommodate either paper or

electronic documents.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 68 GAO-04-719 Customs Modernization

59

Objective 3 Results

Observation 2

• Workflow requirements were deferred because eCP and CBP needed

additional time to evaluate and select among commercial product options.

• Data recovery functionality was deferred because CBP chose to delay 

implementation of disaster recovery for ACE until Releases 3 and 4.

• Broker and carrier account functionality was deferred because full support for 

the functionality was not needed until Releases 3 and 4.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 69 GAO-04-719 Customs Modernization

60

Objective 3 Results

Observation 3

Observation 3: Release 2 delays have set in motion a pattern of increased 

reliance on concurrent activities, continued release delays, and cost

overruns.

As we previously reported,1 the importance of ACE to our country’s security and 

trade facilitation prompted CBP to take steps aimed at expediting ACE acquisition

and deployment, including introducing greater reliance on performing release

design, development, and testing activities concurrently. While such concurrency

can permit faster acquisition and deployment, it also introduces risks, which the 

ACE contractor has said include limited understanding of requirements before

design and development activities begin, uncertainty regarding the timely 

availability of commercial hardware and software products, and increased nearterm funding requirements. Other risks include contention for limited resources

(such as key personnel, as well as development and testing equipment and 

facilities) and dependencies among releases not being met.

1U.S. General Accounting Office, Customs Service Modernization: Management Improvements Needed on High-Risk

Automated Commercial Environment Project, GAO-02-545 (Washington, D.C.: May 13, 2002).
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 70 GAO-04-719 Customs Modernization

61

Objective 3 Results

Observation 3

Time required to resolve defects resulted in overlap of Release 2 integration and 

acceptance testing.

Resolving Release 2 defects also resulted in more overlap between testing

activities. The original Release 2 testing schedule projected an overlap of about 1.5 

months between SWIT and SIT, and no overlap between SIT and SAT. However,

due to the extension of testing periods caused by the high number of defects, the 

overlap between SWIT and SIT was about 4.5 months, and the SAT period was 

completely concurrent with SIT. The next slide shows the overlap of Release 2 

testing activities. 
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 71 GAO-04-719 Customs Modernization

62

Objective 3 Results

Observation 3

Source: GAO analysis based on CBP data.

Release 2 Testing Periods

9/29/200210/6/2002

10/13/200210/20/200210/27/2002

11/3/2002

11/10/200211/17/200211/24/2002

12/1/200212/8/2002

12/15/200212/22/200212/29/2002

1/5/2003

1/12/20031/19/20031/26/2003

2/2/20032/9/2003

2/16/20032/23/2003

3/2/20033/9/2003

3/16/20033/23/20033/30/2003

4/6/2003

4/13/20034/20/20034/27/2003

5/4/2003

5/11/20035/18/20035/25/2003

6/1/20036/8/2003

6/15/20036/22/20036/29/2003

Planned:

SWIT

SIT

SAT

Actual:

SWIT

SIT

SAT
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 72 GAO-04-719 Customs Modernization

63

Objective 3 Results

Observation 3

Delays in completing Release 2 increased its overlap with Release 3 and have 

contributed to Release 3 delays.

According to program officials and variance reports, some ongoing Release 3 work 

had to be curtailed when key resources slated for Release 3 (e.g., test and 

development environment, equipment, and key personnel) had to be diverted to 

address Release 2 PTRs and complete Release 2 test activities. Although some 

resource diversion continued through June 2003, as Release 3 resources were

used to complete Release 2 SIT (including regression testing of software changes

made to address PTRs), one report states that after June 2003, about 50 percent 

of the planned Release 3 SIT resources were redirected to support Release 2 

deployment activities.

According to program officials, Release 1 delays, coupled with Release 2 

dependencies on Release 1 and subsequent contention for key resources between

Releases 2 and 3, have had a domino effect, causing Release 3 resource shortfalls 

and consequent delays. For example, as of January 2004, a delay of 5 to 6 months 

in the March 2004 milestone for determining Release 3 operational readiness was 

reported.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 73 GAO-04-719 Customs Modernization

64

Objective 3 Results

Observation 3

Release 3 delays will increase overlap with Release 4 and could cause Release 4 

delays.

According to program officials and documentation, Release 3 and 4 development

began in August 2001, concurrent with ongoing Release 1 and 2 testing, but before 

they were aware of the large number of Release 1 and 2 defects. According to 

these officials, work was undertaken concurrently to minimize the time between 

releases and thereby maximize the efficiency in deploying releases.

While these officials agreed that the cascading Release 1 and 2 problems will

continue to impact Release 3, they stated that Release 4 will not be affected. One 

reason that they cited is that Release 4 will be tested in a recently acquired

separate laboratory environment from Release 3, thus eliminating the contention

for these key resources. 

Despite these statements, program variance reports show that Release 4 is being 

impacted. According to the reports, Release 3 resources targeted for Release 4 are 

not becoming available as early as planned because of delayed Release 3 

development and SWIT activities. Unless resources from an alternative source are 

forthcoming, this could impact Release 4 schedule commitments.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 74 GAO-04-719 Customs Modernization

65

Objective 3 Results

Observation 3

Increased concurrency and schedule delays resulted in Releases 1 through 4 

costing more than planned.

The baseline cost estimate for Releases 1 and 2 was $86.1 million. However, the 

actual cost was reported to be $109.4 million—an overrun of $23.3 million (27 

percent).

The baseline cost estimate for Release 3 and 4 was $149.5 million. However, CBP 

now estimates the cost to be $185.7 million—an overrun of about $36.2 million (24 

percent).

According to program documentation, including cost variance analysis reports,

factors contributing to the cost overruns included

• resources to eliminate Release 1 and 2 defects;

• overtime to minimize Release 3 and 4 schedule delays;

• unavailability of testing and development environments;

• Release 3 delays caused by resources being used to support delayed 

Release 2 activities;
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 75 GAO-04-719 Customs Modernization

66

Objective 3 Results

Observation 3

• added Release 3 and 4 scope as a result of requirements shift from Release

2; and

• Release 3 receipt of unresolved defects from Release 2.

CBP reported that it funded the $23.3 million Release 1 and 2 overrun with a $7.9 

million forfeiture of eCP’s incentive fees and a $15.4 million reallocation of funds 

from Release 5. CBP also reported that it is considering funding the $36.2 million

Release 3 and 4 overrun out of its accumulated management reserve.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 76 GAO-04-719 Customs Modernization

67

Objective 3 Results

Observation 4

Observation 4: Releases 1 and 2 are largely operating according to adjusted 

requirements; defect density has stabilized and system is mostly up and

running when needed.

Two indicators of how well a system is operating are defect density (i.e., the 

volume and severity of unresolved problems) and system availability (i.e., the time 

the system is operating satisfactorily, expressed as a percentage of time that the 

system is required to be operational).

Releases 1 and 2 began operating on October 17, 2003. According to CBP, these 

releases are providing support to 73 import accounts and 3300 users. Further, 

these accounts represent (1) about seven percent of the 1100 accounts expected

to be supported by the end of 2004 and (2) about 19 percent and 20 percent of 

annual import value and trade duties collected, respectively.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 77 GAO-04-719 Customs Modernization

68

Objective 3 Results

Observation 4

Release 1 and 2 defect density has stabilized, but nontrivial defects remain.

As indicated by the shaded area on the following graph representing the period

since Releases 1 and 2 began operating, the defect density associated with both 

releases has stabilized. In particular, CBP defect data show that as of October 17, 

2003, these releases had no severe or critical PTRs, and 44 moderate and 21

minor PTRs open. As of January 2004, these data show improvement, with 1 

severe, 34 moderate, and 17 minor PTRs open, and documentation shows that the 

severe defect was resolved on February 3, 2004.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 78 GAO-04-719 Customs Modernization

69

Objective 3 Results

Observation 3

Release 2 Unresolved Defects by Week

0

10

20

30

40

50

60

70

80

90

100

11/18/2002

12/2/2002

12/16/200212/30/2002

1/13/20031/27/20032/10/20032/24/20033/10/20033/24/2003

4/7/2003

4/21/2003

5/5/2003

5/19/2003

6/2/2003

6/16/20036/30/20037/14/20037/28/20038/11/20038/25/2003

9/8/2003

9/22/200310/6/2003

10/20/2003

11/3/2003

11/17/2003

12/1/2003

12/15/200312/29/2003

1/12/20041/26/2004

Severity 1 Severity 2 Severity 3

Planned (1/2003):

Actual:

SIT SAT UAT and Pilot Period

SIT SAT UAT and Pilot Period

10/17/2003

Number of PTRs

Weeks

Source: GAO based on CBP data
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 79 GAO-04-719 Customs Modernization

70

Objective 3 Results

Observation 4

Notwithstanding the stabilization of PTRs represented by these data, 34 moderate

PTRs remain open, and by definition these relate to instances where Releases 1 

and 2 do not work as specified or produce an error that degrades or impacts

system performance or user functionality. The only reason that they are not viewed 

as severe is that an acceptable (reasonable and effective) workaround is in place 

as a temporary fix, but such workarounds generally produce efficiency losses.

Release 1 and 2 mostly available when needed.

ACE is required to be available 99.9 percent of the time (24 hours a day, 7 days a 

week). Based on available eCP data, Releases 1 and 2 have met this requirement

on all but 4 days since October 17, 2003, when they began operating. These days, 

outages, and the respective availability levels are provided in the table. 

Date System availability Outage time

December 22, 2003 82.51 percent 4 hours, 12 minutes

January 12, 2003 99.65 percent 5 minutes

January 23, 2003 96.53 percent 50 minutes

January 26, 2003 97.15 percent 41 minutes

Source: eCP data.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 80 GAO-04-719 Customs Modernization

71

Objective 3 Results

Observation 4

For each of these outages, eCP documents a root cause analysis. Some of the 

outages can be attributed to problems such as an expired secure socket layer

certificate, full log file for servers, and loss of database connection between two 

subsystems. In addition to identifying the root cause, eCP reports on the mitigation

actions taken to prevent a reoccurrence of the problem. For example, to address 

the problem with the log file, eCP has opened a PTR for automated monitoring of 

file systems. Until the PTR is resolved, eCP reported that it is performing manual

monitoring of the log file systems daily. 
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 81 GAO-04-719 Customs Modernization

72

Objective 3 Results

Observation 5

Observation 5: Following Release 1 problems, steps were taken to avoid

future problems, but success of actions is unclear.

Effective project management, including directing a contractor to perform a 

particular task (e.g., developing and implementing a corrective action plan),

requires, among other things, a clear statement of objectives, identification of 

necessary resources, and defined measures of progress and results. 

As a result of the Release 1 cost overruns and schedule delays, CBP directed eCP, 

in February 2003, to develop and implement a plan to improve program

performance. According to the plan, which is dated April 2003, the objective was to 

demonstrate serious, tangible, and measurable actions to resolve core program

issues and achieve fundamental program improvement.

The plan identified eight problem areas that eCP was to correct:

• Ineffective leadership

• Unclear roles and responsibilities

• Misaligned priorities
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 82 GAO-04-719 Customs Modernization

73

Objective 3 Results

Observation 5

• Weak system engineering

• Poor program planning

• Inaccurate and inconsistent financial data

• Unmet contract commitments

• Unstable system performance

However, the plan did not define the level of resources required to execute it, and it 

did not specify how progress and improvement in each of these areas would be 

determined, including any specific measures and metrics.

In October 2003, program management review documents show that all the

corrective actions were reported to be completed. In January 2004, CBP conducted

an evaluation to assess eCP’s performance. Although the resulting January report 

was not available in time to be included in our review, CBP officials told us that 

implementation of eCP’s corrective action plan is complete. They provided the

following examples of corrective actions that eCP has taken to address each of the 

eight problem areas (see following slides).
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 83 GAO-04-719 Customs Modernization

74

Objective 3 Results

Observation 5

1 Enterprise resource planning is an integrated software solution used to manage a company’s resources.

Assessed each release to identify the desired business results that are being

addressed so that there is visibility into the benefits and capabilities that are being

gained through ACE.

Misaligned priorities

Business process owners now manage functional areas through the system life 

cycle. In the prior operating model, the business process teams would only be 

responsible for planning and defining the requirements, while the engineering teams

would be responsible for building, preparing, and delivering the support for accounts

management. In the new operating model, the business process owner would be

responsible for ensuring that accounts management in Release 1 and 2 is planned

and executed throughout all phases of the systems life cycle.

Unclear roles and

responsibilities

Increased the number and caliber of resources. For example, eCP has brought on

board an Enterprise Resource Planning1 expert, a chief technology officer, a systems

architect, and a workforce transformation leader.

Ineffective

leadership

Problem areas CBP-provided examples of eCP actions to correct problems
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 84 GAO-04-719 Customs Modernization

75

Objective 3 Results

Observation 5

Improved quality of deliverables with higher standards of acceptance evident with the

requirements, integrated baseline reviews, and preparation for proposals.

Improved collaboration between eCP and the government with active monitoring and

control.

Unmet contract 

commitments

Improved financial controls and financial reporting as a result of new tools and

automation, evident in the monthly cost performance reviews and program

management reviews.

Inaccurate and

inconsistent

financial data

Revised the program plan to include more detail on the content of the various

releases, descriptions of the end-state architecture, descriptions of testing and

development needs integrated with cost and schedule; and cost and schedule

estimates based on historical performance.

Poor program

planning

Improved testing and configuration management. For example, the environment

engineering group is now using a repeatable environment build process, and the

business process owners participate in the testing activities and provide an internal

check outside the software development team.

Improved quality of products as a result of using the Ascendant Life Cycle 

Methodology1 and tailoring it to the ACE program. For example, the deliverables now

require less rework than previously and are more aligned with CBP’s expectations.

Weak system

engineering

Problem areas CBP-provided examples of eCP actions to correct problems

1 The Ascendant Life Cycle methodology is a specific development methodology for use with the SAP Enterprise Resource

Planning software package.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 85 GAO-04-719 Customs Modernization

76

Objective 3 Results

Observation 5

Stable and controlled testing environments. For example, for Release 3, more

defects were identified in early testing activities, while in Release 2, the majority were

identified in later tests, which is an indication of infrastructure stability.

Unstable system

performance

Problem areas CBP-provided examples of eCP actions to correct problems

Source: GAO based on CBP data.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 86 GAO-04-719 Customs Modernization

77

Objective 3 Results

Observation 5

Despite the examples offered, program officials could not point to metrics that 

demonstrate improved performance. Instead they stated that the corrective actions 

have not been implemented long enough to see substantial improvements and that 

the corrective actions will require time to demonstrate results. They added,

however, that they have continued to see some improvement in the quality of 

deliverables.

More concrete measures of performance that are available do not suggest

adequate improvements. For example, as previously noted, Releases 3 and 4 have 

experienced 5 to 6 month delays and are over budget by $36.2 million. Program

officials attributed the problems that occurred to problems experienced in Release

1 affecting Releases 2, 3, and 4, before the corrective action plan was developed

and implemented. They also stated that contract task orders are generally

completed on time and under budget.

According to program officials, the effect of the corrective actions will only be

evident when eCP begins achieving milestones on schedule and within budget.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 87 GAO-04-719 Customs Modernization

78

Objective 3 Results

Observation 5

CBP expects these results to be seen with the delivery and approval of the Global 

Business Blueprint1 in May 2004, which is the first major activity to be initiated after 

the corrective actions were implemented.

Cost to develop and implement corrective action plan is unknown.

Effective project management, including directing a contractor to perform a 

particular task (e.g., developing and implementing a corrective action plan),

includes estimating the associated cost and monitoring the activity to ensure that 

cost estimates are met.

According to program officials, they do not know the costs that the government

incurred and the payments that the government made to eCP to develop and 

implement its plan for correcting problems associated with its delivery of proposed

system capabilities on time and within budget. Rather, they said that these costs

are spread across multiple contract tasks and are not identifiable.

Program officials told us that while CBP requested eCP to track the costs of

implementing the corrective actions, eCP representatives told them that it would be

impossible to track these costs separately because they would be incurred as part 

of other ongoing tasks that eCP is required to perform under executed task orders.

1 Global Business Blueprint is a planning project for future releases that is to establish an integrated business framework

and identify appropriate technology enablers to support ACE cargo management.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 88 GAO-04-719 Customs Modernization

79

Objective 3 Results

Observation 6

Observation 6: Initial releases represent a small fraction of the total ACE

software.

A system’s software size can be expressed in function points. Function points are 

standard units of measure that describe software in terms of end-user inputs and 

resulting outputs. They can be used to measure the size of any software regardless

of the language, development method, or platform. Function point counts are 

derived from a variety of sources, including requirements documentation,

interviews with end users, and business process descriptions. The International 

Function Point Users Group (IFPUG) maintains the standards for identifying and 

quantifying function points. The ACE system has been sized using function points 

with IFPUG-certified counters.

All ACE releases collectively consist of 11,936 function points. Releases 1 and 2 

together contain 618 function points, and Releases 3 and 4 together contain 1,016 

function points. The remaining 10,302 function points are contained in the three 

selectivity releases and Releases 5, 6, and 7, with the majority (8400) contained in 

Releases 6 and 7.

The following chart summarizes the function point counts for Releases 1 and 2, 

Releases 3 and 4, and Releases 5 through 7 plus Selectivity 1 through 3.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 89 GAO-04-719 Customs Modernization

80

Objective 3 Results

Observation 6

Size of ACE Software by Release Groupings

Releases 5, 6, and 7

Selectivity 1, 2, and 3

86 percent

Releases 3 and 4

9 percent

Releases 1 and 2

5 percent

Source: GAO based on CBP data.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 90 GAO-04-719 Customs Modernization

81

Objective 3 Results

Observation 6

Releases 1, 2, 3, and 4 reportedly account for 1634 function points being

addressed by custom software and COTS, including SAP. In contrast, Releases 5, 

6, and 7 plus Selectivity 1, 2, and 3 constitute 10,302 function points, 5450 of which 

are to be addressed by custom software and other COTS with the balance of 4852 

to be addressed by the SAP product. The graph on the following page provides this 

information by release.

Despite these relative sizes, the scheduled time for completing the last six releases

is about the same as the time anticipated to complete the first four releases, even 

though the software size of the last six is over three times that of the first four, and 

this does not include SAP software. Specifically, as of January 2004, the total time 

projected for design, development, and deployment of Releases 1 to 4 is 

approximately 3 years. According to program documentation, the total time 

projected for design, development, and deployment of Releases 5, 6, and 7 and 

Selectivity 1, 2, and 3 is approximately 3-1/2 years.

Program officials stated that they expect the Global Business Blueprint activity that 

they are currently performing to better identify the extent to which function points

can be addressed by custom software, other COTS, and SAP in Releases 5, 6, and 

7 and Selectivity 1, 2, and 3.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 91 GAO-04-719 Customs Modernization

82

Objective 3 Results

Observation 6

618

1016

486 612 510

1156 1132

250

44

566 694

2076

2776

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Releases

1-2

Releases

3-4

Release 5 Selectivity

1

Release 6 Selectivity

2

Release 7 Selectivity

3

Number of Function Points

SAP software

Other COTS

Custom software

Custom and

COTS

The following graph illustrates the amount of custom software, other COTS, and 

SAP software associated with each release:1

Source: GAO based on eCP data

1Releases 1 to 4 function points were not allocated between custom, other COTS, and SAP; however,

these releases consist of custom and COTS software, including SAP.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 92 GAO-04-719 Customs Modernization

83

Objective 3 Results

Observation 7

Observation 7: Reliability of key cost estimates in expenditure plan is unclear.

Cost estimates in expenditure plan varied from independent cost estimates.

CBP has established a process for developing cost estimates in expenditure plans. 

This process calls for estimates from both eCP and from an independent cost

estimating contractor. The process also calls for reconciling these estimates before 

cost estimates are incorporated into an expenditure plan.

In formulating the fiscal year 2004 ACE expenditure plan, CBP obtained eCP cost 

estimates in early October 2003, and most independent cost estimating contractor 

estimates in late October 2003. However, CBP used the eCP estimates in the 

expenditure plan without reconciling them to those of the independent cost estimator. 

According to program officials, the reconciliation step was not done because CBP 

did not have sufficient time to do so before completing the plan and submitting it for 

DHS review in mid-October 2003. However, CBP did task its support contractor with 

reconciling the estimates, including analyzing the reasons for differences. This

reconciliation was completed in mid-November 2003, about 2 months before the 

expenditure plan was submitted to the appropriations committees. The net difference 

between eCP’s estimates in the plan and the independent estimates is $3.4 million.

However, the sum of the differences for individual expenditure plan elements is 

$39.8 million, as shown on the following table.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 93 GAO-04-719 Customs Modernization

84

Objective 3 Results

Observation 7

Comparison of expenditure plan cost estimates to independent estimates

 Difference between

Expenditure Plan Cost

Estimates and Independent

Cost Estimates

Element1

Net Total Explanation of difference

Release 6 definition and

design

0.4 0.4 eCP used a broader interpretation of the Release 6 scope

than the independent estimator.

Selectivity 2 definition

and design

2.7 2.7 eCP used a 21 percent higher software size estimate and 9 

percent higher assumed labor rate than the independent

estimator.

Implementation

infrastructure and support

0 0 N/A

Foundation program

management

–4.4 4.4 Cost data reflecting previous experience with this activity were

not incorporated into the independent estimate.

Foundation architecture

and engineering

7.8 7.8 Scope was not defined, and the movement of 10 to 13

positions to this activity was not identified in the independent

estimate.

Workforce transformation

and training

10.7 10.7 eCP used a broader view of the scope of the workforce

transformation task statement than the independent estimator.

Operations and 

maintenance

–13.8 13.8 Independent cost estimate used a broader view of hardware

and software maintenance costs.

Total 3.4 39.8

Source: GAO analysis based on CBP data.

1

Additional expenditure plan elements (i.e., CBPMO, Federally Funded Research and Development Center, International Trade Data System, SAP, and

management reserve) were not estimated by the independent estimator and thus not included in this analysis.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 94 GAO-04-719 Customs Modernization

85

Objective 3 Results

Observation 7

Expenditure plan cost estimates partially satisfy relevant guidance.

One indicator of the reliability of cost estimates is the effectiveness of processes

used to derive the estimates. SEI has defined guidelines for assessing such

processes and determining the reliability of estimates.1

To determine the reliability of the cost estimates in the expenditure plan, the

support contractor and eCP analyzed the estimates against SEI guidance. The cost 

estimates analyzed pertained to the following plan elements: Release 6 definition

and design, Selectivity 2 definition and design, foundation program management,

architecture and engineering, workforce transformation and training, and operations

and maintenance. The analysis shows that the expenditure plan estimates partially

satisfy the SEI’s guidance. The results of this analysis follow.

1 SEI’s institutional estimating guidelines are defined in Checklists and Criteria for Evaluating the Cost and Schedule

Estimating Capabilities of Software Organizations, and project-specific estimating guidelines are defined in A Manager’s

Checklist for Validating Software Cost and Schedule Estimates.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 95 GAO-04-719 Customs Modernization

86

Objective 3 Results

Observation 7

SEI guidance Satisfied Assessment of criteria satisfaction

1. Are the objectives of the

estimate clear and correct?

Yesa

According to the support contractor, the objective of the

estimate was clearly stated. Also, the tasks and activities to be

included in, and excluded from, the estimate were clearly

identified.

2. Has the task been

appropriately sized?

Partiallyb

According to the support contractor, software sizing estimates

were determined by requirements and function point count

performed by certified staff.However, the process for estimating

the size of the task was not adequately documented.

3. Are the estimated cost and

schedule consistent with

demonstrated

accomplishments on other

projects?

Partially According to the support contractor, the cost and schedule

models used to develop the estimate were calibrated to eCP’s

historical data on other programs and demonstrated experience

from release 2. However, eCP has not demonstrated how its

estimating models take into account demonstrated performance

across different system development methodologies (e.g.,

custom development versus SAP implementation).

4. Have the factors that affect

the estimate been identified

and explained?

Partially According to the support contractor, parameter values and

planning assumptions were documented. However,

uncertainties in parameter values were not identified.

Source: GAO.

a

Satisfies every aspect of the condition that we reviewed.

b

Satisfies many, but not all, key aspects of the condition that we reviewed.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 96 GAO-04-719 Customs Modernization

87

Objective 3 Results

Observation 7

SEI guidance Satisfied Assessment of criteria satisfaction

5. Have steps been taken

to ensure the integrity

of the estimating

process?

Partially According to the support contractor, eCP set up a planning team

that includes certified analysts and experienced estimators.

However, eCP’s schedule estimating process was not

adequately documented, the cost models do not support

schedules with the extensive level of parallel development

activities that are necessary to achieve the proposed schedule,

and the shift of requirements from Release 2 to future releases

may not be accurately reflected in the estimate.

6. Is the organization’s

historical evidence

capable of supporting

a reliable estimate?

Partially According to the support contractor, eCP draws from a database

that contains productivity data and supports cost percentages

from a large set of completed projects. However, cost models did

not have consistent methods for recording historical data, and

eCP has not demonstrated how it measures and improves its

software processes based on historical evidence.

7. Has the situation

changed since the

estimate was

prepared?

Yes According to the support contractor, program changes (e.g.,

schedule changes) were incorporated in estimates. Additionally,

the estimate was used as the basis for assigning resources,

deploying relevant schedules, and making commitments.

Source: GAO.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 97 GAO-04-719 Customs Modernization

88

Conclusions

The fiscal year 2004 ACE expenditure plan, along with program documentation and

officials’ commitments, largely satisfies the legislative conditions imposed by

Congress. Further, many of the recommendations that we have previously made to 

strengthen ACE management have been addressed, and CBP leadership has

committed to addressing the remaining recommendations. However, much remains 

to be done before these open recommendations are fully implemented. In 

particular, progress on overcoming human capital challenges has been slow.

To CBP’s credit, the first two ACE releases are operating largely as intended.

Achieving this initial operating capability, however, has not occurred without some 

difficulty, such as borrowing resources from future releases to overcome problems

on near-term releases; similar difficulties that could impact cost, schedule, and 

capability commitments are being experienced on ongoing releases. Prospectively, 

this pattern is not likely to change unless the degree of concurrency among

activities within and between releases is better controlled, and the underlying

reasons for introducing this concurrency are addressed. While time and resources

were invested in management improvements over a year ago to address the root 

causes, the absence of meaningful measures for determining whether these

investments will be successful, coupled with limited progress on some of our 

previous recommendations, makes the chances of CBP delivering future ACE 

release capabilities on time and within budget uncertain at best.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 98 GAO-04-719 Customs Modernization

89

Recommendations

To assist CBP in managing ACE and increasing the chances that future releases

will deliver promised capabilities on time and within budget, we recommend that the 

DHS Secretary, through the Under Secretary for Border and Transportation

Security and the CBP Commissioner, direct the CBP CIO to 

• ensure the independence of its IV&V agent;

• ensure that future expenditure plans are based on cost estimates that are

reconciled with independent cost estimates; 

• reconsider the ACE acquisition schedule and cost estimates in light of early

release problems, including these early releases’ cascading effects on future 

releases and their relatively small size compared to later releases, and in light

of the need to avoid the past levels of concurrency among activities within and 

between releases;

• define measures, and collect and use associated metrics, for determining

whether prior and future program management improvements are successful; 

and

• report quarterly to the House and Senate Appropriations Committees on 

efforts to address the above, as well as our previously made but still open

recommendations.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 99 GAO-04-719 Customs Modernization

90

Agency Comments

We provided this briefing to and discussed its contents with DHS, CBP, and ACE 

program officials. These officials, including the DHS Chief Information Officer and 

the CBP Acting Chief Information Officer, generally agreed with our findings,

conclusions, and recommendations, and stated that the briefing was fair and

balanced. They also provided some technical comments, which we have

incorporated into the briefing, as appropriate.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 100 GAO-04-719 Customs Modernization

91

Attachment 1

Scope and Methodology

Scope and Methodology

To accomplish our objectives, we analyzed the ACE fiscal year 2004 expenditure

plan and supporting documentation, comparing them to relevant federal

requirements and guidance and to applicable best practices; and we interviewed

DHS and CBP officials and ACE program contractors. In particular, we reviewed

• DHS and CBP investment management practices, using OMB A-11, part 7;

• DHS and CBP activities for assessing ACE compliance with the DHS

enterprise architecture;

• CBP acquisition management practices, using the Software Engineering

Institute’s (SEI) Software Acquisition Capability Maturity Model (SA-CMM);1

1 The Software Acquisition Capability Maturity Model® developed by Carnegie Mellon University’s Software Engineering

Institute (SEI) defines five levels of maturity that provide a roadmap for continuously improving an organization’s acquisition

management processes.
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 101 GAO-04-719 Customs Modernization

92

Attachment 1

Scope and Methodology

• CBP cost estimating program, using SEI’s institutional and project-specific

estimating guidelines;1

• CBP human capital management activities, against its human capital 

management strategy;

• ACE testing activities, system defect data, and system performance data 

using industry best practices;2

• independent verification and validation (IV&V) activities using the Institute of 

Electrical and Electronics Engineers Standard for Software Verification and

Validation;3

1 SEI’s institutional estimating guidelines are defined in Checklists and Criteria for Evaluating the Cost and Schedule

Estimating Capabilities of Software Organizations, and SEI’s project-specific estimating guidelines are defined in A Manager’s

Checklist for Validating Software Cost and Schedule Estimates.

2 U.S. General Accounting Office, Year 2000 Computing Crisis: A Testing Guide, GAO/AIMD-10.1.21 (Washington, D.C.:

November 1998).

3Institute of Electrical and Electronics Engineers (IEEE) Standard for Software Verification and Validation, IEEE Std 1012-

1998 (New York: Mar. 9, 1998).
Appendix I

IT: Early Releases of Customs Trade System 

Operating, but Pattern of Cost and Schedule 

Problems Needs to Be Addressed

Page 102 GAO-04-719 Customs Modernization

93

Attachment 1

Scope and Methodology

• cost and schedule data, and functional requirements from program

management documentation; and

• CBP and its contractor activities for improving the ACE program with program

improvement action plans.

We did not independently verify DHS-, CBP-, and contractor-provided data or

calculations nor do we attest to their accuracy.

For DHS-, CBP-, and contractor-provided data that we did not substantiate, we 

have made appropriate attribution indicating the data's sources.

We conducted our work at CBP headquarters and contractor facilities in the 

Washington, D.C., metropolitan area from January through March 2004 in 

accordance with generally accepted government auditing standards.
Page 103 GAO-04-719 Customs Modernization

Appendix II

Comments from the U.S. Department of 

Homeland Security ApendxiI
Appendix II

Comments from the U.S. Department of 

Homeland Security

Page 104 GAO-04-719 Customs Modernization
Appendix II

Comments from the U.S. Department of 

Homeland Security

Page 105 GAO-04-719 Customs Modernization
Appendix II

Comments from the U.S. Department of 

Homeland Security

Page 106 GAO-04-719 Customs Modernization
Page 107 GAO-04-719 Customs Modernization

Appendix III

GAO Contact and Staff Acknowledgments ApendxiI

GAO Contact Mark Bird, (202) 512-6260

Staff 

Acknowledgments

In addition to the person named above, Barbara Collier, Bill Cook, Neil 

Doherty, Scott Farrow, Michael P. Fruitman, Tamra Goldstein, Neela 

Lakhmani, and Freda Paintsil made key contributions to this report. 

(310282)
GAO’s Mission The General Accounting Office, the audit, evaluation and investigative arm of 

Congress, exists to support Congress in meeting its constitutional responsibilities 

and to help improve the performance and accountability of the federal government 

for the American people. GAO examines the use of public funds; evaluates federal 

programs and policies; and provides analyses, recommendations, and other 

assistance to help Congress make informed oversight, policy, and funding 

decisions. GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

Obtaining Copies of 

GAO Reports and 

Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and fulltext files of current reports and testimony and an expanding archive of older 

products. The Web site features a search engine to help you locate documents 

using key words and phrases. You can print these documents in their entirety, 

including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and 

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site 

daily. The list contains links to the full-text document files. To have GAO e-mail this 

list to you every afternoon, go to www.gao.gov and select “Subscribe to 

e-mail alerts” under the “Order GAO Products” heading.

Order by Mail or Phone The first copy of each printed report is free. Additional copies are $2 each. A check 

or money order should be made out to the Superintendent of Documents. GAO 

also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single 

address are discounted 25 percent. Orders should be sent to:

U.S. General Accounting Office

441 G Street NW, Room LM

Washington, D.C. 20548

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Public Affairs Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800

U.S. General Accounting Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Service Requested

Presorted Standard

Postage & Fees Paid

GAO

Permit No. GI00