United States Government Accountability Office

GAO Report to Congressional Committees

INFORMATION 

TECHNOLOGY 

Customs Has Made 

Progress on 

Automated 

Commercial 

Environment System, 

but It Faces LongStanding Management 

Challenges and New 

Risks 

May 2006 

GAO-06-580 
What GAO Found

United States Government Accountability Office

Why GAO Did This Study

Highlights

Accountability Integrity Reliability

May 2006

INFORMATION TECHNOLOGY 

Customs Has Made Progress on 

Automated Commercial Environment 

System, but It Faces Long-Standing 

Management Challenges and New Risks 

Highlights of GAO-06-580, a report to 

Subcommittees on Homeland Security, 

Senate and House Committees on

Appropriations 

The Department of Homeland

Security (DHS) is conducting a 

multiyear, multibillion-dollar 

acquisition of a new trade 

processing system, planned to

support the movement of legitimate 

imports and exports and strengthen 

border security. By congressional 

mandate, plans for expenditure of

appropriated funds on this system, 

the Automated Commercial

Environment (ACE), must meet

certain conditions, including GAO

review. This study addresses 

whether the fiscal year 2006 plan

satisfies these conditions; it also 

describes the status of DHS’s 

efforts to implement prior GAO

recommendations for improving 

ACE management, and provides 

observations about the plan and 

DHS’s management of the program. 

What GAO Recommends

To help ensure the success of ACE, 

GAO recommends, among other 

things, that DHS develop realistic 

ACE performance measures and 

targets; align these with ACE 

program goals, benefits, and

desired business outcomes; and 

minimize concurrent development 

of ACE releases. DHS agreed with 

GAO’s recommendations and 

described actions that it has under

way and planned to address them. 

The fiscal year 2006 ACE expenditure plan, including related program 

documentation and program officials’ statements, either satisfied or partially

satisfied the legislative conditions imposed by the Congress; however, more 

can be done to better address several aspects of these conditions. In 

addition, DHS has addressed some recommendations that GAO has 

previously made, but progress has been slow in addressing several 

recommendations aimed at strengthening ACE management. For example, 

DHS has more to do to implement the recommendation that it establish an 

ACE accountability framework that, among other things, ensures that 

expenditure plans report progress against commitments made in prior plans. 

Implementing a performance and accountability framework is important for 

ensuring that promised capabilities and benefits are delivered on time and 

within budget. In addition, describing progress against past commitments is

essential to permit meaningful congressional oversight. 

Among GAO’s observations about the ACE program and its management are 

several related to the need to effectively set and use performance goals and 

measures. Although the program set performance goals, these targets were 

not always realistic. For example, in fiscal year 2005, the program set a 

target that 11 percent of all Customs and Border Protection (CBP) 

employees would use ACE. However, this target does not reflect the fact that 

many CBP employees will never need to use the system. Additionally, the 

program has established 6 program goals, 11 business results, 23 benefits, 

and 17 performance measures, but the relationships among these are not 

fully defined or adequately aligned with each other. For example, not every 

goal has defined benefits, and not every benefit has an associated

performance measure. Without realistic ACE performance measures and 

targets that are aligned with the overall program goals and desired results, 

DHS will be challenged in its efforts to establish an accountability 

framework for ACE that will help to ensure that the program delivers its 

expected benefits. 

In addition, DHS plans to develop several increments, referred to as 

“releases,” concurrently; in the past, such concurrency has led to cost 

overruns and schedule delays because releases contended for the same 

resources, and resources that were to be used on later releases were 

diverted to earlier ones. However, because of DHS’s belief that such 

concurrent development will allow it to deliver ACE functionality sooner, it 

is reintroducing the same problems that resulted in past shortfalls. 

www.gao.gov/cgi-bin/getrpt?GAO-06-580.

To view the full product, including the scope 

and methodology, click on the link above. 

For more information, contact Randolph C. 

Hite at (202) 512-3429 or hiter@gao.gov. 
Contents 

Letter 1

Compliance with Legislative Conditions 3

Status of Our Open Recommendations 3 

Observations about ACE Management 7 

Conclusions 9

Recommendations for Executive Action 11 

Agency Comments 12 

Appendix I Briefing to Subcommittees on Homeland Security, 

House and Senate Committees on Appropriations 13 

Appendix II Comments from the Department of Homeland 

Security 143 

Appendix III Contact and Staff Acknowledgments 150 

Page i GAO-06-580 Customs Modernization 
Abbreviations 

ACE Automated Commercial Environment 

CBP Customs and Border Protection 

CBPMO Customs and Border Protection Modernization Office 

CIO chief information officer

CMU Carnegie Mellon University 

DHS Department of Homeland Security 

EA enterprise architecture 

EVM earned value management 

IT information technology 

IV&V independent verification and validation 

OMB Office of Management and Budget 

ORR operational readiness review 

SA-CMM® Software Acquisition Capability Maturity Model 

SEI Software Engineering Institute 

US-VISIT United States Visitor and Immigrant Status Indicator 

Technology 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately. 

Page ii GAO-06-580 Customs Modernization 
United States Government Accountability Office

Washington, DC 20548 

May 31, 2006

The Honorable Judd Gregg 

Chairman 

The Honorable Robert C. Byrd 

Ranking Minority Member 

Subcommittee on Homeland Security 

Committee on Appropriations 

United States Senate 

The Honorable Harold Rogers 

Chairman 

The Honorable Martin Olav Sabo 

Ranking Minority Member 

Subcommittee on Homeland Security 

Committee on Appropriations 

House of Representatives 

In February 2006, U.S. Customs and Border Protection (CBP), within the 

Department of Homeland Security (DHS), submitted to the Congress its 

fiscal year 2006 expenditure plan for the Automated Commercial 

Environment (ACE) program. ACE is to be CBP’s new import and export

processing system. The program’s goals include 

• supporting border security by enhancing analysis and information sharing 

with other government agencies and

• streamlining time-consuming and labor-intensive tasks for CBP personnel 

and the trade community through the provision of a single Web-based 

interface. 

DHS currently plans to acquire and deploy ACE in 11 increments, referred 

to as “releases,” in approximately 8.5 years. The first three releases are 

deployed and operating, and the fourth release is being deployed. Other 

releases are in various states of definition and development. For the ACE 

life-cycle, the risk-adjusted cost estimate is about $2.8 billion; through 

fiscal year 2005, ACE-appropriated funding has been about $1.6 billion. 

Page 1 GAO-06-580 Customs Modernization
The Department of Homeland Security Appropriations Act of 2006 1

appropriates about $320 million for ACE. The act states that DHS may not 

obligate any funds for ACE until it submits for approval to the House and 

Senate Committees on Appropriations a plan for expenditure that 

1. meets the capital planning and investment control review requirements 

established by the Office of Management and Budget (OMB), including 

Circular A-11, part 7;2

2. complies with DHS’s enterprise architecture; 

3. complies with the acquisition rules, requirements, guidelines, and 

systems-acquisition management practices of the federal government; 

4. includes a certification by the DHS Chief Information Officer that an 

independent verification and validation agent is currently under 

contract; 

5. is reviewed and approved by the DHS Investment Review Board 

(IRB),3

 the Secretary of Homeland Security, and OMB; and 

6. is reviewed by GAO. 

As required by the act, we reviewed the ACE fiscal year 2006 expenditure 

plan. Our objectives were to (1) determine whether the expenditure plan 

satisfies certain legislative conditions, (2) determine the status of our open 

ACE recommendations, and (3) provide any other observations about the 

expenditure plan and DHS’s management of the ACE program. 

On March 10, 2006, we briefed your offices on the results of this review. 

This report transmits the results of our work. The full briefing, including

our scope and methodology, can be found in appendix I. 

 1

Pub. L. 109-90 (Oct. 18, 2005). 

2

OMB Circular A-11 establishes policy for planning, budgeting, acquisition, and 

management of federal capital assets. 

3

The purpose of the Investment Review Board is to integrate capital planning and 

investment control, along with the budgeting, acquisition, and management of investments.

It is also to ensure that spending on investments directly supports and furthers the mission, 

and that this spending provides optimal benefits and capabilities to stakeholders and 

customers. 

Page 2 GAO-06-580 Customs Modernization
Compliance with 

Legislative Conditions 

The legislative conditions that the Congress placed on the use of fiscal 

year 2006 ACE appropriated funds have been either partially or fully 

satisfied by the latest expenditure plan and related program 

documentation and activities. However, more can be done to better 

address several aspects of these conditions. For example: 

• One legislative condition states that the plan should meet OMB’s capital 

planning and investment control review requirements, which include 

addressing security and privacy issues. However, a privacy impact 

assessment4

 for ACE has been in draft for several months and is not yet 

approved. Another capital planning and investment control review 

requirement is that performance goals and measures be provided in the 

business case for ACE. Although CBP describes selected performance 

goals and measures, the goals (i.e., targets) are not always realistic (we 

provide further discussion of this issue later in this report). 

• According to another legislative condition, the expenditure plan must 

comply with DHS’s enterprise architecture. However, DHS does not have a 

documented methodology for evaluating programs for compliance with its 

enterprise architecture, other than relying on the professional expertise of 

its staff. 

• According to a third legislative condition, the DHS Chief Information 

Officer is to certify that an independent verification and validation (IV&V) 

agent is under contract. Although DHS satisfied this condition, the scope 

of the IV&V contractor’s activities is not consistent with the operative 

industry standard, which states that IV&V should extend to key system 

products and development processes.5

CBP has addressed some recommendations, while progress has been slow 

on others. Each recommendation, along with the status of actions to 

address it, is summarized below. 

Status of Our Open 

Recommendations 

• Ensure that future expenditure plans are based on cost estimates that are 

reconciled with independent cost estimates. 

 4

The purpose of a privacy impact assessment is to ensure that there is no collection, 

storage, access, use, or dissemination of identifiable personal or business information that 

is not both needed and permitted. 

5

Institute of Electrical and Electronics Engineers Computer Society, Standard for Software 

Verification and Validation 1012-1998 (June 8, 2005). 

Page 3 GAO-06-580 Customs Modernization
Complete.

6

 In October 2005, CBP, with contractor support, compared the 

program plan cost estimate with the independent cost estimate. According 

to the analysis performed, the two estimates are consistent. 

• Develop and implement a rigorous and analytically verifiable cost 

estimating program that embodies the tenets of effective estimating, as 

defined in the institutional and project-specific estimating models 

developed by the Software Engineering Institute (SEI).7

In progress. CBP has taken steps such as (1) hiring a contractor to 

develop cost estimates (including contract task order cost estimates) that 

are independent of CBP’s estimates, and (2) tasking a support contractor 

with evaluating both the independent and CBP estimates against the 

criteria defined by SEI. According to the results of the support 

contractor’s evaluation, the independent estimates satisfied the SEI 

criteria; CBP’s estimates largely satisfied the criteria. 

However, according to the support contractor, CBP’s cost estimating had 

limitations. First, the CBP estimate did not adequately consider past 

projects in its cost and schedule estimates. In addition, the CBP estimate 

was an aggregation of estimates developed separately for three ACE 

components, each according to a different cost estimating methodology; 

the support contractor advised against this approach, recommending that 

component estimates be based on the same methodology. 

• Immediately develop and implement a human capital management strategy 

that provides both near- and long-term solutions to program office human 

capital capacity limitations, and report quarterly to the Appropriations 

Committees on the progress of efforts to do so. 

In progress. CBP has expanded its contractor and government workforce 

dedicated to the ACE program by merging staff assigned to trade-related 

legacy systems with the ACE program staff. In addition, it is beginning to

use subject matter experts from existing field operations advisory boards 

 6

CBP’s implementation of this recommendation is complete with respect to the fiscal year 

2006 expenditure plan. 

7

SEI’s institutional and project-specific estimating guidelines are defined respectively in 

Robert E. Park, Checklists and Criteria for Evaluating the Cost and Schedule Estimating

Capabilities of Software Organizations, CMU/SEI-95-SR-005, and A Manager’s Checklist 

for Validating Software Cost and Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburgh,

Pa.: Carnegie Mellon University Software Engineering Institute, 1995). 

Page 4 GAO-06-580 Customs Modernization
to help program officials define requirements for future releases. However, 

it does not have a documented human capital strategy covering its ACE 

program. 

• Have future ACE expenditure plans specifically address any proposals or 

plans, whether tentative or approved, for extending and using ACE 

infrastructure to support other homeland security applications, including 

any impact on ACE of such proposals and plans.

In progress. The expenditure plan describes steps both planned and under 

way to ensure that ACE infrastructure supports both ACE and other 

homeland security applications. For example, it states that both ACE and 

the United States Visitor and Immigrant Status Indicator Technology8

 (USVISIT) program should conform to the DHS enterprise architecture, which

is to define standard shared services that the two systems can request. 

Such a services oriented architecture is intended to promote reuse, as well 

as reducing overlap and duplication.

• Define measures, and collect and use associated metrics, for determining 

whether prior and future program management improvements are 

successful. 

In progress. CBP continues to make changes that are intended to improve 

overall program management, but it has not consistently defined measures 

to determine whether the changes are successful. For example, CBP has 

reorganized its Office of Information Technology; this reorganization is 

intended to improve program management by providing (1) enhanced 

government oversight of ACE development, (2) better definition of 

requirements for future ACE releases, and (3) faster and cheaper delivery 

of ACE capabilities. However, program officials told us that they have not

established measures or targets for determining whether the 

reorganization is providing these benefits. 

• Define and implement an ACE accountability framework that fulfills 

several conditions: 

1. The framework should cover all program commitment areas, including 

key expected or estimated system (a) capabilities, use, and quality; 

 8

US-VISIT is a governmentwide program to collect, maintain, and share information on

foreign nationals for enhancing national security and facilitating legitimate trade and travel, 

while adhering to U.S. privacy laws and policies.

Page 5 GAO-06-580 Customs Modernization
(b) benefits and mission value; (c) costs; and (d) milestones and 

schedules. 

In progress. CBP has prepared an initial version of an accountability 

framework that it intends to improve as it proceeds. The framework is 

built around measuring progress against costs, milestones, schedules, 

and risks for select releases; however, the benefit measurement has 

not been well defined, and the performance targets are not always 

realistic. 

2. The framework should ensure currency, relevance, and completeness 

of all program commitments made to the Congress in expenditure 

plans. 

In progress. The fiscal year 2006 expenditure plan includes inaccurate, 

dated, and incomplete information and omits other relevant

information. For example, the plan did not include information 

regarding CBP’s decision to eliminate the dependencies among the 

screening and targeting releases and the cargo releases, and to take 

advantage of the capabilities of its existing Automated Targeting 

System.9

3. The framework should ensure reliable data that are relevant to

measuring progress against commitments. 

In progress. The data that CBP uses to measure progress against 

commitments are not consistently reliable. For example, data in the 

defect tracking system show that defects in Release 4 (which is now 

operational) have not been closed; however, program officials told us 

that many of these defects have been resolved. 

4. The framework should ensure that future expenditure plans report 

progress against commitments contained in prior expenditure plans. 

In progress. The current expenditure plan does not adequately 

describe progress against commitments made in previous plans. For 

example, the plan provides a summary of the funding requested in 

each of the previous six expenditure plans, but it does not provide 

information on whether these funding amounts were actually 

expended or obligated as planned. 

 9

The Automated Targeting System is a DHS system that targets containers for inspection 

based on a perceived level of risk. 

Page 6 GAO-06-580 Customs Modernization
5. The framework should ensure that criteria for exiting key readiness 

milestones adequately consider indicators of system maturity, such as 

the severity of open defects. 

In progress. ACE milestone exit criteria provide for addressing the risk 

associated with severe defects that are unresolved. Using these 

criteria, CBP passed several release milestones with severe defects 

still open. However, CBP officials were unable to provide us with any 

documentation on how they assessed the inherent risks of passing 

these milestones with open severe defects. 

6. The framework should ensure clear and unambiguous delineation of 

the respective roles and responsibilities of the government and the 

prime contractor. 

Complete. The current ACE program plan describes general roles and 

responsibilities for the government and the prime contractor. More 

detailed roles have been documented in a roles and responsibilities 

matrix that assigns primary responsibility for each activity. 

• Report quarterly to the House and Senate Appropriations Committees on

efforts to address our open recommendations. 

In progress. CBP submitted quarterly reports to both Committees on its 

efforts to address our open recommendations; however, progress in 

addressing our recommendations was not always reported accurately. 

We have several observations about the development of ACE releases, as 

well as several more concerning the performance of ACE releases that are 

deployed and operating.

Observations about 

ACE Management 

• ACE development: Steps have been taken to address a past pattern of 

ACE release shortfalls, but new release management weaknesses are 

emerging.

As we have previously observed, CBP established a pattern of borrowing 

resources from future releases to address problems with the quality of 

earlier releases; this led to schedule delays and cost overruns. This pattern

has continued with the most recently deployed cargo release, which 

developed problems that caused delays with a subsequent screening and 

Page 7 GAO-06-580 Customs Modernization
targeting release.10 CBP took steps to mitigate this problem by eliminating 

the dependencies between the cargo releases and the screening and 

targeting releases. 

However, CBP’s planned schedule for developing additional releases 

includes a significant level of concurrence, because of CBP’s interest in 

delivering ACE functionality sooner. Such concurrence between ACE 

release activities has led to cost overruns and schedule delays in the past. 

Thus, the revised ACE plans and actions are potentially reintroducing the 

same problems that produced past shortfalls. 

We made several specific observations related to these weaknesses, 

including the following:

• On two recent releases, key milestones were passed despite unresolved 

severe defects. Officials told us that the risk of proceeding did not 

outweigh the need to get the releases to users, and thereby gaining user 

acceptance and feedback. However, the risks were not documented 

and formally managed. 

• Concurrence in developing early ACE releases caused schedule slips 

and cost overruns. Despite these experiences, CBP has established a 

risky plan that involves considerable overlap across the development 

schedules for three future releases. 

• Although the use of earned value management (EVM) is an OMB 

requirement, it was not being used to manage the development of two 

recent releases.11 For example, CBP discontinued use of EVM on one 

release because this method was not familiar to staff who were 

transferred to work on the program. 

 10Screening is the method of determining high-risk people or shipments before their arrival 

at a port. Targeting is the risk-based determination of whether a shipment should undergo 

additional documentary review or physical inspection. 

11Earned value management is a tool for measuring program progress by comparing the 

value of work accomplished during a given period with that of the work expected in that

period; this comparison permits performance to be evaluated based on calculated 

variances from the planned cost and schedule.

Page 8 GAO-06-580 Customs Modernization
• ACE operations: Operational performance has been mixed, and mission

impact is unclear. 

ACE releases one through four are in operation. To date, these releases’ 

operational performance has been uneven. For example, ACE has largely 

been meeting its goals for being available and responsive in processing 

virtually all daily transactions, and has decreased truck processing times 

at some ports. However, ACE is not being used by as many CBP and trade 

personnel as was expected, and truck processing times at other ports have 

increased. Moreover, overall user satisfaction has been low. 

In addition, ACE goals, expected mission benefits, and performance 

measures are not fully defined and adequately aligned with each other. For 

example, not every goal has defined benefits, every benefit is defined only

in terms of efficiency gains, not every benefit has an associated business 

result, and not every benefit and business result has associated

performance measures. 

Further, where performance measures have been defined, the associated

targets are not always realistic. For example, the performance target in

fiscal year 2005 for ACE usage was that 11 percent of all CBP employees 

would use ACE. However, that many CBP employees will never need to 

use the system. This performance target does not reflect that. 

Because performance measures are not always realistic or aligned with 

program goals and benefits, it is unclear whether ACE has realized—or 

will realize—the mission value that it was intended to bring to CBP’s and 

other agencies’ trade- and border security-related operations.

The legislative conditions that the Congress placed on the use of fiscal 

year 2006 ACE appropriated funds have been either partially or fully 

satisfied by the latest expenditure plan and related program 

documentation and activities. Nevertheless, more can be done to better 

address several aspects of these conditions, such as ensuring that the 

program’s privacy impact assessment is approved, measuring ACE 

performance and results, ensuring architectural alignment, and employing

effective IV&V practices. Given that the legislative conditions are 

collectively intended to promote accountability and increase the chances 

of program success, it is important that each receives DHS’s full attention.

Also important to ACE’s success is the swift and complete implementation 

of the recommendations that we have previously made to complement the 

Conclusions 

Page 9 GAO-06-580 Customs Modernization
legislative conditions and improve program management, performance, 

and accountability. In this regard, some recommendations have been 

addressed, while progress has been slow on others, such as 

• accurately reporting to the Appropriations Committees on CBP’s progress 

in implementing our prior recommendations; 

• developing and implementing a strategic approach to meeting the 

program’s human capital needs; 

• using criteria for exiting key milestones that adequately consider 

indicators of system maturity, such as severity of open defects and the 

associated risks; and 

• developing and implementing a performance and accountability 

framework for ensuring that promised capabilities and benefits are 

delivered on time and within budget.

To its credit, CBP has taken several steps to stem the pattern of cost, 

schedule, and performance shortfalls that it experienced on early ACE 

releases. However, future releases are unlikely to realize the impact of 

these steps because revised ACE plans and actions are reintroducing the 

same pattern that led to early release shortfalls. This pattern includes not 

formally and transparently considering, and proactively addressing, the 

risks associated with passing key release milestones with known severe 

defects; building considerable overlap and concurrence in the

development schedules of releases that will contend for the same 

resources; and not performing EVM on all releases. If this pattern 

continues, the prospects for a successful program will be diminished. 

Although availability and responsiveness targets are largely being met and 

long-standing help desk limitations are being addressed, the prospects for 

a successful program nevertheless remain unclear. The true measure of 

ACE’s success is arguably the mission value that it brings to CBP’s and 

other agencies’ trade- and border security-related operations and users. 

Such value depends both on the operational performance of ACE and on 

CBP’s ability to demonstrate that this performance is achieving program 

goals, delivering expected benefits, and producing desired business 

results. At this juncture, however, neither the system’s performance nor its 

value is clear because of several factors: the operational performance of 

deployed releases has been mixed; users’ satisfaction has been low; the 

relationships among goals, benefits, and desired business outcomes are 

Page 10 GAO-06-580 Customs Modernization
not evident; and the range of measures needed to create a complete and 

realistic picture of ACE’s performance is missing.

In summary, a number of ACE activities have been and are being done 

well; these have contributed to the program’s progress to date and will go 

a long way in determining the program’s ultimate success. However, it will 

be important for CBP to effectively address long-standing ACE 

management challenges along with emerging problems. Until it does so, 

ACE will remain a risky program. 

To assist CBP in managing ACE—and increasing the chances that it will 

deliver required capabilities on time and within budget, demonstrating 

promised mission benefits and results—we recommend that the Secretary 

of Homeland Security direct the appropriate departmental officials to fully 

address those legislative conditions associated with having an approved 

privacy impact assessment and ensuring architectural alignment. 

We also recommend that the Secretary, through CBP’s Acting 

Commissioner, direct the Assistant Commissioner for Information and 

Technology to 

Recommendations for 

Executive Action 

• fully address those legislative conditions associated with measuring ACE 

performance and results and employing effective IV&V practices; 

• accurately report to the appropriations committees on CBP’s progress in 

implementing our prior recommendations; 

• include in the June 30, 2006, quarterly update report to the appropriations 

committees a strategy for managing ACE human capital needs and the 

ACE framework for managing performance and ensuring accountability;

• document key milestone decisions in a way that reflects the risks 

associated with proceeding with unresolved severe defects and provides 

for mitigating these risks; 

• minimize the degree of overlap and concurrence across ongoing and 

future ACE releases, and capture and mitigate the associated risks of any 

residual concurrence; 

• use EVM in the development of all existing and future releases; 

Page 11 GAO-06-580 Customs Modernization
• develop the range of realistic ACE performance measures and targets 

needed to support an outcome-based, results-oriented accountability 

framework, including user satisfaction with ACE; and 

• explicitly align ACE program goals, benefits, desired business outcomes, 

and performance measures. 

In written comments on a draft of this report signed by the Director, 

Departmental GAO/OIG Liaison, DHS agreed with our findings concerning 

progress in addressing our prior recommendations, and it agreed with the 

recommendations in this report. DHS also described actions that it has 

under way and planned to address the recommendations. The 

department’s comments are reprinted in appendix II. 

We are sending copies of this report to the Chairmen and Ranking 

Minority Members of other Senate and House committees and 

subcommittees that have authorization and oversight responsibilities for 

homeland security. We are also sending copies to the DHS Secretary, the 

CBP Commissioner and, upon their request, to other interested parties. In 

addition, the report will be available at no charge on the GAO Web site at 

http://www.gao.gov. 

Should you or your offices have any questions on matters discussed in this

report, please contact me at (202) 512-3459 or at hiter@gao.gov. Contact 

points for our Offices of Congressional Relations and Public Affairs may 

be found on the last page of this report. Other contacts and key 

contributors to this report are listed in appendix III. 

Randolph C. Hite 

Director, Information Technology Architecture 

 and Systems Issues 

Agency Comments 

Page 12 GAO-06-580 Customs Modernization
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 13 GAO-06-580 Customs Modernization

Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations 

1

Information Technology: Customs Has Made Progress on

Automated Commercial Environment System, but it Faces 

Long-Standing Management Challenges and New Risks

Briefing to the Staffs of the

Subcommittees on Homeland Security,

Senate and House Committees on Appropriations 

March 10, 2006
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 14 GAO-06-580 Customs Modernization

2

Briefing Overview

Introduction

Objectives

Results in Brief

Background

Results

• Legislative Conditions

• Status of Recommendations

• Observations

Conclusions

Recommendations

Agency Comments

Attachment 1: Scope and Methodology
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 15 GAO-06-580 Customs Modernization

3

Introduction

The Department of Homeland Security’s (DHS) Customs and Border Protection

(CBP)1 is about 5 years into its trade processing modernization program, known 

as the Automated Commercial Environment (ACE). The goals of ACE are as

follows:

1 CBP was formed from the former U.S. Customs Service and other entities with border protection responsibility.

2 Targeting capabilities are currently being provided by DHS’s Automated Targeting System. This system targets containers

for inspection based on a perceived level of risk. ACE is intended to leverage these capabilities. 

Support border security by enhancing analysis and information sharing with other

government agencies to deal with increasing new security threats to our nation. 

Provide CBP personnel with the technology and information needed to decide, before

a shipment reaches the border, what should be targeted2

because it is a security 

threat, and what should be expedited because it complies with U.S. laws. 

Provide an integrated, fully automated information system to enable the efficient

collection, processing, and analysis of commercial import and export data. 

Streamline time-consuming and labor-intensive tasks for CBP personnel and the

trade community, through a single, Web-based interface, reducing costs for the 

government and the trade community. 

Enable users to process, view, and manage their accounts nationally, and obtain

historical information on cargo, conveyances, and crew, based on screening and

targeting rules.

Enable CBP to comply with legislative mandates to improve efficiency/effectiveness

and provide better customer service to U.S. citizens. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 16 GAO-06-580 Customs Modernization

4

Introduction 

The Department of Homeland Security Appropriations Act, 2006,1 appropriates 

about $320 million for ACE and states that DHS may not obligate any funds for 

ACE until it submits for approval to the House and Senate Committees on 

Appropriations a plan for expenditure that

1. meets the capital planning and investment control review requirements

established by the Office of Management and Budget (OMB), including

Circular A-11, part 7;2

2. complies with DHS’s enterprise architecture;

3. complies with the acquisition rules, requirements, guidelines, and systems 

acquisition management practices of the federal government;

4. includes a certification by the Chief Information Officer of DHS that an

independent verification and validation agent is currently under contract;

1 Pub. L. 109-90 (Oct. 18, 2005).

2 OMB Circular A-11 establishes policy for planning, budgeting, acquisition, and management of federal capital assets.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 17 GAO-06-580 Customs Modernization

5

Introduction

5. is reviewed and approved by the DHS Investment Review Board (IRB),1 the

Secretary of Homeland Security, and OMB; and

6. is reviewed by GAO.

On February 2, 2006, DHS submitted its fiscal year 2006 expenditure plan for 

$316.8 million to the House and Senate Appropriations Subcommittees on 

Homeland Security. 

DHS currently plans to acquire and deploy ACE in 11 increments, referred to as

releases. The first three releases are fully deployed and operating, and the fourth 

release is being deployed. Other releases are in various stages of definition and 

development.

1The purpose of the Investment Review Board is to integrate capital planning and investment control, budgeting, acquisition,

and management of investments. It is also to ensure that spending on investments directly supports and furthers the mission

and that this spending provides optimal benefits and capabilities to stakeholders and customers.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 18 GAO-06-580 Customs Modernization

6

Objectives

Objectives

As agreed, our objectives were to

• determine whether the ACE fiscal year 2006 expenditure plan satisfies the 

legislative conditions,

• determine the status of our open recommendations on ACE, and

• provide any other observations about the expenditure plan and DHS’s

management of the ACE program.

We conducted our work at CBP headquarters and contractor facilities in the 

Washington, D.C., metropolitan area, as well as at the port in Blaine, Washington,

from July 2005 through March 2006, in accordance with generally accepted

government auditing standards. Details of our scope and methodology are provided

in attachment 1.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 19 GAO-06-580 Customs Modernization

7

Results in Brief

Objective 1: Satisfaction of legislative conditions

The legislative conditions that the Congress placed on the use of fiscal year 2006 

ACE appropriated funds have been either partially or fully satisfied by the latest

expenditure plan and related program documentation and activities. Nevertheless,

more can be done to better address several aspects of these conditions, such as

having an approved privacy impact assessment, measuring ACE performance and 

results, ensuring architectural alignment, and employing effective independent

verification and validation (IV&V) practices. The following table summarizes the 

status of each of the legislative conditions. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 20 GAO-06-580 Customs Modernization

8

Results in Brief

Legislative condition Status

1. Meets the capital planning and investment control review requirements 

established by OMB, including OMB Circular A-11, part 7.

Partially satisfieda

2. Complies with DHS’s enterprise architecture. Partially satisfied

3. Complies with the acquisition rules, requirements, guidelines, and 

systems acquisition management practices of the federal government.

Satisfied b

4. Includes a certification by the Chief Information Officer of DHS that an 

independent verification and validation agent is currently under contract. 

Satisfied 

5. Is reviewed and approved by the DHS Investment Review Board,

Secretary of Homeland Security, and OMB. 

Satisfied 

6. Is reviewed by GAO. Satisfied 

Source: GAO.

a Partially satisfied means that the plan, in combination with supporting documentation, either satisfied or provided for satisfying

many, but not all, key aspects of the condition that we reviewed. 

b Satisfied means that the plan, in combination with supporting documentation, either satisfied or provided for satisfying every

aspect of the condition that we reviewed. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 21 GAO-06-580 Customs Modernization

9

Results in Brief

Objective 2: Status of actions to implement our open recommendations

Some recommendations have been addressed, while progress has been slow on

others, such as

• accurately reporting to the Appropriations Committees on CBP’s progress in 

implementing our prior recommendations; 

• developing and implementing a strategic approach to meeting the program’s 

human capital needs;

• using criteria for exiting key milestones that adequately consider indicators of

system maturity, such as severity of open defects and the associated risks;

and

• developing and implementing a performance and accountability framework for 

ensuring that promised capabilities and benefits are delivered on time and

within budget.

The following table summarizes the status of each of the open recommendations. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 22 GAO-06-580 Customs Modernization

10

Results in Brief

GAO recommendation 

Number of 

months open Status

1. Ensure that future expenditure plans are based on cost

estimates that are reconciled with independent cost

estimates.

7 monthsa Completeb,c 

2. Develop and implement a rigorous and analytically

verifiable cost estimating program.

46 months

 In progressd 

3. Immediately develop and implement a human capital 

management strategy that provides both near- and

long-term solutions; develop and implement missing

human capital practices.

46 months In progress

Source: GAO.

a Recommendation was also completed with respect to the fiscal year 2005 expenditure plan. b With respect to the fiscal year 2006 expenditure plan. c Complete means that actions have been taken to fully implement the recommendation. d In progress means that actions are under way to implement the recommendation.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 23 GAO-06-580 Customs Modernization

11

Results in Brief

GAO recommendation

Number of

months open Status 

4. Have future ACE expenditure plans specifically address any

proposals or plans for extending and using ACE infrastructure 

to support other homeland security applications. 

36 months In progress

5. Define measures, and collect and use associated metrics, for 

determining whether prior and future program management 

improvements are successful. 

22 months In progress

6. Define and implement an ACE accountability framework that 

ensures 

 

a. coverage of all program commitment areas, including key 

expected or estimated system (1) capabilities, use, and

quality; (2) benefits and mission value; (3) costs; and

(4) milestones and schedules.

12 months In progress

b. currency, relevance, and completeness of all such 

commitments made to the Congress in expenditure plans. 

12 months In progress

Source: GAO.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 24 GAO-06-580 Customs Modernization

12

Results in Brief

GAO recommendation

Number of

months open Status

c. reliable data relevant to measuring progress against

commitments.

12 months In progress 

d. reporting in future expenditure plans progress against 

commitments contained in prior expenditure plans.

12 months In progress 

e. use of criteria for exiting key readiness milestones that

adequately consider indicators of system maturity, 

such as severity of open defects. 

12 months In progress 

f. clear and unambiguous delineation of the respective

roles and responsibilities of the government and the 

prime contractor.

12 months Complete

7. Report quarterly to the House and Senate Appropriations

Committees on efforts to address open GAO

recommendations. 

22 months In progress 

Source: GAO. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 25 GAO-06-580 Customs Modernization

13

Results in Brief

Objective 3: Observations

Steps have been taken to address past pattern of ACE release shortfalls, but new

release management weaknesses are emerging.

• Release 4 pilot revealed performance problems that caused the pilot period to

be extended and the pilot scope to be reduced.

• Release 4 operational readiness review was passed despite unresolved 

severe defects, and Release 4 is now being deployed.

• Release 4 quality problems and enhancement needs have led to changes in

how ACE release requirements are defined.

• Release 4 problems delayed Screening 1 and led to a revised strategy for

delivering all screening and targeting releases.

• Screening 1 key milestones were passed despite unresolved severe defects. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 26 GAO-06-580 Customs Modernization

14

Results in Brief

• Past pattern of cost, schedule, and performance shortfalls with Releases 2, 3, 

and 4 makes new strategy of concurrently developing Releases 5, 6, and 7 

risky.

• Earned value management (EVM), a technique for measuring progress toward

meeting deliverables, is not being used to manage Screening 1 and Release 

5.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 27 GAO-06-580 Customs Modernization

15

Results in Brief

ACE’s operational performance has been mixed, and mission impact is unclear.

• Availability and responsiveness performance targets are largely being met. 

• Processing times for trucks crossing the border at key ports vary.

• Long-standing help desk limitations are being addressed.

• Usage by CBP and the trade is lower than expected.

• User satisfaction was reported as low. 

• Performance targets are not always realistic.

• Goals, expected mission benefits, and performance measures are not 

adequately aligned.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 28 GAO-06-580 Customs Modernization

16

Results in Brief

To assist CBP in managing ACE and increasing the chances that it will deliver

required capabilities on time and within budget and demonstrate promised mission 

benefits and results, we are recommending that DHS

• fully address legislative conditions associated with having an approved privacy 

impact assessment, measuring ACE performance and results, ensuring 

architectural alignment, and employing effective IV&V practices;

• accurately report to the Appropriations Committees on CBP’s progress in

implementing our prior recommendations;

• include in the June 30, 2006, quarterly update report to the Appropriations 

Committees a strategy for managing ACE human capital needs and the ACE

framework for managing performance and ensuring accountability;

• document key milestone decisions in a way that reflects the risks associated 

with proceeding with unresolved severe defects and provides for mitigating

these risks;
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 29 GAO-06-580 Customs Modernization

17

Results in Brief

• minimize the degree of overlap and concurrency across ongoing and future 

ACE releases, and capture and mitigate the associated risks of any residual

concurrency;

• use EVM in the development of all existing and future releases; 

• develop the range of realistic ACE performance measures and targets needed 

to support an outcome-based, results-oriented accountability framework,

including user satisfaction with ACE; and

• explicitly align ACE program goals, benefits, desired business outcomes, and 

performance measures. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 30 GAO-06-580 Customs Modernization

18

Results in Brief

In their oral comments on a draft of this briefing, DHS and CBP officials, including 

the Executive Director of Cargo Management Systems, CBP, generally agreed with

our findings, conclusions, and recommendations and stated that the presentation

was fair and balanced. They also provided clarifying information that we

incorporated as appropriate in this briefing.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 31 GAO-06-580 Customs Modernization

19

Background

ACE-Related Business Functions 

ACE is to support eight major CBP business areas.

1. Release1 Processing: Processing of cargo for import or export; tracking of 

conveyances, cargo, and crew; and processing of in-bond, warehouse,

Foreign Trade Zone, and special import and export entries.

2. Entry2 Processing: Liquidation and closeout of entries and entry summaries 

related to imports, and processing of protests and decisions.

3. Finance: Recording of revenue, performance of fund accounting, and

maintenance of the general ledger.

4. Account Relationships: Maintenance of trade accounts, their bonds and

CBP-issued licenses, and their activity.

5. Legal and Policy: Management of import and export legal, regulatory, 

policies and procedures, and rulings issues. 

1A release is the act of CBP permitting imported merchandise to enter the United States.

2An entry is the documentation required to be submitted to CBP in order for it to permit imported merchandise to enter the 

United States.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 32 GAO-06-580 Customs Modernization

20

Background

ACE-Related Business Functions

6. Enforcement: Enforcement of laws, regulations, policies and procedures, 

and rulings governing the import and export of cargo, conveyances, and crew.

7. Business Intelligence: Gathering and reporting data, such as references for 

import and export transactions, for use in making admissibility and release 

decisions.

8. Risk: Decision making about admissibility and compliance of cargo using 

risk-based mitigation, screening,1 and targeting.2

1Screening is the method of determining high-risk people or shipments before their arrival at a port. 

2Targeting is the risk-based determination of whether a shipment should undergo additional documentary review or

physical inspection.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 33 GAO-06-580 Customs Modernization

21

Background

ACE Technical Architecture

The ACE technical architecture is to consist of layers or tiers of computer 

technology:

• The Client Tier includes user workstations and external system interfaces.

• The Presentation Tier provides the mechanisms for the user workstations

and external systems to access ACE.

• The Integration Services Tier provides the middleware for integrating and 

routing information between ACE software applications and legacy systems.

• The Applications Tier includes the ACE software applications comprising 

commercial products (e.g., SAP1) and custom-developed software that provide 

the functionality supporting CBP business processes. 

• The Data Tier provides the data management and warehousing services for 

ACE, including database backup, restore, recovery, and space management.

Security and data privacy are to be embedded in all five layers.

1 SAP is a commercial enterprise resource planning software product that has multiple modules, each performing separate

but integrated business functions. ACE will use SAP to support many of its business processes and functions. CBP’s

Modernization Office is also using SAP as part of a joint project with its Office of Finance to support financial management,

procurement, property management, cost accounting, and general ledger processes.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 34 GAO-06-580 Customs Modernization

22

Background

ACE Technical Architecture

Simplified View of ACE Technical Architecture
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 35 GAO-06-580 Customs Modernization

23

Background

Acquisition Strategy

CBP is the component agency within DHS that is responsible for ACE. Currently, 

CBP is headed by an Acting Commissioner. Within CBP, the ACE program is

located within the Office of Information Technology, which is headed by the 

Assistant Commissioner for Information and Technology. 

ACE is being acquired and implemented through a series of incremental releases.

On April 27, 2001, CBP awarded a contract to IBM Global Services to develop and 

implement ACE. IBM and its subcontractors are collectively called the ACE Support 

Team (formerly called e-Customs Partnership).

CBP currently plans to acquire the 11 ACE releases in about 8.5 years. Screening

2 is to be acquired in two “drops,” or subreleases; Release 5 is to be acquired in 

two drops; Release 6 is to be acquired in three drops; and Release 7 is to be 

acquired in two drops.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 36 GAO-06-580 Customs Modernization

24

Background

Summary of ACE Releases

Planned Schedule for ACE
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 37 GAO-06-580 Customs Modernization

25

Background

Summary of ACE Releases

The following presents the functionality provided by the 11 ACE releases, their 

status, and associated plans. 

Release 1 (ACE Foundation): Provide information technology (IT) infrastructure—

computer hardware and system software—to support subsequent system releases. 

This release was deployed in October 2003 and is operating.

Release 2 (Account Creation): Give initial group of CBP national account 

managers1 and importers access to account information, such as trade activity. 

This release was deployed in October 2003 and is operating.

Release 3 (Periodic Payment): Provide additional account managers and 

importers, as well as brokers and carriers,2 access to account information; provide 

initial financial transaction processing and CBP revenue collection capability,

allowing importers and their brokers to make monthly payments of duties and fees. 

This release was deployed in July 2004 and is operating. 

1 CBP national account managers work with the largest importers.

2 Brokers obtain licenses from CBP to conduct business on behalf of the importers by filling out paperwork and obtaining a

bond; carriers are individuals or organizations engaged in transporting goods for hire.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 38 GAO-06-580 Customs Modernization

26

Background

Summary of ACE Releases

Release 4 (e-Manifest: Trucks): Provide electronic truck manifest1 processing and 

interfacing to legacy enforcement systems and databases. As discussed later, this 

release is operating at 39 truck border crossings as of March 8, 2006. Additional 

enhancement releases for Release 4 have been deployed since May 2005.

Screening 1 (Screening Foundation): Establish the foundation for screening 

cargo and conveyances by centralizing criteria and results into a single standard

database; allow users to define and maintain data sources and business rules. This 

release is scheduled for deployment beginning in March 2006. 

Screening 2 (Targeting Foundation): Establish the foundation for advanced

targeting capabilities by enabling CBP’s National Targeting Center to search 

multiple databases for relevant facts and actionable intelligence. This release is

scheduled for deployment in two drops:

• Screening 2 Targeting Platform (TP): Provide a platform to collect and 

search relevant data and other information from multiple databases. This drop

is scheduled for deployment beginning in June 2006.

1Manifests are lists of passengers or invoices of cargo for a vehicle, such as a truck, ship, or plane.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 39 GAO-06-580 Customs Modernization

27

Background

Summary of ACE Releases

• Screening 2 Targeting Foundation (TF): Build on the targeting platform to 

add new data sources, enhance screening business rules, and provide 

reporting capabilities. This drop is scheduled for deployment beginning in

October 2006; however, CBP deployed a prototype to the National Targeting 

Center as part of an effort to gather detailed requirements.

Release 5 (Entry Summary, Accounts, and Revenue): Leverage SAP

technologies to enhance and expand accounts management, financial 

management, and entry summary functionality. This release is being developed in

two drops: 

• Master Data and Enhanced Accounts (Drop A1): Use SAP to deliver 

enhanced account creation and maintenance functionality and expand the 

types of accounts managed in ACE. This drop is scheduled for deployment 

beginning in May 2007.

• Entry Summary and Revenue (Drop A2): Expand ACE to encompass entry 

summary, interfaces with participating government agencies, calculation of

duties and fees, reconciliation processing, and refunds. This drop is scheduled

for deployment beginning in July 2008.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 40 GAO-06-580 Customs Modernization

28

Background

Summary of ACE Releases

Screening 3 (Advanced Targeting Capabilities): Provide enhanced screening for 

reconciliation, intermodal manifest, Food and Drug Administration data, and inbond, warehouse, and Foreign Trade Zone authorized movements; integrate

additional data sources into targeting capability; and provide risk management 

capability. This release is scheduled for deployment beginning in February 2007. 

Screening 4 (Full Screening and Targeting): Provide full screening and targeting 

functionality supporting all modes of transportation and all transactions within the 

cargo management life cycle, including enhanced screening and targeting

capability with additional technologies. This release is scheduled for deployment

beginning in December 2008.

Release 6 (e-Manifest: All Modes and Cargo Security): Provide electronic 

manifest capability for rail, air, and sea shipments; provide a multimodal manifest;1

enable full tracking of cargo, conveyances, individuals, and equipment; and 

enhance enforcement processes for rail, air, and sea. This release is planned for 

development in three drops: 

1The multimodal manifest involves the processing and tracking of cargo as it transfers between different modes of 

transportation, such as cargo that arrives by ship, is transferred to a truck, and then is loaded onto an airplane.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 41 GAO-06-580 Customs Modernization

29

Background

Summary of ACE Releases

• E-Manifest: Rail and Sea (Drop M1): Extend the electronic manifest

functionality to rail and sea shipments; convert rail, sea, and truck electronic

manifests into the multimodal manifest. Drop M1 is scheduled for deployment 

beginning in July 2008. 

• E-Manifest: Air (Drop M2): Provide the electronic manifest capability to air

shipments, and bring all modes of transportation into the multimodal manifest. 

Drop M2 is scheduled for deployment beginning in October 2007.

• E-Manifest: Enhanced Tracking (Drop M3): Provide the capability to track 

cargo, conveyances, individuals, and equipment, providing more timely and 

accurate shipment status information. Drop M3 is scheduled for deployment 

beginning in June 2009.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 42 GAO-06-580 Customs Modernization

30

Background

Summary of ACE Releases

Release 7 (Exports and Cargo Control): Implement the remaining accounts 

management, revenue, manifest, and release and export functionality. This release 

is planned for development in two drops: 

• ESAR: Drawback, Protest, and IASS (Drop A3): Provide the import activity

summary statement (IASS),

1 drawback functionality, and enhanced protest; 

provide on-line processing for trade account applications. Drop A3 is 

scheduled for deployment beginning in December 2009.

• E-Manifest: Final Exports and Manifest (Drop M4): Extend the electronic

manifest for mail, pipeline, and hand carry; provide for electronic export 

processing. Drop M4 is scheduled for deployment beginning in December 

2009.

1An import activity summary statement is a summary of an importer’s shipment activities over a specific period of time that is

transmitted electronically to CBP on a periodic basis by importers and brokers.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 43 GAO-06-580 Customs Modernization

31

Background

Deployment Status

As of March 8, 2006, ACE, Releases 1 through 4 capabilities have been deployed 

to 39 of the 99 truck ports (see table).

State Deployed ports State Deployed ports 

Washington Blaine 

Sumas 

Lynden 

Point Roberts

Oroville 

Danville 

Ferry 

Laurier 

Boundary 

Frontier 

Metaline Falls

 Arizona Nogales 

Douglas 

Naco 

Sasabe

Lukeville 

 New Mexico Columbus

North Dakota Pembina 

Neche 

Hannah 

Sarles 

 Santa Teresa 

 Walhalla Hansboro Texas Brownsville 

 Maida Pharr 

Progreso 

Minnesota Noyes Rio Grande City

Roma 

Michigan Detroit Algonac Eagle Pass 

 Port Huron 

Marine City

Sault Ste. Marie Del Rio 

Presidio 

Source: GAO analysis based on CBP data. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 44 GAO-06-580 Customs Modernization

32

Background

ACE Satisfaction of Modernization Act Requirements 

ACE is intended to support CBP satisfaction of the provisions of Title VI of the 

North American Free Trade Agreement, commonly known as the Modernization 

Act. Subtitle B of the Modernization Act contains the various automation provisions

that were intended to enable the government to modernize international trade 

processes and permit CBP to adopt an informed compliance approach with

industry. The following table illustrates how each ACE release is to fulfill the 

requirements of Subtitle B.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 45 GAO-06-580 Customs Modernization

33

Background

ACE Satisfaction of Modernization Act Requirements
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 46 GAO-06-580 Customs Modernization

34

Background

Contract Tasks

Thus far, CBP has executed 21 contract task orders—12 have been completed and 

9 are active.

Status and Description of Completed Task Orders

No. Name Start

Date 

completed Description 

001 Program 

management

August 

2001 

July 2003 Initial program and project management; continued by

task 009.

002 Enterprise 

architecture

and 

engineering 

August 

2001 

June 2003 Initial enterprise architecture and system engineering; 

continued by task 010.

003 Requirements 

and planning

August 

2001 

July 2002 Initial requirements development and program planning

effort; continued by tasks for specific

increments/releases. 

004 Releases 1

and 2 

February 

2002 

November

2003

Design, development, testing, and deployment of

Releases 1 and 2 (initially intended to build Increment 

1, which was subsequently divided into four releases) 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 47 GAO-06-580 Customs Modernization

35

Background

Contract Tasks

No. Name Start 

Date

completed Description 

005 Requirements 

definition 

February 

2002 

March 2004 Development of Release 5 project plan, documentation

of ACE business processes, and development of an

ACE implementation strategy.

008 Releases 3 

and 4

August 

2002 

May 2005 Design, development, and testing of Releases 3 and 4,

and deployment of Release 3.

009 Foundation

program

management

February 

2003 

October

2003 

Follow-on to task 001 to continue program and project 

management activities.

010 Foundation

architecture

and 

engineering

February 

2003 

December 

2003 

Follow-on to task 002 to continue enterprise 

architecture and system engineering activities; 

continued by task 017.

011 Infrastructure 

and facilities 

August 

2002 

March 2003 Acquisition and setup of the necessary infrastructure

and facilities for the contractor to design, develop, and

test releases.

012 Operations and

maintenance

April 2003 September

2004 

Establishment of the infrastructure to operate and

maintain releases. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 48 GAO-06-580 Customs Modernization

36

Background

Contract Tasks

No. Name Start

Date 

completed Description

013 Legacy scripts 

modernization

June 2003 October

2003 

Conversion of scripts for interfacing desktop 

applications (MS Word and Excel) and mainframe

computer applications.

014 Knowledgebased risk 

management

September

2003

March 2004 Development, demonstration, and delivery of a

prototype to provide CBP insight into whether

knowledge-based risk management should be used in

ACE.

Source: GAO analysis based on CBP data.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 49 GAO-06-580 Customs Modernization

37

Background

Contract Tasks

Nine contract task orders are active.

Status and Description of Active Task Orders

No. Name Start Status Description 

006 Enterprise 

process 

improvement

February 

2002

April 2006 

planned 

completion 

Enterprise process improvement integration. 

007 International

Trade Data

System (ITDS)

January 

2002

December 

2006 planned

completion 

Assistance for participating government agencies to

define requirements for an integrated ACE/ITDS 

system.

015 Technology 

prototypes

October

2003

September

2006 planned

completion

Development and demonstration of technology

prototypes to provide CBP insight into whether the

technologies should be used in ACE.

016 Foundation

program 

management:

Communication, 

training, and

outcomes 

October

2003

September

2006 planned

completion 

Program management and support to organizational

change management through activities such as 

impact assessments, end user training,

communication, and outreach.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 50 GAO-06-580 Customs Modernization

38

Background

Contract Tasks

No. Name Start Status Description

017 Architecture

and 

Engineering 

December 

2003 

October 2006

planned 

completion

Coordination of program activities and alignment of 

enterprise objectives and technical plans through 

architecture and engineering activities. 

018 Enterprise Life 

Cycle 

Methodology 

January 

2004 

September

2006 planned

completion

Application of the CBP Enterprise Life Cycle 

Methodology to integrate multiple projects and other 

ongoing Customs operations into CBPMO. 

019 Operations and

Maintenance

March 

2004 

October 2006

planned 

completion

Follow-on to task 012 includes establishment, 

integration, configuration, and maintenance of the

infrastructure to support Releases 2, 3, and 4.

020 Screening and 

Targeting 

March 

2004 

March 2006

planned 

completion

Design, develop, test, and deploy the Screening and 

Targeting (S&T) operational capability. 

021 Releases 5

and 6 

May 2004 April 2006

planned 

completion

Project definition and initial design for Release 5; initial 

project authorization and definition for Release 6. 

Source: GAO analysis based on CBP data.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 51 GAO-06-580 Customs Modernization

39

Background

Chronology of Seven ACE Expenditure Plans 

Since March 2001, seven ACE expenditure plans have been submitted.1 

Collectively, the seven plans have identified a total of $1,698.1 million in funding.

• On March 26, 2001, CBP submitted to its appropriations committees the first 

expenditure plan seeking $45 million for the modernization contract to sustain 

Customs and Border Protection Modernization Office (CBPMO) operations, 

including contractor support. The appropriations committees subsequently 

approved the use of $45 million, bringing the total ACE funding to $50 million.

• On February 1, 2002, the second expenditure plan sought $206.9 million to

sustain CBPMO operations; define, design, develop, and deploy Increment 1,

Release 1 (now Releases 1 and 2); and identify requirements for Increment 2 

(now part of Releases 5, 6, and 7 and Screenings 1 and 2). The 

appropriations committees subsequently approved the use of $188.6 million, 

bringing total ACE funding to $238.6 million. 

1 In March 2001, appropriations committees approved the use of $5 million in stopgap funding to fund program management

office operations.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 52 GAO-06-580 Customs Modernization

40

Background

Chronology of Seven ACE Expenditure Plans 

• On May 24, 2002, the third expenditure plan sought $190.2 million to define, 

design, develop, and implement Increment 1, Release 2 (now Releases 3 and 

4). The appropriations committees subsequently approved the use of $190.2 

million, bringing the total ACE funding to $428.8 million.

• On November 22, 2002, the fourth expenditure plan sought $314 million to 

operate and maintain Increment 1 (now Releases 1, 2, 3, and 4); to design 

and develop Increment 2, Release 1 (now part of Releases 5, 6, and 7 and

Screening 1); and to define requirements and plan Increment 3 (now part of 

Releases 5, 6, and 7 and Screenings 2, 3, and 4). The appropriations

committees subsequently approved the use of $314 million, bringing total ACE 

funding to $742.8 million.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 53 GAO-06-580 Customs Modernization

41

Background

Chronology of Seven ACE Expenditure Plans 

• On January 21, 2004, the fifth expenditure plan sought $318.7 million to

implement ACE infrastructure; to support, operate, and maintain ACE; and to 

define and design Release 6 (now part of Releases 5, 6, and 7) and 

Selectivity 2 (now Screening 2 and 3). The appropriations committees 

subsequently approved the use of $316.8 million, bringing total ACE funding 

to $1,059.6 million. 

• On November 8, 2004, the sixth expenditure plan sought $321.7 million for 

design and development of Release 5 and Screening 2, definition of 

Screening 3, ACE program management, architecture and engineering, and 

operations and maintenance. The appropriations committees subsequently

approved the use of $321.7 million, bringing total ACE funding to $1,381.3

million.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 54 GAO-06-580 Customs Modernization

42

Background

Chronology of Seven ACE Expenditure Plans 

• On February 02, 2006, CBP submitted its seventh expenditure plan, seeking 

$316.8 million for detailed design and development of Release 5, 

development of Release 6, deployment of Screening 2, development and

deployment of Screening 3, program management and operations, and ACE 

operations, maintenance, and infrastructure implementation.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 55 GAO-06-580 Customs Modernization

43

Background

Summary of Funding

Summary of the ACE Fiscal Year 2006 Expenditure Plan

Plan activity Fundinga

Entry Summary, Accounts, and Revenue (Release 5), 

Design and Development $52.3

e-Manifest: All Modes and Cargo Security (Release 6), 

Design and Development 33.5

Screening and Targeting, Development 24.8

International Trade Data System 15.8

Foundation Architecture and Engineering 15.5

Operations and Maintenance 50.0

Implementation Infrastructure and Support 62.4

Communications, Training, and Outcomes 9.0

ACE Foundation Program Management 25.6

Program Office Operations 27.9

Total $316.8

Source: CBP.

a Millions of dollars.

The plan does not include management reserve funding. As of December 15, 2005,

the program had about $33.8 million in unused management reserve funding from

prior years.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 56 GAO-06-580 Customs Modernization

44

Background

ACE Testing and Related Milestones

Test Description Related milestonea

System 

integration test 

Verify that related system, subsystem, or 

module components are capable of integrating 

and interfacing with each other. 

Test Readiness Review 

System 

acceptance test

Verify that the developed system, subsystem, 

or module operates in accordance with 

requirements. 

Production Readiness 

Review 

User 

acceptance test

Verify that the functional scope of the release 

meets the business functions for the users.

Operational Readiness 

Review 

Source: eCP.

a

Generally, the identified milestone review comes at the conclusion of the related test.

Development of each ACE release includes system integration and system

acceptance testing, followed by a pilot period that includes user acceptance testing.

Generally, the purpose of these tests is to ensure that the system meets defined 

system requirements or satisfies user needs. The associated readiness reviews are 

to ensure that the system is ready to proceed to the next stage of testing or 

operation.

Tests and their related milestones are described in the following table.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 57 GAO-06-580 Customs Modernization

45

Background

Defect Categories

Defects identified during testing and operation of the system are classified into one 

of four severity categories, as described below.

Category Description

Critical

(Severity 1)

Defect prevents or precludes the performance of an operational or missionessential capability, jeopardizes safety or security, or causes the system,

application, process, or function to fail to respond or to end abnormally.

Severe

(Severity 2)

Defect prevents or precludes system from working as specified and/or

produces an error that degrades or impacts the system or user functionality.

Moderate

(Severity 3)

Defect prevents or precludes system from working as specified and/or

produces an error that degrades or impacts the system or user functionality.

An acceptable (reasonable and effective) work-around is in place that

rectifies the defect until a permanent fix can be made. 

Minor

(Severity 4)

Defect is inconsequential, cosmetic, or inconvenient and does not prevent

users from using the system to accomplish their tasks. Source: eCP.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 58 GAO-06-580 Customs Modernization

46

Background

Previous GAO Observations

Since 2001, we have reviewed CBP’s six prior expenditure plans and made a

number of observations that relate to cost overruns, schedule delays, quality 

limitations, and program management shortcomings. Among other things, we

observed the following: 

• Release 1 and 2 testing revealed a sufficient volume and significance of 

system defects to result in schedule delays.1

• Following cost overruns and schedule delays with Release 1, steps were

taken to avoid future problems, but the means for measuring the success of 

the actions was not in place.1

• Delays in Release 2 began a pattern of increased reliance on concurrent

activities, which in turn caused future release delays and cost overruns.1

• Release 3 testing and pilot activities were delayed and produced system

defect trends that raised questions about decisions to pass key milestones

and about the state of system maturity.2

1 GAO, Information Technology: Early Releases of Customs Trade System Operating, but Pattern of Cost and Schedule

Problems Needs to Be Addressed, GAO-04-719 (Washington, D.C.: May 14, 2004).

2 GAO, Information Technology: Customs Automated Commercial Environment Progressing, but Need for Management

Improvements Continues, GAO-05-267 (Washington, D.C.: Mar. 14, 2005).
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 59 GAO-06-580 Customs Modernization

47

Background

Previous GAO Observations

• Release 4 test phases were delayed and overlapped, and tests revealed a 

higher than expected volume and significance of defects, which again raised 

questions about decisions to pass key milestones and about the state of 

system maturity.

1

• Progress toward activating ACE importer accounts had not met expectations.1

1 GAO-05-267.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 60 GAO-06-580 Customs Modernization

48

Objective 1: Legislative Conditions

Capital Planning Requirements

DHS and OMB satisfied or partially satisfied each of its legislative conditions; GAO

satisfied its legislative condition.

Condition 1. The plan is to meet the capital planning and investment control

review requirements established by OMB, including Circular A-11, part 7, which 

establishes policy for planning, budgeting, acquisition, and management of federal 

capital assets. The plan, in conjunction with related program documentation and 

program officials’ statements, partially satisfied the condition.

The table that follows provides examples of the results of our analysis.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 61 GAO-06-580 Customs Modernization

49

Objective 1: Legislative Conditions

Capital Planning Requirements

Example of A-11 

conditions Results of our analysis 

Provide justification and 

describe acquisition 

strategy. 

The expenditure plan provides a high-level justification for 

ACE. In addition, supporting documentation describes the 

acquisition strategy for ACE releases covered in the 

expenditure plan, including Release 5 (Drops A1 and A2), 

Release 6 (Drops M1 and M2), Screening and Targeting 

Releases (S2 and S3), and International Trade Data System 

capabilities. 

Summarize life cycle costs 

and cost/benefit analysis, 

including the return on 

investment. 

CBP issued a cost/benefit analysis (CBA) for ACE on 

September 16, 2004. This analysis reported a life cycle cost 

estimate of $3.1 billion and a benefit to cost ratio of 8.29. This

means CBP expects that on average, there will be a return of 

$8.29 in benefits for every dollar spent on ACE. Subsequently, 

CBP reported that this ratio had increased to 10.77.1

1 We did not independently review the reliability of this benefit to cost ratio. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 62 GAO-06-580 Customs Modernization

50

Objective 1: Legislative Conditions

Capital Planning Requirements

Example of A-11

conditions Results of our analysis 

Address security and 

privacy. 

ACE Releases 1 through 4 passed security acceptance 

testing and have been accredited. CBP reported that security

plans will be developed for Releases 5 through 6. It also

reported that the privacy impact assessment for ACE has

been in draft for approximately 7 months and is not yet 

approved. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 63 GAO-06-580 Customs Modernization

51

Objective 1: Legislative Conditions

Capital Planning Requirements

Example of A-11

conditions Results of our analysis 

Provide performance goals

and measures. 

The plan and supporting documentation describe selected

ACE performance goals and measures. However, the 

performance goals (targets) are not always realistic. For 

example, CBP has established a single performance goal to

apply to all ports, despite differences among the ports.

Additionally, the performance measures that CBP is using are 

not aligned to the expected benefits of ACE. According to 

CBP, the performance measurement program continues to 

evolve as it gains more understanding of useful and 

meaningful measures. (See the observations section of the 

briefing for more detailed information on performance goals

and measures.) 

Source: GAO analysis.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 64 GAO-06-580 Customs Modernization

52

Objective 1: Legislative Conditions

Enterprise Architecture Compliance

Condition 2. The plan is to comply with DHS’s enterprise architecture (EA). The 

plan, including related program documentation and program officials’ statements, 

partially satisfied this condition.

The DHS Enterprise Architecture Board, supported by the Enterprise Architecture

Center of Excellence, is responsible for ensuring that projects demonstrate

adequate technical and strategic compliance with the department’s EA.

In May 2005, CBP requested that the DHS Enterprise Architecture Board evaluate 

its analysis of ACE’s alignment with the department’s EA. Using the ACE fiscal 

year 2006 business case, the ACE program plan, and other documentation, the

Center of Excellence evaluated alignment with version 2.0 of the DHS EA business 

model, data architecture, technical reference model, and transition strategy. In July

2005, the center approved CBP’s analysis and recommended that the request for 

program alignment be approved. The Enterprise Architecture Board subsequently 

concurred with the center’s recommendation.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 65 GAO-06-580 Customs Modernization

53

Objective 1: Legislative Conditions

Enterprise Architecture Compliance

DHS required CBP to provide documentation to support ACE’s alignment with

aspects of the EA such as the business architecture, the data model, the transition 

strategy, and the technical reference model. However, DHS officials told us that

they do not have a documented methodology for evaluating programs for 

compliance with the DHS EA, other than relying on the professional expertise of the 

members of the Center of Excellence. Moreover, no analysis or documentation was 

produced by the evaluators that could be used to verify the degree of alignment.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 66 GAO-06-580 Customs Modernization

54

Objective 1: Legislative Conditions

Acquisition Requirements

Condition 3. The plan is to comply with the acquisition rules, requirements, 

guidelines, and systems acquisition management practices of the federal 

government. The plan, in conjunction with related program documentation, satisfied

this condition.

The Software Acquisition Capability Maturity Model (SA-CMM®), developed by

Carnegie Mellon University’s Software Engineering Institute (SEI), is consistent 

with the acquisition guidelines and systems acquisition management practices of 

the federal government, and it provides a management framework that defines 

acquisition practices for such process areas as acquisition planning, solicitation, 

requirements development and management, project management, contract 

tracking and oversight, and evaluation.

In November 2003, SEI assessed ACE acquisition management processes and 

practices against the SA-CMM and assigned a level 2 rating, indicating that CBP 

had instituted basic acquisition management processes and controls in the 

following areas: acquisition planning, solicitation, requirements development and 

management, project management, contract tracking and oversight, and 

evaluation.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 67 GAO-06-580 Customs Modernization

55

Objective 1: Legislative Conditions

Independent Verification and Validation 

Condition 4. The DHS Chief Information Officer (CIO) is to certify that an

independent verification and validation (IV&V) agent is under contract. DHS 

satisfied this condition.

On January 26, 2006, the DHS CIO certified that an IV&V agent is under contract 

for the ACE program. However, the CIO also reported that the contractor’s

approach needs to be improved. For example, the CIO said that the contractor 

needs to address ACE satisfaction of quality standards and user needs. 

Further, the scope of the contractor’s activities is not consistent with the operative

industry standard, which states that IV&V should extend to key system products 

and development processes.1 CBP’s IV&V contract, awarded in December 2004, 

recognizes the importance of this scope of activities by requiring the contractor to

implement a program consistent with this standard.

1 IEEE Computer Society, Standard for Software Verification and Validation 1012-1998 (June 8, 2005).
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 68 GAO-06-580 Customs Modernization

56

Objective 1: Legislative Conditions

Independent Verification and Validation

In fiscal year 2005, CBP expended approximately $310,000 on IV&V. However,

these resources have been used to assess certain program management

processes, such as ACE help desk activities and progress in hiring Office of 

Information Technology staff. They have not been used to, for example, examine 

the development of ACE requirements or the quality of ACE software releases.

According to CBP officials, the scope of the IV&V contractor’s activities has not

included ACE product quality because they believe that such verification and 

validation activities would duplicate the program’s own quality assurance and

testing activities. While we agree that the IV&V agent should not duplicate work

that is already performed by an entity that is independent of the program’s cost and 

schedule processes, both the DHS CIO and the IV&V agent’s statement of work

have directed that product quality be addressed. For fiscal year 2006, CBP plans to

spend $856,000 on IV&V.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 69 GAO-06-580 Customs Modernization

57

Objective 1: Legislative Conditions

Review by DHS and OMB

Condition 5. The plan is to be reviewed and approved by the DHS Investment

Review Board (IRB), the Secretary of Homeland Security, and OMB. DHS and 

OMB satisfied this condition.

On November 21, 2005, the DHS IRB reviewed the ACE program. The Under

Secretary for Management approved the expenditure plan on behalf of the 

Secretary of Homeland Security on February 2, 2006.

OMB approved the plan on December 30, 2005.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 70 GAO-06-580 Customs Modernization

58

Objective 1: Legislative Conditions

Review by GAO

Condition 6. GAO is to review the plan. We satisfied this condition.

Our review was completed on March 10, 2006.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 71 GAO-06-580 Customs Modernization

59

Objective 2: Open Recommendations

CBP has implemented one of our seven open recommendations, and

implementation of the remaining six is in progress. The status of each of these 

recommendations is summarized below.

GAO recommendation

Number of

months open Status 

1. Ensure that future expenditure plans are based on cost

estimates that are reconciled with independent cost 

estimates.

7 months a Completeb,c 

2. Develop and implement a rigorous and analytically 

verifiable cost estimating program. 

46 months

 In progress d 

3. Immediately develop and implement a human capital

management strategy that provides both near- and 

long-term solutions; develop and implement missing 

human capital practices.

46 months In progress 

Source: GAO.

a Recommendation was also completed with respect to the fiscal year 2005 expenditure plan. b With respect to the fiscal year 2006 expenditure plan. c Complete means that actions have been taken to fully implement the recommendation. d In progress means that actions are under way to implement the recommendation.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 72 GAO-06-580 Customs Modernization

60

Objective 2: Open Recommendations

GAO recommendation

Number of

months open Status 

4. Have future ACE expenditure plans specifically address any

proposals or plans for extending and using ACE infrastructure 

to support other homeland security applications. 

36 months In progress

5. Define measures, and collect and use associated metrics, for 

determining whether prior and future program management 

improvements are successful. 

22 months In progress

6. Define and implement an ACE accountability framework that 

ensures 

 

a. coverage of all program commitment areas, including key 

expected or estimated system (1) capabilities, use, and

quality; (2) benefits and mission value; (3) costs; and

(4) milestones and schedules.

12 months In progress

b. currency, relevance, and completeness of all such 

commitments made to the Congress in expenditure plans. 

12 months In progress

Source: GAO.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 73 GAO-06-580 Customs Modernization

61

Objective 2: Open Recommendations

GAO recommendation 

Number of

months open Status 

c. reliable data relevant to measuring progress against

commitments. 

12 months In progress

d. reporting in future expenditure plans progress against 

commitments contained in prior expenditure plans. 

12 months In progress

e. use of criteria for exiting key readiness milestones that

adequately consider indicators of system maturity, 

such as severity of open defects. 

12 months In progress 

f. clear and unambiguous delineation of the respective

roles and responsibilities of the government and the 

prime contractor. 

12 months Complete

7. Report quarterly to the House and Senate Appropriations

Committees on efforts to address open GAO 

recommendations. 

22 months In progress

Source: GAO.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 74 GAO-06-580 Customs Modernization

62

Objective 2: Open Recommendations

Independent Cost Estimates

Open recommendation 1: Ensure that future expenditure plans are based on cost

estimates that are reconciled with independent cost estimates. 

Status: Complete1

The cost estimate in the fiscal year 2006 expenditure plan is based on the 

estimates in the current ACE program plan. CBP, with contractor support,

compared the program plan cost estimate with the independent cost estimate.

According to the analysis performed, the two estimates are consistent. The 

analysis was completed in October 2005, about 3 months before the fiscal year 

2006 expenditure plan was submitted to the Appropriations Committees. 

1With respect to the fiscal year 2006 expenditure plan.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 75 GAO-06-580 Customs Modernization

63

Objective 2: Open Recommendations

Effective Cost Estimating

Open recommendation 2: Develop and implement a rigorous and analytically 

verifiable cost estimating program that embodies the tenets of effective estimating

as defined in SEI’s institutional and project-specific estimating models.1

Status: In progress

CBP has defined and documented processes for estimating program costs for the 

expenditure plan (including management reserve costs). It has also hired a 

contractor to develop cost estimates, including contract task order cost estimates,

that are independent of CBP’s estimates. Further, to ensure sufficiency and 

completeness of the estimates, CBP tasked a support contractor with evaluating 

both the independent and the CBP estimates against the criteria defined by SEI. 

1For these models, see Robert E. Park, Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities 

of Software Organizations (Pittsburgh, Pennsylvania: SEI, Carnegie Mellon University, 1995); A Manager’s Checklist for 

Validating Software Cost and Schedule Estimates (Pittsburgh, Pennsylvania: SEI, Carnegie Mellon University, 1995).
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 76 GAO-06-580 Customs Modernization

64

Objective 2: Open Recommendations

Effective Cost Estimating

According to the results of the support contractor’s evaluation, the independent 

estimates satisfied all seven of the SEI criteria, and CBP’s estimates satisfied six of

the criteria and partially satisfied the remaining one. For the partially satisfied 

criterion, the evaluation found that the CBP estimate did not adequately consider 

past projects in its cost and schedule estimates.

In addition, the support contractor found that CBP’s estimate was the aggregation 

of three ACE component estimates, each of which was developed by the group 

responsible for a given component using different cost estimating methodologies. 

As a result, the support contractor recommended that CBP ensure that component 

estimates be based on the same methodology.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 77 GAO-06-580 Customs Modernization

65

Objective 2: Open Recommendations

Human Capital Strategy

Open recommendation 3: Immediately develop and implement a human capital 

management strategy that provides both near- and long-term solutions to program

office human capital capacity limitations, and report quarterly to the Appropriations 

Committees on the progress of efforts to do so.

Status: In progress

CBP does not have a documented human capital strategy covering its ACE 

program. As we have previously reported, such a strategy should provide for 

defining the positions needed (including core competencies) to perform core

program functions; assessing and inventorying current workforce skills and abilities;

assessing any gaps between needed and existing workforce levels and 

capabilities; and filling identified gaps via such means as new staff, training existing

staff, and augmenting staff with contractor support.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 78 GAO-06-580 Customs Modernization

66

Objective 2: Open Recommendations

Human Capital Strategy

In lieu of a documented human capital strategy, CBP officials told us that they have 

taken various steps to bolster their ACE workforce as part of a less formal strategy. 

For example:

• CBP expanded its contractor and government workforce dedicated to the ACE

program by merging staff assigned to trade-related legacy systems with the 

ACE program staff—creating a Cargo Management Systems Program Office. 

According to the officials, this merger has enabled them to leverage the 

knowledge of staff who have been working with cargo systems for the past 10 

to 20 years, and thus increased the level of expertise available to ACE.

• CBP recently trained staff working on a major release on earned value 

management.

• CBP began using subject matter experts from existing field operations

advisory boards to help program officials define requirements for future 

releases. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 79 GAO-06-580 Customs Modernization

67

Objective 2: Open Recommendations

Human Capital Strategy

Further, CBP has reported quarterly to the appropriations committees on its human 

capital goals and objectives for ACE.

Nevertheless, CBP officials acknowledged that they have not developed and

followed a formal strategy that systematically compares competency-based staffing

needs to on-board capabilities and defines and implements long-term and shortterm plans for closing any shortfalls and associated strategies. Further, they have

not reported to the appropriations committees on these shortfalls. They stated that

they intend to develop a formal human capital strategy, and in doing so, will reflect

the activities that they have already taken.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 80 GAO-06-580 Customs Modernization

68

Objective 2: Open Recommendations

Extending ACE Infrastructure

Open Recommendation 4: Have future ACE expenditure plans specifically

address any proposals or plans, whether tentative or approved, for extending and 

using ACE infrastructure to support other homeland security applications, including 

any impact on ACE of such proposals and plans.

Status: In progress

The expenditure plan describes steps under way and planned to leverage ACE’s

relationship with other homeland security applications. According to the plan,

• ACE and US-VISIT1 conform to the DHS enterprise architecture, which is to 

define standard shared services that the two systems can request. Such a 

services oriented architecture is intended to promote reuse and reduce

overlap and duplication. According to CBP officials, they are currently

exploring shared services areas. 

1United States Visitor and Immigrant Status Indicator Technology (US-VISIT) is a governmentwide program to collect,

maintain, and share information on foreign nationals for enhancing national security and facilitating legitimate trade and travel,

while adhering to U.S. privacy laws and policies.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 81 GAO-06-580 Customs Modernization

69

Objective 2: Open Recommendations

Extending ACE Infrastructure

• ACE and US-VISIT have begun to use a common infrastructure to deploy and 

operate their systems. For example, in locations where ACE and US-VISIT

have been fully deployed, such as at the port of Blaine, Washington, the two 

systems operate on the same network and the same workstations.

However, the plan does not discuss the impact to ACE (e.g., cost and schedule) 

with regard to these efforts and plans.

Besides what is described in the plan, ACE officials told us that they meet once a 

month with US-VISIT officials to share lessons learned on program management 

topics, such as risk management, change management, and configuration 

management.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 82 GAO-06-580 Customs Modernization

70

Objective 2: Open Recommendations

Measuring Success

Open recommendation 5: Define measures, and collect and use associated 

metrics, for determining whether prior and future program management 

improvements are successful.

Status: In progress

CBP continues to make changes that are intended to improve overall program

management.

• CBP has merged aspects of its Office of Information Technology that 

managed trade-related legacy systems with its former ACE program office, 

thereby creating the Cargo Management Systems Program Office. This

reorganization, according to CBP, is to result in (1) enhanced government

oversight of ACE development, (2) better definition of requirements for future 

releases, and (3) faster and cheaper delivery of ACE capabilities. However, 

program officials told us that they have not established measures or targets 

for determining whether the reorganization is providing these benefits. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 83 GAO-06-580 Customs Modernization

71

Objective 2: Open Recommendations

Measuring Success

• CBP eliminated the dependencies between the screening and targeting 

releases and the cargo releases by leveraging its existing Automated

Targeting System in delivering ACE screening and targeting capabilities. (This 

topic is discussed in further detail in the observations section.) It has also

established measures for determining the impact of this change, including 

saving the program $10 million in life-cycle costs and allowing the screening

and targeting releases to be fully deployed 1 year ahead of schedule.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 84 GAO-06-580 Customs Modernization

72

Objective 2: Open Recommendations

Accountability Framework

Open recommendation 6: Define and implement an ACE accountability 

framework that fulfills several conditions:

a. Covers all program commitment areas, including key expected or estimated

system (a) capabilities, use, and quality; (b) benefits and mission value; (c) costs;

and (d) milestones and schedules.

Status: In progress

CBP has prepared an initial version of an accountability framework that program

officials said they intend to improve, but have nevertheless begun using. This

framework is built around measuring progress against costs, milestones and 

schedules, and risks for select releases. It is also intended to permit measurement 

at different levels of aggregation, and for whatever incremental periods of time

desired (e.g., monthly). However, as we discuss later, the benefit commitments 

have not been well defined and the performance targets are not always realistic.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 85 GAO-06-580 Customs Modernization

73

Objective 2: Open Recommendations

Accountability Framework

b. Ensures currency, relevance, and completeness of all program commitments

made to the Congress in expenditure plans.

Status: In progress

The fiscal year 2006 expenditure plan continues to include inaccurate, dated, and 

incomplete information and to omit other relevant information.

• The plan did not include information regarding CBP’s decision to eliminate the

dependencies among the screening and targeting releases and the cargo 

releases and to leverage its existing Automated Targeting System capabilities, 

nor did the plan reflect the $10 million cost reduction and the 1 year schedule 

acceleration that this change is intended to produce. 

• The plan includes milestone completion dates for Releases 5, 6, and 7 and 

Screening 1 that are not accurate. For example, the expenditure plan shows

that the preliminary design review for Release 5, drops A1 and A2, was 

scheduled to occur in August 2005; however, it did not occur until November 

2005. Similarly, although the plan states that the critical design review for

Release 6, drop M1, was scheduled to occur in November 2005, it is currently 

scheduled to take place August 2006.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 86 GAO-06-580 Customs Modernization

74

Objective 2: Open Recommendations

Accountability Framework

According to CBP officials, they did not update the plan because they wanted to

provide it to the Appropriations Committees as soon as possible. They also stated 

that they use the congressional quarterly reports to provide the Appropriations 

Committees with more current, relevant, and complete information about ACE.

However, these quarterly reports are generally submitted 3 to 4 months after the 

end of each quarter. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 87 GAO-06-580 Customs Modernization

75

Objective 2: Open Recommendations

Accountability Framework

c. Ensures reliable data relevant to measuring progress against commitments.

Status: In progress

The data that CBP uses to measure progress against commitments are not 

consistently reliable. For example:

• Data in the defect tracking system show that Release 4 is operating with

long-standing defects and that new defects have not been closed. For 

example, as of January 18, 2006, the system showed that a number of

defects were open:

3 Severity 4

4 Severity 3

12 Severity 2

Number Severity
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 88 GAO-06-580 Customs Modernization

76

Objective 2: Open Recommendations

Accountability Framework

• Program officials told us that many of these defects are in fact resolved. 

However, the defect tracking system does not accurately reflect this status for 

two reasons: 

• staff were using multiple systems to track defects and these systems were

not always reconciled; and

• newer staff were inexperienced in using the defect tracking system. 

According to program officials, they are manually reconciling the data between 

the multiple systems, and plan to implement a new system that will track all 

defects in one central system. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 89 GAO-06-580 Customs Modernization

77

Objective 2: Open Recommendations

Accountability Framework

d. Ensure future expenditure plans report progress against commitments contained 

in prior expenditure plans.

Status: In progress

The fiscal year 2006 expenditure plan does not adequately describe progress 

against commitments made in previous plans. For example:

• The plan provides a summary of the funding requested in each of the previous 

six expenditure plans. However, it does not provide information on whether 

these funding amounts were actually expended or obligated as planned.

• The plan includes a new schedule for ACE releases, but it does not report

progress relative to the schedule presented in the fiscal year 2005 plan.

• The plan does not explain the extent to which Release 5 design and 

development planned in the fiscal year 2005 plan was actually accomplished.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 90 GAO-06-580 Customs Modernization

78

Objective 2: Open Recommendations

Accountability Framework

e. Ensure criteria for exiting key readiness milestones adequately consider

indicators of system maturity, such as severity of open defects.

Status: In progress

According to CBP officials, ACE milestone exit criteria provide for considering the 

risk associated with having unresolved severe defects. Specifically, the criteria

state that critical and severe defects must be resolved or there must be a plan in

place to resolve them. Using these criteria, CBP passed several release milestones 

with severe defects still open:

• Release 4 operational readiness review was passed with 9 severe defects

open,

• Screening 1 test readiness review was passed with 3 severe defects open,

and

• Screening 1 production readiness review was passed with 2 severe defects

open.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 91 GAO-06-580 Customs Modernization

79

Objective 2: Open Recommendations

Accountability Framework

In making these decisions, CBP officials told us that they considered the 

associated risk and concluded that the risk was acceptable. In particular, they

stated that it was more important to get the releases in the hands of users and 

thereby gain user acceptance and receive user feedback sooner than it was to first

resolve the defects. 

However, CBP officials were unable to provide us with any documentation on how 

the inherent risks of passing these milestones with open severe defects were

assessed, and the ACE risk inventory does not include any risks associated with 

these decisions.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 92 GAO-06-580 Customs Modernization

80

Objective 2: Open Recommendations

Accountability Framework

f. Ensures clear and unambiguous delineation of the respective roles and 

responsibilities of the government and the prime contractor. 

Status: Complete.

The current version of the ACE program plan describes general roles and 

responsibilities for the government and the prime contractor. More detailed roles 

and responsibilities of CBP, the prime contractor, and the support contractors have 

been documented in a roles and responsibilities matrix that assigns primary 

responsibility for each activity, such as testing, training, and maintenance. In

addition, the task orders describe the specific responsibilities and expectations of 

the contractors. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 93 GAO-06-580 Customs Modernization

81

Objective 2: Open Recommendations

Implementation Reporting

Open recommendation 7: Report quarterly to the House and Senate 

Appropriations Committees on efforts to address open GAO recommendations.

Status: In progress

CBP submitted reports to both Committees on its efforts to address open GAO

recommendations for the quarters ending March 31, 2005; June 30, 2005; and

September 30, 2005. CBP also plans to submit a report for the quarter ending

December 31, 2005.

However, progress in addressing our recommendations was not always reported 

accurately. For example, CBP reported that it will review the expenditure plan

throughout the approval process to ensure that it incorporates the most current 

program commitments. However, it did not. Additionally, in the September 2005 

report, CBP stated that the fiscal year 2006 expenditure plan would have a section 

that describes progress made against program commitments made in all prior

expenditure plans. However, the expenditure plan does not include this information.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 94 GAO-06-580 Customs Modernization

82

Objective 3: Development Observations

Steps have been taken to address past pattern of ACE release shortfalls, but

new release management weaknesses are emerging.

As we have observed in our previous reviews, CBP established a pattern of

addressing quality problems with earlier releases by borrowing resources from

future releases, which led to schedule delays and cost overruns. This pattern has 

continued with Release 4, which developed problems that caused delays with

Screening 1. CBP has taken steps to mitigate this problem by eliminating the 

dependencies between the cargo releases and the screening and targeting 

releases.

However, CBP’s planned schedule for developing Releases 5, 6, and 7 includes a

significant level of concurrency, because of CBP’s interest in delivering ACE 

functionality sooner. As we have reported in the past, such concurrency between 

release activities has led to cost overruns and schedule delays. Thus, the revised 

ACE plans and actions are potentially reintroducing the same problems that 

resulted in shortfalls in the past. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 95 GAO-06-580 Customs Modernization

83

Objective 3: Development Observations

Release 4 Pilot Problems

Release 4 pilot revealed performance problems that caused the pilot period 

to be extended and the pilot scope to be reduced.

The Release 4 pilot was intended to ensure that the release met its requirements

before it was deployed to all truck ports. Examples of pilot activities include training

of CBP and trade personnel in how to use the release and conducting user 

acceptance testing. As planned, the pilot was to be conducted at two truck ports—

Blaine, Washington, and Buffalo, New York—and the testing was to occur during a

10 week period. The pilot was to conclude with the operational readiness review on

February 23, 2005.

However, the pilot only occurred at Blaine, the pilot period covered 17 weeks, and 

the operational readiness review did not occur until April 14, 2005. The following 

are significant events that occurred during the pilot.

• During the initial days of the pilot, Release 4 slowed truck processing. To

address this, users identified needed system enhancements, primarily 

intended to reduce the number of steps required to process trucks. As a 

result, the pilot was suspended 11 days after it began so that the 

enhancements could be developed and implemented.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 96 GAO-06-580 Customs Modernization

84

Objective 3: Development Observations

Release 4 Pilot Problems

• The pilot resumed about 5 weeks after it was suspended. However, new

problems were encountered, such as slow system response times and screen 

freezes. Additionally, the release was not properly displaying alerts for

potential criminals or terrorists. As a result, the pilot was again suspended 

about 6 weeks after it was resumed.

• The pilot resumed about 3 weeks after the second suspension. On April 14, 

2005, about 7 weeks later than planned, Release 4 passed operational 

readiness review.

• Because of the ongoing problems with Release 4 in Blaine, CBP cancelled its

plans to include Buffalo in the pilot.

A comparison of the planned versus actual pilot schedules is summarized in the

following figure.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 97 GAO-06-580 Customs Modernization

85

Objective 3: Development Observations

Release 4 Pilot Problems

Planned versus Actual Time Frame for Release 4 Pilot Activities
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 98 GAO-06-580 Customs Modernization

86

Objective 3: Development Observations

Release 4 Milestone Reviews and Quality Problems

Release 4 operational readiness review was passed despite unresolved 

severe defects, and Release 4 is now being deployed.

CBP’s criteria for passing key milestones—such as the Release 4 operational 

readiness review (ORR)—stipulate that all critical and severe defects must either 

be resolved or there must be plans in place to resolve the defect. As noted earlier,

we have recommended that any decisions to pass key milestones adequately 

consider indicators of system maturity, such as open severe defects.

At the time of the Release 4 operational readiness review, CBP reported that 9 

severe defects remained open, and that it had plans in place to resolve each of 

these defects. Of these 9, one was subsequently cancelled, 4 were closed within

approximately 6 weeks of ORR, 3 were closed several months after ORR, and 1 

remains open about 10 months later. According to program officials, the remaining 

open defect has been lowered from severe to moderate status, but the change has 

yet to be reflected in the defect tracking system.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 99 GAO-06-580 Customs Modernization

87

Objective 3: Development Observations

Release 4 Milestone Reviews and Quality Problems

CBP officials told us that they considered the risk associated with passing ORR

with these 9 severe defects, and concluded that the risk was acceptable. In

particular, they indicated that it was important to get Release 4 in the hands of

users and thereby gain user acceptance and receive user feedback sooner.

However, it is important to note that deploying a system with known operational 

problems, while likely to encourage user feedback, may actually limit user

acceptance, particularly given the number of Release 4 enhancements needed. 

(The next section discusses these enhancements.)

Further, CBP officials were unable to provide us with any documentation on how

the inherent risks of passing this milestone with open severe defects were

assessed, and the ACE risk inventory does not include any risks associated with 

this decision. 

Since ORR, Release 4 has been deployed to 38 truck ports on the northern and 

southern U.S. borders; by December 2006, CBP plans to deploy Release 4 to the 

remaining 60 truck ports.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 100 GAO-06-580 Customs Modernization

88

Objective 3: Development Observations

Release 4 Definition Limitations and Enhancements

Release 4 quality problems and enhancement needs have led to changes in 

how ACE release requirements are defined.

Inadequate requirements definition was a major reason for the problems and 

delays encountered during the Release 4 pilot in Blaine. Specifically, program

officials told us that the requirements definition process did not fully identify the key

functionality contained in the legacy system that ACE needed to provide. Also, the 

process did not adequately consider the capabilities that ACE would need to

provide in an actual operational environment. For example, Release 4

requirements did not reflect the large volumes of transactions common in an

operational setting, such as documenting notifications of potential security 

violations. 

According to program officials, the requirements definition process was insufficient 

in part because personnel involved in defining requirements did not have sufficient

experience with the legacy systems that ACE was to replace and/or interface with.

Moreover, they said that although Release 4 met the requirements that were

defined for it in 2001, since then CBP’s mission and operations have changed, 

creating the need to introduce additional system requirements.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 101 GAO-06-580 Customs Modernization

89

Objective 3: Development Observations

Release 4 Definition Limitations and Enhancements

To address these limitations, the number and scope of release enhancements 

(subreleases) have been larger than anticipated. To date, CBP has implemented 

two enhancement releases that add, for example, performance enhancements to

transaction processing and improvements to the usability of reports. Although CBP

has yet to determine the total number of enhancement subreleases that will be 

developed for Release 4, program officials stated that more are needed. 

To improve requirements definition for future releases, CBP has changed its 

requirements definition process. For instance, program officials said that they are 

• leveraging existing field operations advisory boards to augment program staff

defining the requirements for future releases, 

• using contractors to analyze legacy system functionality to ensure that ACE 

requirements reflect it, and

• regularly involving key representatives from the trade community to more fully 

define ACE requirements and help ensure that ACE meets users’ needs.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 102 GAO-06-580 Customs Modernization

90

Objective 3: Development Observations

Release 4 Impacts on Screening 1

Release 4 problems delayed Screening 1 and led to a revised strategy for

delivering all screening and targeting releases.

The Release 4 problems caused delays in developing and testing Screening 1 

because resources that were to be used on Screening 1 were diverted to Release 

4. For example: 

• The test environment’s availability to support Screening 1 was delayed about

11 weeks because it was still supporting Release 4.

• The staff targeted for Screening 1 development and test activities were being 

held on Release 4 longer than planned.

As a result, the combined critical design review/test readiness review for Screening 

1 was delayed by 60 days. As we have previously reported, this diversion of 

resources from future releases to address problems on prior releases has been a 

pattern on ACE for many years, owing largely to the concurrency in the

development of releases and the releases’ dependency on the same resources. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 103 GAO-06-580 Customs Modernization

91

Objective 3: Development Observations

Release 4 Impacts on Screening 1

To correct this pattern, as well as to leverage existing targeting functionality, CBP 

decided to take two steps:

• to “decouple” all screening and targeting releases from ACE’s cargo releases

and 

• to use its existing system, the Automated Targeting System (and associated

resources), to deliver needed screening and targeting capabilities.

In addition to addressing the pattern of delays caused by the releases competing 

for the same resources, CBP estimates that these changes will save the program

$10 million and allow the screening and targeting releases to be fully deployed 

1 year ahead of schedule.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 104 GAO-06-580 Customs Modernization

92

Objective 3: Development Observations

Screening 1 Milestone Reviews and Quality Problems

Screening 1 key milestones were passed despite unresolved severe defects.

As previously noted, CBP’s criteria for passing key milestones, such as the test 

readiness review and production readiness review for Screening 1, stipulate that all 

critical and severe defects must either be resolved or have work-off plans in place.

Also, we have recommended that any decisions to pass key milestones adequately 

consider indicators of system maturity, such as open severe defects.

A number of severe defects were open at the time of these milestones:

• At the test readiness review on November 14, 2005, CBP reported that three 

severe defects were open.

• At the production readiness review on December 22, 2005, CBP reported that 

two severe defects were open.

According to CBP officials, these key milestones were passed because work-off

plans were in place for resolving each severe defect. Thus, in their view the risk of

proceeding did not outweigh the need to get Screening 1 to the users and thereby 

gain user acceptance and receive user feedback sooner. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 105 GAO-06-580 Customs Modernization

93

Objective 3: Development Observations

Screening 1 Milestone Reviews and Quality Problems

However, deploying a system with known operational problems, while likely to

encourage user feedback, may actually limit user acceptance. Further, CBP 

officials were unable to provide us with any documentation on how the inherent 

risks of passing these milestones with open severe defects were assessed, and the

ACE risk inventory does not include any associated risks. 

Screening 1 is scheduled for deployment beginning on March 23, 2006. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 106 GAO-06-580 Customs Modernization

94

Objective 3: Development Observations

Concurrent Development Risks

Past pattern of development problems with Releases 2, 3, and 4 makes new

strategy of concurrently developing Releases 5, 6, and 7 risky. 

As we have previously reported,1 concurrent development of system components 

introduces risks that can adversely impact program cost and schedules. According

to ACE contractors, these risks can include limited understanding of requirements

before design and development activities begin, uncertainty regarding the timely

availability of commercial hardware and software products, and increased nearterm funding requirements. Other risks include contention for limited resources

(such as key personnel, as well as development and testing equipment and 

facilities) and dependencies among releases not being met.

Concurrency in developing early ACE releases caused schedule slips and cost 

overruns. As we previously reported, overlapping the development of Releases 2

and 3 caused delays in Release 3 and resulted in Releases 1 through 4 costing 

more than planned. 

1 GAO-04-719.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 107 GAO-06-580 Customs Modernization

95

Objective 3: Development Observations

Concurrent Development Risks

Factors contributing to the schedule slips and cost overruns include the following:

• Additional resources were needed to eliminate Release 1 and 2 defects.

• Unavailable testing and development environments extended Release 3 and 4 

schedules.

• Increased scope of Releases 3 and 4 to include Release 2 deferred

requirements.

Despite these experiences, CBP established a new ACE program plan in July 2005 

that involves considerable overlap across the development schedules for Releases 

5 and 6 and for Releases 6 and 7. According to CBP, the additional risk introduced 

by this concurrency is outweighed by the potential for delivering ACE functionality 

sooner.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 108 GAO-06-580 Customs Modernization

96

Objective 3: Development Observations

Concurrent Development Risks

However, performance to date in meeting the highly concurrent milestones in the 

July 2005 program plan shows that delays are already occurring that are 

introducing even more schedule overlap and thus program risk. For example, both 

Releases 5 and 6 have experienced significant design-related delays, as shown by

the table below. These delays have in turn increased the amount of development 

overlap across Releases 5, 6 and 7, as shown by the next slide, which compares

the July 2005 and January 2006 schedules for these releases. 

Delays in Meeting Release 5 and 6 Design-Related Milestones

Release/drop Milestone Planned Actual/revised plan Delay 

Release 5/

Drops A1 and 

A2 

Preliminary

Design Review

August 2, 2005 November 10, 2005 3+ months 

Release 5/

Drop A1

Critical Design

Review

October 31,

2005

Planned for May 4,

2006 

6+ months

Release 6/

Drop M1

Critical Design

Review

November 30,

2005

Planned for August 3,

2006 

8+ months
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 109 GAO-06-580 Customs Modernization

97

Objective 3: Development Observations

Concurrent Development Risks

Original versus Revised Schedules for Developing Releases 5, 6, and 7
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 110 GAO-06-580 Customs Modernization

98

Objective 3: Development Observations

Concurrent Development Risks

The risk associated with this concurrency in development schedules is exacerbated 

by several factors.

• Releases/drops have extensive data and resource (e.g., testing environments) 

dependencies. For example, Release 6, drop M1, which is to provide 

electronic manifest capabilities to rail and sea ports, is dependent on the data 

that will be provided by Release 5, drop A1, which is to add trade account 

types and corresponding data. Further, CBP officials have said that Release 

5, drop A2, and Release 6, drop M1, must be tested together in the same

testing environment. Therefore, if either of the drops is delayed, the other will

be delayed as well.

• Just as delays on drops have already increased contention for resources,

further delays could introduce additional contention.

• Release 5, 6, and 7 include the vast majority of ACE’s functionality (87 

percent) and are to produce the more significant mission benefits.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 111 GAO-06-580 Customs Modernization

99

Objective 3: Development Observations

Earned Value Management Not Being Used

Earned value management is not being used to manage Screening 1 and 

Release 5.

CBP has not used earned value management (EVM) to manage Release 5 and 

Screening 1. EVM is a management tool for measuring program progress by 

comparing, during a given period of time, the value of work accomplished with the 

amount of work expected to be accomplished; this comparison permits 

performance to be evaluated based on calculated variances from the planned cost

and schedule. EVM is both an industry accepted practice and an OMB 

requirement.

For Screening 1, CBP discontinued use of EVM in June 2005, when it made the 

decision to decouple the screening and targeting releases from the cargo releases. 

At that time, it decided to take advantage of the functionality of its legacy targeting 

system, Automated Targeting System (ATS), as well as the expertise of ATS staff.

According to CBP officials, when ATS staff began working on Screening 1, they 

were unfamiliar with EVM and therefore did not use it. CBP officials stated that in

lieu of EVM, they monitored the actual costs of work performed. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 112 GAO-06-580 Customs Modernization

100

Objective 3: Development Observations

Limited Use of Earned Value Management

In addition, CBP has not used EVM for Release 5, which has been under 

development for 22 months and for which $29.5 million has reportedly been 

expended. According to CBP officials, use of EVM was not possible because the

revision of the Release 5 scope and strategy delayed definition of requirements

and prevented cost and schedule baselines from being established. Therefore,

Release 5 work had to be conducted under intermittent authorizations to proceed, 

which did not have measurable baselines.

To respond to these EVM limitations, according to program officials, Screening 1 

staff have now been trained on EVM. They also agreed that prior Release 5 work

should have been managed by measurable baselines. They said that they plan to 

use EVM for future Release 5 work once fiscal year 2006 funds become available.

In addition, they said that they intend to establish baselines for any work performed 

under authorizations to proceed, and appropriate performance metrics will be

applied.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 113 GAO-06-580 Customs Modernization

101

Objective 3: Operation Observations

ACE’s operational performance has been mixed, and mission impact is 

unclear.

As described earlier, ACE Releases 1 to 4 are in operation. To date, these 

releases’ operational performance has been uneven. For example, ACE has largely 

been meeting its goals for being available and responsive in processing virtually all 

daily transactions, and it has decreased truck processing times at some ports.

However, ACE is not being used by as many CBP and trade personnel as was 

expected, and truck processing times at other ports have increased. Moreover, 

overall user satisfaction has been low.

In addition, ACE goals, expected mission benefits, and performance measures are 

not fully defined and adequately aligned. Where performance measures have been 

defined, the associated targets are not always realistic. As a result, it is unclear 

whether ACE has realized or will realize the mission value that it was intended to

bring to CBP’s and other agencies’ trade- and border security-related operations.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 114 GAO-06-580 Customs Modernization

102

Objective 3: Operation Observations

Availability Targets Largely Met

Availability and responsiveness targets are largely being met.

CBP has defined ACE availability in terms of the percentage of transactions that

are to be executed successfully each day. According to a service level agreement

between the ACE Support Team and CBP, 99.9 percent of all ACE transactions on

any given day are to be successful. The ACE Support Team reports that ACE met

this requirement on all but 22 days between January 1, 2005, and January 27, 

2006.

For each of the 22 days that the system did not meet the agreement, the ACE

Support Team identified and corrected the root cause. For example, outages were

caused by 

• a server accidentally being shut down by data center personnel,

• a software error that disabled a transaction function, and

• a network switch that malfunctioned.

To address these causes, the ACE Support Team instituted new data center 

procedures, deleted ACE code that caused the transaction error, and established 

methods for identifying network problems sooner.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 115 GAO-06-580 Customs Modernization

103

Objective 3: Operation Observations

Availability Targets Largely Met

Another service level agreement between the ACE Support Team and CBP

requires the system to execute all transactions within 6 seconds 95 percent of the

time. The ACE Support Team reports that ACE met this requirement on all but 16

days between January 1, 2005, and January 27, 2006, and no incidents have been 

reported since August 5, 2005.

For each of the 16 days that the system did not meet the agreement, the ACE

Support Team identified the root cause. For instance:

• eight incidents were due to a problem with a program that measures network 

performance, which has since been addressed through changes to

operational procedures, and

• one incident was caused by an improperly configured server that has since 

been reconfigured.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 116 GAO-06-580 Customs Modernization

104

Objective 3: Operation Observations

Truck Processing Times Vary

Processing times for trucks crossing the border at key ports vary.

CBP has identified more efficient truck processing at the ports as an expected ACE

benefit. To ascertain whether the benefit is being realized, it also defined truck 

processing time as the performance measure to be used, and it set a performance 

target of reducing processing times at the ports by 6 percent in fiscal year 2005.

However, at the two ports for which CBP established baselines to measure truck 

processing against performance targets in fiscal year 2005, CBP reports that truck

processing times have actually increased. For example:

• At Pembina, North Dakota, by the end of fiscal year 2005, processing time

had increased by about 14 percent. CBP officials attributed this increase to a 

lack of user proficiency and confidence with ACE.

• At Nogales, Arizona, by the end of fiscal year 2005, processing time had 

increased by about 70 percent. CBP officials attributed this increase to ACErelated changes to booth operations, such as the new requirement to enter 

empty trucks in the system.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 117 GAO-06-580 Customs Modernization

105

Objective 3: Operation Observations

Truck Processing Times Vary

According to CBP officials, widespread truck processing efficiency gains will not be

realized until a majority of truck manifests are submitted electronically, which is not 

expected to occur until use of electronic manifests becomes a requirement in early

fall 2006. Nevertheless, since fiscal year 2005, CBP reports that some ports have

experienced processing improvements. For example:

• At the Ambassador Bridge in Detroit, Michigan, processing time decreased by 

approximately 27 percent between October 3 and December 15, 2005. CBP 

officials attributed this decrease to improvements in the quality of training and 

a new user interface toolbar feature.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 118 GAO-06-580 Customs Modernization

106

Objective 3: Operation Observations

Help Desk Improvements

Long-standing help desk limitations are being addressed.

According to an independent technology research firm,

1 effective help desk 

services include providing users timely updates on the status of their requests; 

conducting user satisfaction surveys; and establishing, collecting, and reporting 

operational metrics, such as the number of requests resolved during the initial call

to the help desk. 

The current ACE help desk does not perform all the practices associated with an 

effective help desk. To its credit, the existing help desk does, for example, monitor

and measure such activities as the number of requests that are resolved during the 

initial call to the help desk, the number of calls that are not received because the 

user hangs up, and the average time it takes to answer a help desk call. However,

it does not

• collect data and inform users on the status of their help desk requests or 

• survey users on their satisfaction with help desk services. 

1 Chip Gliedman, Thirty-One Best Practices for the Service Desk (Forrester Research, Inc., 2005).
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 119 GAO-06-580 Customs Modernization

107

Objective 3: Operation Observations

Help Desk Improvements

CBP has long recognized the limitations of the ACE help desk. In January 2003, it 

decided to implement a new system that would provide greater help desk

capabilities. However, the first phase of the new system was not implemented until

about 3 years later (February 2006). According to CBP officials, the delay was due 

to competing demands for limited resources. 

• The first phase is to enable users to check the status of their existing requests

and to resolve certain problems without calling the help desk. In addition, the

new system is to automatically e-mail users with a notification of resolution, 

which is to provide a link to a customer satisfaction survey. 

• The second phase is to include more advanced functionality such as allowing 

users to open and update their own help requests. CBP is working to develop

a schedule for the second phase.

In addition to the new help desk service, CBP has established an ACE Portal

Support Center to provide additional support to CBP and the trade community on

nontechnical issues: for example, submitting and processing electronic truck

manifests, setting up and using accounts, generating reports, and resetting 

passwords. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 120 GAO-06-580 Customs Modernization

108

Objective 3: Operation Observations

Usage Lower Than Expected

Usage by CBP and the trade is lower than expected.

CBP and trade usage of deployed ACE releases has been lower than expected. 

Specifically: 

• The goal for fiscal year 2005 was for 11 percent of CBP employees to use 

ACE. However, as of the end of the fiscal year, 8 percent were using it. 

According to CBP officials, they are rethinking this goal to recognize that not 

all CBP employees have a need to use ACE.

• The goal for fiscal year 2005 was for 20 percent of all monthly payments of 

fees and duties to be collected using ACE. However, as of the end of fiscal

year 2005, ACE collected about 11 percent of the total fees and duties. To

increase ACE use in paying fees and duties, CBP has

• eliminated its requirement for importers paying their way to be members of

the Customs-Trade Partnership against Terrorism (this organization is 

focused on developing, enhancing, and maintaining effective security

practices throughout the trade industry) and

• eliminated some of the paperwork and other requirements that have since 

been deemed unnecessary. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 121 GAO-06-580 Customs Modernization

109

Objective 3: Operation Observations

Usage Lower Than Expected

The following graph depicts the expected and actual percentage of ACE-collected

fees and duties for fiscal year 2005.

Fiscal Year 2005 Expected versus Actual Percentage of Duties and Fees 

Collected Using ACE
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 122 GAO-06-580 Customs Modernization

110

Objective 3: Operation Observations

Usage Lower Than Expected

• The goal for fiscal year 2005 was for 5 percent of all manifests1 to be filed 

electronically using ACE. However, the actual percentage filed in this manner

was less than 1 percent. CBP officials attributed this low percentage to the

trade community’s reluctance to invest resources to change, and the fact that

electronic manifests must be submitted in advance of a truck’s arrival. They 

also said that even though the goal for fiscal year 2006 is 20 percent, a 

significant increase in electronic manifests is unlikely to occur until it is 

mandatory for the trade to use this method, which is not expected to occur 

until early fall 2006. 

The following table shows CBP’s progress towards reaching its fiscal year 

2005 and 2006 electronic manifest goals.

1 Electronic manifests provide truck information such as driver/passenger data, vehicle data, and shipment details to CBP

officers.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 123 GAO-06-580 Customs Modernization

111

Objective 3: Operation Observations

Usage Lower Than Expected

CBP’s Limited Progress Towards Reaching Fiscal Year 2005 and 2006 

Electronic Manifest Goals.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 124 GAO-06-580 Customs Modernization

112

Objective 3: Operation Observations

User Satisfaction Low

User satisfaction was reported as low.

In February and March of 2005, a CBP user satisfaction survey was conducted that

covered, among other things, CBP IT systems, including ACE. Of the 187 

respondents, 39 percent indicated that they are dissatisfied or very dissatisfied with

ACE. The reason most often given for this response was that the system was not

easy to use. For example, according to the survey responses, ACE required 

officers to navigate through several screens in order to process each truck.

Other ACE user concerns identified include the following: 

• Passwords did not allow users to access the system.

• Response times were slow.

• Initial training of users was not adequate.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 125 GAO-06-580 Customs Modernization

113

Objective 3: Operation Observations

User Satisfaction Low

In response to the ACE survey results, CBP officials are developing a prioritized list 

of recommendations for improving user satisfaction, as well as a strategy for

surveying CBP and trade users at several ports at which Release 4 has been

deployed. The goal is to gain further insights into users’ satisfaction and to identify

potential areas for improvement. According to CBP, they plan to start surveying the 

ports in the spring of 2006.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 126 GAO-06-580 Customs Modernization

114

Objective 3: Operation Observations

Performance Targets Not Realistic

Performance targets are not always realistic.

Meaningful measurement of operational performance requires, among other things, 

realistic performance targets against which to gauge results. However, defined 

ACE performance targets are not always realistic. For example: 

• The performance target in fiscal year 2005 for ACE usage was that 11 percent 

of all CBP employees would use ACE. However, many CBP employees will

never need to use the system. Thus, the defined performance target does not

reflect this. CBP officials stated that they plan to redefine this measure to

focus on CBP employees who have a reason to use ACE. 

• The performance target for fiscal year 2005 for truck processing was a 6 

percent decrease in truck processing times at each port. However, each port 

varies in terms of truck volumes, operational hours, cargo and antiterrorism

activities, and port policies. A single performance goal for every port does not

recognize these differences.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 127 GAO-06-580 Customs Modernization

115

Objective 3: Operation Observations

Performance Goals Not Realistic

• According to CBP officials, the fiscal year 2005 and 2006 targets for 

processing electronic manifests (5 and 20 percent, respectively) were 

arbitrarily set. They stated that until electronic filing of manifests is required—

which is expected to take place in early fall 2006—it is unlikely that there will 

be any significant increase in the rate of electronic submissions.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 128 GAO-06-580 Customs Modernization

116

Objective 3: Operation Observations

Benefits and Measures Not Aligned

Goals, expected mission benefits, and performance measures are not fully 

defined and adequately aligned.

The Clinger-Cohen Act and associated OMB guidance1 require the use of effective 

IT management practices, including measuring the contributions of IT investments

to achieving agency mission outcomes. To this end, OMB requires2 that agencies 

should, among other things,

• establish program performance goals and expected benefits;

• develop outcome-based performance measures to assess actual program

performance (i.e., achievements) against expected benefits; and

• ensure that goals, expected benefits, and performance measures are

properly aligned. 

1Clinger-Cohen Act of 1996, 40 U.S.C. 1101-11703; and OMB Circular A-130, Management of Federal Information Resources

(Nov. 30, 2000).

2 OMB Circular No. A-11, Part 7 (revised June 2005).
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 129 GAO-06-580 Customs Modernization

117

Objective 3: Operation Observations

Benefits and Measures Not Aligned

CBP has defined ACE goals, expected benefits, desired business results, and 

performance measures. The six ACE goals cited earlier are summarized below. 

• Support border security through enhanced analysis and interagency 

information sharing.

• Provide information to enable decisions before a shipment reaches the 

border as to what to target and what to expedite.

• Enable efficient collection, processing, and analysis of commercial import

and export data.

• Streamline time-consuming, labor-intensive tasks for CBP and the trade.

• Enable national account management and informed, rule-based screening 

and targeting.

• Comply with legislative mandates to improve efficiency/effectiveness and 

provide better customer service.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 130 GAO-06-580 Customs Modernization

118

Objective 3: Operation Observations

Benefits and Measures Not Aligned

CBP has also identified 23 ACE benefits, all of which relate to gains in efficiency. 

Examples of these benefits are as follows:

• importer account profile efficiency gains,

• forms processing efficiency gains,

• driver verification efficiency gains, and

• cargo release efficiency gains.

CBP has also identified 11 ACE desired business results. Examples are as follows:

• improved accuracy and timeliness of information to support threat assessment 

decisions,

• detected unfair/illegal trade activities,

• increased compliance rates, and

• improved responsiveness and adaptability to policy, statutory, and regulatory 

changes and trade volume increases.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 131 GAO-06-580 Customs Modernization

119

Objective 3: Operation Observations

Benefits and Measures Not Aligned

In addition, CBP has defined 17 performance measures. Examples are as follows:

• percentage of internal CBP population using ACE functionality to manage

trade information,

• percentage of total duties and fees paid through periodic monthly statements,

and

• percentage reduction of truck processing time.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 132 GAO-06-580 Customs Modernization

120

Objective 3: Operation Observations

Benefits and Measures Not Aligned

However, the relationships among these program goals, benefits, business results,

and performance measures have not been clearly established and are not always 

apparent. Further, not every goal has defined benefits, every benefit is defined only 

in terms of efficiency gains, not every benefit has an associated business result, 

and not every benefit and business result have associated performance measures. 

For example:

• The two ACE goals focused on homeland security do not align with any stated

ACE benefit.

• The ACE benefit of greater efficiency in processing forms does not clearly 

align to any performance measures.

• The desired business result related to improved threat assessment decisions

does not align to any ACE benefit or performance measure.

• There are 23 ACE benefits but only 17 measures for gauging performance 

relative to these benefits.

The following table illustrates the lack of clearly defined relationships among ACE 

benefits, desired business results, and performance measures for Releases 2, 3,

and 4.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 133 GAO-06-580 Customs Modernization

121

Objective 3: Operation Observations

Benefits and Measures Not Aligned

Benefits Desired business results Performance measures 

Release 2 

• Efficiency gains related to account

profile establishment and 

maintenance for importer accounts

• Efficiency gains related to account

data report preparation for importer

accounts

• Efficiency gains related to account

compliance assessment

• Enable an accounts based approach

• Provide a single interface to the trade community

• Enable modern commercial business practices of the

trade 

• Percentage of internal CBP 

population using ACE 

functionality to manage

trade information 

• Percentage of trade 

accounts with access to

ACE functionality to 

manage trade information 

Release 3 

• Efficiency gains related to account

profile establishment and 

maintenance for broker accounts 

• Efficiency gains related to company

report preparation for broker

accounts 

• Enable modern commercial business practices of the

trade 

• Improve responsiveness and adaptability to changes 

• Safeguard the revenue of the government

• Improve workload management

• Provide a single interface to the trade community

• Enable increased compliance rate 

• Percentage of total duties

and fees paid through 

periodic monthly statements

Release 4 

• Efficiency gains related to single 

sign-on to ACE system and

consolidated interface for preferred 

and standard truck e-release 

• Efficiency gains related to

conveyance and driver verification

• Efficiency gains related to CBP 

forms processing

• Efficiency gains related to in-bond 

release processing for trucks 

• Improve accuracy and timeliness of information to 

support threat assessment decisions 

• Detect unfair/illegal trade activities 

• Enable modern commercial business practices

• Enable increased compliance rates

• Improve responsiveness and adaptability changes 

• Provide a single user interface to trade community

• Improve workload management

• Integrate participating government agencies’

participation for executing regulatory requirements

• Percentage reduction of

CBP truck processing time

• Percentage of truck 

manifests that are filed

electronically at each port of 

entry

Source: DHS.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 134 GAO-06-580 Customs Modernization

122

Conclusions

The legislative conditions that the Congress placed on the use of fiscal year 2006 

ACE appropriated funds have either been partially or fully satisfied by the latest

expenditure plan and related program documentation and activities. Nevertheless,

more can be done to better address several aspects of these conditions, such as

having an approved privacy impact assessment, measuring ACE performance and 

results, ensuring architectural alignment, and employing effective IV&V practices.

Given that the legislative conditions are collectively intended to promote

accountability and increase the chances of program success, it is important that

each receives DHS’s full attention.

Also important to ACE’s success is the swift and complete implementation of the

recommendations that we have previously made to complement the legislative 

conditions and improve program management, performance, and accountability. In 

this regard, some recommendations have been addressed, while progress has 

been slow on others, such as 

• accurately reporting to the Appropriations Committees on CBP’s progress in 

implementing our prior recommendations;
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 135 GAO-06-580 Customs Modernization

123

Conclusions

• developing and implementing a strategic approach to meeting the program’s 

human capital needs;

• using criteria for exiting key milestones that adequately consider indicators of

system maturity, such as severity of open defects and the associated risks;

and 

• developing and implementing a performance and accountability framework for 

ensuring that promised capabilities and benefits are delivered on time and

within budget.

To its credit, CBP has taken several steps to stem the pattern of cost, schedule, 

and performance shortfalls that it experienced on early ACE releases. However, 

future releases are unlikely to realize the impact of these steps because revised 

ACE plans and actions are reintroducing the same pattern that led to early release 

shortfalls. This pattern includes not formally and transparently considering and

proactively addressing the risks associated with passing key release milestones

with known severe defects, building considerable overlap and concurrency in the 

development schedules of releases that will contend for the same resources, and

not performing EVM on all releases. If this pattern continues, the prospects for a 

successful program will be diminished.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 136 GAO-06-580 Customs Modernization

124

Conclusions

Although availability and responsiveness targets are largely being met and longstanding help desk limitations are being addressed, the prospects for a successful 

program nevertheless remain unclear. The true measure of ACE’s success is

arguably the mission value that it brings to CBP’s and other agencies’ trade- and

border security-related operations and users. Such value depends both on the 

operational performance of ACE and on CBP’s ability to demonstrate that this

performance is achieving program goals, delivering expected benefits, and 

producing desired business results. At this juncture, however, neither the system’s

performance nor its value is clear because of several factors: the operational

performance of deployed releases has been mixed; users’ satisfaction has been 

low; the relationships among goals, benefits, and desired business outcomes are 

not evident; and the range of measures needed to create a complete and realistic 

picture of ACE’s performance is missing.

In summary, a number of ACE activities have been and are being done well; these 

have contributed to the program’s progress to date and will go a long way in 

determining the program’s ultimate success. However, CBP needs to effectively 

address long-standing ACE management challenges along with emerging 

problems. Until it does so, ACE will remain a risky program.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 137 GAO-06-580 Customs Modernization

125

Recommendations

To assist CBP in managing ACE and increasing the chances that it will deliver

required capabilities on time and within budget and demonstrate promised mission 

benefits and results, we recommend that the DHS Secretary direct the appropriate 

departmental officials to fully address those legislative conditions associated with

having an approved privacy impact assessment and ensuring architectural 

alignment. 

We also recommend that the DHS Secretary, through CBP’s Acting Commissioner,

direct the Assistant Commissioner for Information and Technology to

• fully address those legislative conditions associated with measuring ACE

performance and results and employing effective IV&V practices;

• accurately report to the Appropriations Committees on CBP’s progress in

implementing our prior recommendations;

• include in the June 30, 2006, quarterly update report to the Appropriations 

Committees a strategy for managing ACE human capital needs and the ACE

framework for managing performance and ensuring accountability;
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 138 GAO-06-580 Customs Modernization

126

Recommendations

• document key milestone decisions in a way that reflects the risks associated 

with proceeding with unresolved severe defects and provides for mitigating

these risks;

• minimize the degree of overlap and concurrency across ongoing and future 

ACE releases, and capture and mitigate the associated risks of any residual

concurrency;

• use EVM in the development of all existing and future releases;

• develop the range of realistic ACE performance measures and targets needed 

to support an outcome-based, results-oriented accountability framework,

including user satisfaction with ACE; and

• explicitly align ACE program goals, benefits, desired business outcomes, and 

performance measures. 
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 139 GAO-06-580 Customs Modernization

127

Agency Comments

In their oral comments on a draft of this briefing, DHS and CBP officials, including 

the Executive Director of Cargo Management Systems, CBP, generally agreed with

our findings, conclusions, and recommendations and stated that the presentation

was fair and balanced. They also provided clarifying information that we

incorporated as appropriate in this briefing.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 140 GAO-06-580 Customs Modernization

128

Attachment 1

Scope and Methodology

Scope and Methodology

To accomplish our objectives, we analyzed the ACE fiscal year 2006 expenditure 

plan and supporting documentation, comparing them to relevant federal 

requirements and guidance, applicable best practices, and our prior

recommendations. We also interviewed DHS and CBP officials, ACE program

contractors, and officials at the port of Blaine, Washington. In particular, we

reviewed 

• DHS and CBP investment management practices, using OMB A-11, part 7;

• DHS and CBP certification activities for ensuring ACE compliance with the 

DHS enterprise architecture; 

• DHS and CBP acquisition management efforts, using SEI’s SA-CMM;

• CBP cost estimating program and cost estimates, using SEI’s institutional and 

project-specific estimating guidelines; 

• independent verification and validation (IV&V) activities using the Institute of

Electrical and Electronics Engineers Standard for Software Verification and 

Validation;
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 141 GAO-06-580 Customs Modernization

129

Attachment 1

Scope and Methodology

• CBP actions to coordinate ACE with US-VISIT program documentation; 

• CBP’s reorganization documentation, including the new organizational charts 

and roles and responsibilities matrix;

• ACE’s accountability framework;

• ACE’s performance using service level agreements;

• ACE’s quality, using the ACE Support Team defect data and testing results for 

Release 4 and Screening 1;

• cost and schedule data and program commitments from program

management documentation;

• CBP’s progress toward increasing usage of ACE, against established targets;

• level of user satisfaction, against survey scores; and

• reliability of performance measures, by mapping the measures to benefits.
Appendix I: Briefing to Subcommittees on 

Homeland Security, House and Senate 

Committees on Appropriations

Page 142 GAO-06-580 Customs Modernization

130

Attachment 1

Scope and Methodology

For DHS-, CBP-, and contractor-provided data that our reporting commitments did 

not permit us to substantiate, we have made appropriate attribution indicating the 

data’s source.

We conducted our work at CBP headquarters and contractor facilities in the 

Washington, D.C., metropolitan area and at the port of Blaine, Washington, from

July 2005 through March 2006, in accordance with generally accepted government 

auditing standards.
Appendix II: Comments from the Department 

of Homeland Security

Page 143 GAO-06-580 Customs Modernization

Appendix II: Comments from the Department 

of Homeland Security 
Appendix II: Comments from the Department 

of Homeland Security

Page 144 GAO-06-580 Customs Modernization
Appendix II: Comments from the Department 

of Homeland Security

Page 145 GAO-06-580 Customs Modernization
Appendix II: Comments from the Department 

of Homeland Security

Page 146 GAO-06-580 Customs Modernization
Appendix II: Comments from the Department 

of Homeland Security

Page 147 GAO-06-580 Customs Modernization
Appendix II: Comments from the Department 

of Homeland Security

Page 148 GAO-06-580 Customs Modernization
Appendix II: Comments from the Department 

of Homeland Security

Page 149 GAO-06-580 Customs Modernization
Appendix III: Contac

A

t and Staff 

cknowledgments 

Page 150 GAO-06-580 Customs Modernization

Appendix III: Contact and Staff 

Acknowledgments 

Randolph C. Hite, (202) 512-3459 

In addition to the person named above, Justin Booth, Barbara Collier, 

William Cook, Neil Doherty, Michael Marshlick, Shannin O’Neill, Tomas 

Ramirez, and Jennifer Vitalbo made key contributions to this report. 

GAO Contact 

Staff 

Acknowledgments 

(310623) 
GAO’s Mission The Government Accountability Office, the audit, evaluation and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost

is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts 

newly released reports, testimony, and correspondence on its Web site. To

have GAO e-mail you a list of newly posted products every afternoon, go 

to www.gao.gov and select “Subscribe to Updates.” 

The first copy of each printed report is free. Additional copies are $2 each. 

A check or money order should be made out to the Superintendent of 

Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. Orders 

should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, D.C. 20548 

Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548 

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Mail or Phone 

To Report Fraud,

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Public Affairs 

PRINTED ON RECYCLED PAPER