CUSTOMS ACE PROGRAM MANAGEMENT:

ACE Development at Risk From 

Incomplete Management Systems

OIG-02-102 July 9, 2002

Office of Inspector General

******* 

The Department of the Treasury 
Contents

ACE Program Management (OIG-02-102) Page 1

Audit Report................................................................................................... 3 

 Results in Brief.............................................................................................. 3 

 Background................................................................................................... 4 

 Findings and Recommendations....................................................................... 6 

 Management Control Programs Need To Be Fully Implemented ....................... 6 

 Recommendations................................................................................... 11 

 Integrated Baseline Reviews Not Conducted............................................... 11 

 Recommendation .................................................................................... 14 

 ACE Deliverable Processing Needs Improvement......................................... 14 

 Recommendations................................................................................... 20 

Appendices

 Appendix 1: Objective, Scope, and Methodology ........................................ 23 

Appendix 2: Status of Management Control Programs ................................ 24 

 Appendix 3 Capability Maturity Model Overview........................................ 25 

 Appendix 4: Task Orders ......................................................................... 26 

 Appendix 5: Schedule of Ten Deliverables Selected for Review .................... 28 

 Appendix 6: Management Comments ........................................................ 30 

 Appendix 7 Major Contributors To This Report .......................................... 37 

 Appendix 8 Report Distribution................................................................ 38 

Abbreviations

 ACE Automated Commercial Environment 

 ACS Automated Commercial System 

 CBA Cost Benefit Analysis 

 CMM Capability Maturity Model 

 CMO Customs Modernization Office 

 Customs United State Customs Service 

 DBR Desired Business Results 
Contents

ACE Program Management (OIG-02-102) Page 2

 eCP e-Customs Partnership 

 ELCM Enterprise Life Cycle Methodology 

 IBM International Business Machine 

 IBR Integrated Baseline Review 

 IDEAL Initiating, Diagnosing, Establishing, Acting, and Learning 

 IPT Integrated Product Team 

IT Information Technology 

 OIG Office Of Inspector General 

 OMB Office of Management and Budget 

 PBMS Performance Based Management System 

 RMS Requirements Management System 

 SA-CMM Software Acquisition Capability Maturity Model 

 SEI Software Engineering Institute 

 TAG Technology Architecture Group 

 TRC Technical Review Committee 

 
OIG Audit 

Report

The Department of the Treasury 

Office of Inspector General 

ACE Program Management (OIG-02-102) Page 3

 July 9, 2002 

Robert C. Bonner 

Commissioner 

U.S. Customs Service

The development of the Automated Commercial Environment (ACE) 

is a massive and multifaceted effort that is critical to the long-term 

success of the U.S. Customs Service (Customs) mission. Our audit 

reviewed Customs program management over ACE development 

for the first 6 months of this 4-year program. In the first 6 months 

the Customs Modernization Office (CMO) was simultaneously 

working to: form the various contractor, trade and Customs offices 

into an effective partnership; develop an integrated program 

management system, enterprise architecture, and user 

requirements; and lay the foundation of plans and processes 

necessary to manage the modernization effort. In our opinion, the 

major long-term risk to ACE development would be a continued 

emphasis on schedule over quality and completeness. 

Results in Brief

We found that the six management control systems reviewed were 

not fully implemented, the integrated baseline reviews were not 

performed, and the initial deliverables from the prime contractor 

had significant deficiencies which required re-work. Customs longterm program management plans are basically sound. Therefore, 

ACE can be successfully developed, if both Customs and its 

e-Customs Partnership (eCP): adequately staff the effort; 

communications are improved; management control systems are 

fully implemented; integrated baseline reviews are conducted; and 

the quality of future deliverables improves. We made six 

recommendations to improve program management in these areas.
ACE Program Management (OIG-02-102) Page 4

Customs concurred with our recommendations. The complete text 

of Customs response is included in Appendix 6. This audit was the 

first in a series of current and planned OIG audits of ACE 

development. The purpose of this report is to make 

recommendations to improve the management of ACE 

development. We briefed the CMO Director on the findings in this 

report on March 4, 2002. Our audit work was performed from 

August 2001 through March 2002 at Customs headquarters; the 

National Data Center in Springfield, Virginia; the Customs 

Eisenhower facility in Alexandria, Virginia; the eCP offices in 

Springfield, Virginia; and trade meetings in Crystal City, Virginia. A 

more detailed description of our objective, scope and methodology 

is provided in Appendix 1. 

INTERIM REPORT ON THE ADEQUACY OF STAFFING 

On January 15, 2002, we issued a draft interim report, which 

addressed staffing, communications, and scheduling issues. The 

report concluded that Customs did not yet have the people and 

systems in place to adequately manage the development of ACE. 

Customs accepted our recommendations and the final interim 

report (OIG-02-058) was issued on March 4, 2002. In response to 

our interim report, the CMO will develop a human resources 

strategy by September 30, 2002. Further, the CMO has defined a 

new organizational structure for managing the developmental 

phases of the Modernization program that more than doubles the 

government positions. Additionally, prior to proceeding in each of 

the subsequent 6 releases, the Customs risk environment will be 

re-assessed and mitigation strategies will be developed to meet 

cost and quality targets. This report and the interim report, taken 

together, represent the results of our first audit. This report will 

not repeat the findings and recommendations in the interim report. 

Background

The mission of Customs is to ensure that all goods and persons 

entering and exiting the United States do so in accordance with all 

United States laws and regulations. The Customs Modernization 

Program is a 15-year, $5 billion effort to modernize Customs 

automated systems. Also, the Modernization Program will develop 
ACE Program Management (OIG-02-102) Page 5

and document repeatable processes, and update the Enterprise Life 

Cycle Methodology (ELCM). The ability of Customs to process the 

growing volume of imports while improving compliance with trade 

laws depends heavily on successfully improving the trade 

compliance process and modernizing supporting automated 

systems. Therefore, the first effort is to replace the Automated 

Commercial System (ACS) with ACE to modernize Customs 

commercial programs. According to Customs, ACE will not only 

replace ACS, but will transform the way Customs does business. 

ACE will enable Customs to process and monitor import and export 

shipments and related trade activity more efficiently through trade 

accounts versus individual transactions. ACE will enable Customs 

to release cargo more efficiently by integrating international law 

enforcement intelligence, commercial intelligence, and data mining 

results to focus efforts on high-risk importers and accounts. The 

processes, management, and ELCM developed for ACE will be used 

for subsequent modernization efforts. 

On April 27, 2001, Customs awarded a 5-year, $1.3 billion 

contract to eCP, a coalition of contractors led by International 

Business Machine (IBM) Global Services. While the initial work on 

ACE started on August 13, 2001, negotiations on the first three 

task orders were not completed until September 28, 2001. The 

first three task orders under the contract were: Task 1, Program 

Management; Task 2, Enterprise Engineering; and Task 3, 

Requirements and Planning. Currently, Customs plans to 

implement ACE over the next 4 years. 

An Executive Steering Committee, Chaired by the Commissioner of 

Customs, provides oversight and guidance to the CMO in 

implementing the Modernization Program. The CMO is the program 

office responsible for ACE development. The support contractors 

MITRE and Robbins-Gioia conduct much of the work of the CMO. 

MITRE is an independent not-for-profit corporation chartered to 

work in the public interest. MITRE operates a Federally-funded 

research and development center for the Treasury. MITRE assists 

the CMO with the strategic, engineering, and architecture 

functions. MITRE also supports risk management, and the 

independent verification and validation of deliverables. RobbinsGioia is a consulting company that provides program management 
ACE Program Management (OIG-02-102) Page 6

services to government agencies and provides strategic 

implementation and support services for complex, high-risk 

business and technical endeavors. Robbins-Gioia provides 

significant program management support to the CMO. RobbinsGioia also assists the CMO with executive level support, integration 

leadership, business leads, team leads, and team experts. 

Findings and Recommendations

Finding 1 Management Control Programs Need To Be Fully 

Implemented

There are a number of management programs necessary to manage 

the development of ACE. We selected six programs for review 

that, when taken together, would establish a critical management 

control structure. If properly implemented and integrated, these 

programs should provide Customs with a comprehensive overall 

management control system to monitor the efficiency and 

effectiveness of ACE development. The CMO issues plans to 

provide the high-level guidance for these programs. For each plan 

the CMO establishes a number of processes to implement the 

plans. Our audit evaluated the implementation of the six selected 

programs by reviewing the requirements stated in the plans and the 

institutionalization of the described processes. 

The six management control programs that we reviewed were not 

fully implemented as of February 28, 2002. Of the six programs, 

three had approved plans, however, the related processes were 

completed for only one of the six programs (Appendix 2). All 

six programs are essential to the success of ACE development. 

However, we could not evaluate the integration and effectiveness 

of the control programs since they had not been fully implemented. 

Staffing of the management control programs was not sufficient to 

fully implement the programs timely. Further, not all processes 

have process owners assigned to monitor process procedures and 

ensure that they work. 

As a guideline for managing the development of ACE, Customs 

adopted the Software Engineering Institute’s (SEI) Software 
ACE Program Management (OIG-02-102) Page 7

Acquisition CMM (SA-CMM)1 and the SEI’s IDEAL SM (Initiating, 

Diagnosing, Establishing, Acting, and Learning) model. The CMO is 

currently a CMM Level 1 organization that, with the assistance of 

the prime and support contractors, is attempting to become a 

Level 2 organization this year and a Level 3 organization by the end 

of ACE development in 2005 (See Appendix 3 for an explanation of 

the CMM levels). 

Risk Management Program

Of the six programs we reviewed, risk management was the most 

complete. As of February 2002, there were five procedures being 

drafted to support risk processes. 

A Risk Management program provides early detection of potential 

risk, thus allowing management to mitigate or develop contingency 

plans prior to the risk materializing. The Risk Management program 

is guided by several policies. The Clinger-Cohen Act of 1996 (also 

known as the Information Technology (IT) Management Reform 

Act, Division E of Public Law 104-106) requires agencies to have a 

process for assessing and managing the risks of their information 

technology acquisitions. Office of Management and Budget (OMB) 

Circular A-130 requires agencies to establish and maintain a capital 

planning and investment control process that identifies and 

mitigates risks associated with each information system. The 

Federal Acquisition Regulation 39.102, states, in part, that: 

(1) agencies should analyze risks, benefits, and costs prior to 

contracting for IT; and (2) contracting and program officials are 

jointly responsible for assessing, monitoring, and controlling risk 

when selecting projects and during program implementation. 

MITRE is implementing the risk program processes for Customs. 

However, two processes need additional emphasis: 

• The development of contingency plans. 

• The identification of interdependencies within projects. 

 

1

 Capability Maturity ModelSM is a service mark of Carnegie Mellon University, and CMM is registered in 

the U.S. Patent and Trademark Office. 
ACE Program Management (OIG-02-102) Page 8

As the web portal was not operational, the CMO risk database was 

only accessible to MITRE and not to others in the organization. 

The combination of the CMO and eCP risk databases on the web 

portal will facilitate communication of risks to other managers. 

Also, a database for lessons learned was still needed. Points of 

contact for the processes were assigned when a risk needed 

mitigation, however, there were not always process owners 

officially assigned for managing a process. 

Independent Verification and Validation Program

On March 11, 2002, the CMO Director stated that the plan for this 

program was in draft. Therefore, we could not assess the 

implementation of the program. The purpose of the program is to 

provide independent assurance that Customs receives deliverables 

that meet contract requirements. MITRE has a process in place for 

the review of documents delivered from the prime contractor, but 

the process is being revised based on lessons learned. Processes 

for reviewing deliverables for software development and 

commercial-off-the-shelf software were still being developed. 

Process Improvement Program

The purpose of the process improvement program is to ensure 

processes are institutionalized, including key process areas and to 

ensure support to the organization by the continual improvement of 

processes. Robbins-Gioia is responsible for the implementation of 

this program. The CMO is reviewing the draft plans and processes, 

which will guide the implementation of the Process Improvement 

Program. As of February 28, 2002, not all processes have process 

owners, which makes ensuring if processes have been 

institutionalized more time consuming. 

Performance Measurement Program

The performance measurement program is still in the planning 

stage. The program encompasses the development, 

implementation, and institutionalization of metrics in support of the 

enterprise-wide Customs Modernization Program. In this effort, 

Customs will use performance measures to assess the degree to 
ACE Program Management (OIG-02-102) Page 9

which the eCP meets desired business outcomes and measurable 

objectives. The desired business outcomes and objectives form the 

basis for the award fee plans. Performance measures will be 

selected and agreed to on a task order basis. 

To develop the program, Customs approved metric plans and 

assigned Robbins-Gioia the responsibility to implement the 

program. The CMO Program Control Office has the primary 

responsibility for program metrics. The primary metrics processes 

are tailoring, applying, evaluating, and implementing. The applying, 

evaluating, and implementing processes need to be in place for the 

program to be fully implemented and help management achieve its 

metrics objectives. During our audit, the performance measures 

program had one Robbins-Gioia employee responsible for 

implementing required processes. However, the employee 

performed multiple duties related to other program control areas. 

In early December 2001, the employee started gathering metrics 

relating to risk, issues, cost, and scheduling. By February 28, 

2002, further tailoring of metrics took place. However, the metrics 

were not being shared with other teams in the organization since 

they were still in development. By not sharing data, 

interdependencies may not be considered, which could delay this 

and other programs. 

Quality Assurance Program

As of February 28, 2002, the program’s five main processes have 

not been implemented. The Quality Assurance program provides 

CMO management with objective oversight into modernization 

processes and their associated work products and services to 

support delivery of high-quality products and services. Also, the 

program helps to establish and improve the overall quality of 

CMO’s processes, work products and services. Robbins-Gioia is 

responsible for the implementation of this program. During our 

audit, only one contract person was working, part time, on 

implementing the processes. Also, not all processes had process 

owners to perform Quality Assurance Program requirements. 
ACE Program Management (OIG-02-102) Page 10

Award and Incentive Fee Program

As of February 28, 2002, the CMO has made an effort to 

implement the program. However, implementation was delayed 

because the draft Award Fee Plan was not approved until 

January 22, 2002, nearly 5 months into the contract. This also 

delayed the implementation of processes by the Business 

Management Team. A single award fee pool was established to 

motivate superior technical quality, performance at or under cost 

and on or ahead of schedule, and overall program management. 

Having a single award fee pool will enable the prime contractor’s 

performance to be evaluated against the total program 

performance. 

The prime contractor participated in early October 2001 in drafting 

the Award Fee Plan for Period 1 of the contract. Period 1 started 

on August 13, 2001, and ended on May 12, 2002 (9 months), 

with an award pool of $1,000,000. As of March 8, 2002, the 

CMO had not performed interim contractor performance reviews. 

The purpose of interim reviews is to give enough time for the 

contractor to correct any deficiencies prior to the final 

determination of the award fee amount. Therefore, Customs lost 

an opportunity to discover deficiencies and communicate them to 

the contractor earlier. 

Conclusion

In our opinion, the implementation and integration of effective 

management control programs is essential for Customs to 

adequately manage the development of ACE. The six management 

control programs need to be implemented prior to the development 

of software in Task Order 4. Due to the importance of these six 

programs, efforts must be focused on expediting their 

implementation. 
ACE Program Management (OIG-02-102) Page 11

Recommendations

The Commissioner of Customs should ensure that: 

1. A schedule is established to expedite the complete 

implementation of all six management control programs. 

Management Comment. Customs concurs with our 

recommendation and has taken actions to further implement the 

management control programs. Customs has scheduled the last 

of the six programs to be completed by March 2003. 

OIG Comment. The actions taken and planned by Customs 

satisfy the intent of our recommendation. 

2. Staffing is adequate to fully implement the programs properly, 

including the assignment of process owners to the specific 

processes within each of the programs. 

Management Comment. Customs concurs with our 

recommendation and will identify all process domain owners by 

August 2002. 

OIG Comment. The actions taken and planned by Customs 

satisfy the intent of our recommendation. 

Finding 2

Integrated Baseline Reviews Not Conducted

The contract Task Order 1-2, titled Integrated Baseline Review 

(IBR), required IBR’s to be performed by October 13, 2001. 

Subsequently, Customs revised the due date to January 18, 2002. 

However, as of March 8, 2002, an IBR had not been performed. 

The reason is partially due to Customs and eCP working to 

establish an agreed upon format and schedule for an IBR review. 

The failure to conduct required IBR’s left Customs without a critical 

tool to identify problems. 

The IBR is performed to ensure that reliable plans and performance 

measurement baselines are established to capture the entire scope 
ACE Program Management (OIG-02-102) Page 12

of work and are consistent with contract schedule requirements. 

Also, IBR’s are performed to ensure that adequate resources are 

assigned to complete program tasks. An IBR is a program 

manager-led survey of the contractor’s work plan, or contract 

performance measurement baseline. The ACE Program technical 

team ascertains the work plan executability, and identifies risk 

areas. An IBR includes the following steps: 

! Identify plan 

! Identify schedule 

! Investigate resource loading 

! Investigate performance measurement approach 

! Investigate communication approach 

! Investigate baseline control 

! Investigate forecasting 

! Identify risks 

! Level expectations 

An IBR is a critical in-depth review of the planned work, which 

should be conducted early in the project. Delays in conducting an 

IBR are generally a warning sign that the contractor is not ready or 

is following an incorrect approach. An IBR is a joint effort where 

the contractor gains insight into its customer, and the customer 

gains insight into its contractor and an early warning of deviations 

from plans. It is very important to perform an IBR because it 

establishes a baseline to monitor how well eCP is managing: 

(1) agreed upon items to be accomplished; (2) when items will be 

done; (3) how much items will cost; (4) risk; and (5) the variance 

threshold of 10 percent. Without baseline reviews, the progress of 

deliverables cannot be accurately assessed, and used for better 

planning as well as for timely action on contingencies and delays. 

On January 31, 2002, Customs held a meeting to discuss the IBR 

format with Robbins-Gioia and MITRE. During the meeting, 

emphasis was given to the importance of adherence to standards 

and government regulations such as Clinger-Cohen, OMB A-11, 

and ELCM. In addition, it was decided that the Customs IBR team 

should have the following members: 
ACE Program Management (OIG-02-102) Page 13

• IBR Team Lead – a Customs employee within the Program 

Management Office 

• Cost Analyst- required for a Performance Based Management 

System (PBMS) 

• PBMS Expert - required by OMB A-11 300.8 

• Risk Specialist – needed for risk assessments and mitigation 

plans 

• Schedule Specialist – needed for scheduling and forecasting 

• Technical Expert - more than one may be needed 

• Contracts Specialist – assures compliance with the baseline 

contract. 

On February 1, 2002, Customs Contracting Officer for the eCP 

contract presented a letter to eCP disclosing areas of concern. 

One of the areas addressed was the IBR deliverable. At the time of 

the letter the first IBR had not been scheduled and was already 

2 months past the initial due date. The impact of not having this 

important review is significant in terms of measuring work 

accomplished in the context of costs incurred. 

A February 28, 2002 response from eCP, stated that, they have 

been working jointly with Customs to establish an agreed upon IBR 

format and schedule. Also, that eCP would commit to take the 

initiative and schedule a meeting with Customs to establish the 

format, process and schedule with the ACE Program. 

IBR requirements include: 

• An evaluation of risks against projected lifecycle costs and 

benefits, as required by Clinger-Cohen. 

• Submissions consistent with OMB A-11, Section 300.8 

requirements to maintain all of the baselines, original and 

current, including tracking cost, scheduling, performance 

variances, and to establish a corrective action threshold of ten 

percent variance. In addition, the status of the effectiveness of 

IBR corrective actions, should be reported. 
ACE Program Management (OIG-02-102) Page 14

Recommendation

The Commissioner of Customs should ensure that: 

1. The IBR’s are regularly conducted for Task Orders issued after 

March 2002, as well as for Phase II of Task Order 2. 

Management Comment. Customs concurs with our 

recommendation and has conducted or scheduled appropriate 

IBR’s. All planned IBR’s are scheduled to be completed by 

June 30, 2002. 

OIG Comment. The actions taken and planned by Customs 

satisfy the intent of our recommendation. 

Finding 3 ACE Deliverable Processing Needs Improvement 

The processes for reviewing ACE deliverables need improvement to 

be more timely and effective. Customs and eCP were under tight 

time and resource constraints to complete the deliverables for the 

first three Task Orders. To attempt to meet the established 

milestones, as well as mitigate obstacles, deliverables were being 

submitted to Customs that needed significant rework to reach an 

acceptable state. The submission of deficient deliverables resulted 

in a review process, usually involving several iterations, that was 

labor intensive and time consuming. Further, since some 

deliverables were prepared as high level documents and others 

were conditionally accepted, there will be additional work 

necessary in the future to meet Customs needs and expectations. 

On February 1, 2002, the Customs Contracting Officer issued a 

letter to eCP listing Customs concerns with their contract 

performance. Areas of concern included staffing, quality of 

deliverables, engineering approach, requirements development, 

program reviews, and program integration. The letter stated that 

more than 75 percent of the 25 contract documents delivered to 

date have significant deficiencies requiring rework. Further, the 

letter stated that, “The eCP is responsible for developing the plans 

and processes to produce the work that meets acceptable 

standards.” On February 28, 2002, eCP responded to the criticism 
ACE Program Management (OIG-02-102) Page 15

of its performance on the Customs Modernization Contract. The 

response states, “We do recognize that the quality of the initial 

deliverables were not up to our expectations.” Factors that 

contributed to the problems included: 

• Funding 

• Scheduling 

• An inadequate eCP editing process 

• Lack of a collaborative development process 

• Lack of a mutually agreed acceptance process 

Deliverables are prepared by eCP and then submitted to Customs 

for acceptance review. Customs then has 15 workdays to review 

the deliverable and submit written comments. If a deliverable is 

not accepted, eCP has 10 workdays to respond and resubmit the 

deliverable. After the review process, the due date is the target 

date for the acceptance by Customs of a fully completed 

deliverable. 

There were 28 subtasks for eCP to complete in the first three task 

orders of the contract (Appendix 4). We selected for review a 

judgmental sample of 10 of the 50 deliverables associated with 

these subtasks (Appendix 5). Five of the selected deliverables 

were completed an average 28 days late. Of the remaining five 

selected deliverables at March 5, 2002, two were not completed, 

two were pending Customs review, and one was accepted that 

was not complete. The IBR deliverable associated with 

Task Order 1 Subtask 2 (Task 1-2) was discussed in the prior 

section of this report. The result of our review of the remaining 

nine selected deliverables follows. 

Cost Benefit Analysis Conformance Approach and Plan Task 1-3

The Cost Benefit Analysis (CBA) Conformance Approach and Plan 

deliverable was due on November 2, 2001, and accepted on 

November 8, 2001. However, there was additional work 

necessary to complete the deliverable, due to the wording of the 

deliverable proposal and incomplete clearances for eCP personnel. 

After a collaborate effort between the CMO and eCP, the 

deliverable was completed on December 14, 2001. eCP stated 
ACE Program Management (OIG-02-102) Page 16

that they were not responsible for the development of the overall 

CBA document, but just for “quantitative inputs” into the CBA. 

Also, eCP said that some of the areas that affected their work 

were not having access to data, such as, class and labor rates. 

Another factor affecting the deliverable was the lack of a formal 

Integrated Product Team (IPT). The IPT did not have a charter and 

minutes of meetings were not being maintained. The IPTs need to 

have agreed upon charters that identify responsibilities and lay the 

groundwork on how they will function. Also, documentation, such 

as minutes of meetings, is needed to support a repeatable process. 

Risk Management Process Task 1-5

The Risk Management Process deliverable was due on 

October 18, 2001, and accepted 23 workdays late on 

November 20, 2001. Our audit found that instructions on how to 

use the Risk Assessment Checklist contained incorrect references 

to tables in the Risk Management Process Document. The 

incorrect references included important tables for determining the 

likelihood of risk materializing, the significance of the risk to the 

ACE program, and tables defining the individual risk level. MITRE 

personnel stated that the error was due to a busy schedule that did 

not allow time to check everything over and over again. As a 

result, eCP will ensure table references are correct in the next 

iteration of the Risk Assessment Checklist instructions. 

Requirements Management System (RMS) Task 1-9

The RMS scheduled due date was November 13, 2001. Customs 

granted conditional acceptance of eCP’s usage of the RMS tool on 

December 4, 2001, pending an official acceptance after the 

Customs Technical Review Committee (TRC) approval. Customs 

stated that they would not accept any invoices related to the RMS 

tool, or any other tools until the TRC’s approval. The RMS tool 

was installed and configured by eCP on its own infrastructure using 

a commercial-off-the-shelf product, and available to eCP personnel 

involved with ACE requirements development. However, the 

product was not approved by the TRC, as required by the 

Technical Insertion Process and Investment Management Process 

for inserting IT products into the Technical Reference Model. As of 
ACE Program Management (OIG-02-102) Page 17

March 8, 2002, the Technology Architecture Group (TAG) was still 

evaluating the RMS tool, which then must go to the TRC for review 

and approval. 

Enterprise Infrastructure Task 2-4

The Enterprise Infrastructure deliverable has two due dates. The 

first due date is on February 13, 2002 for the draft version, and 

the second, on April 12, 2002 for the final version. As of March 

8, 2002, comments on the draft were still in process. According 

to eCP, the deliverable was affected by not having completed 

background investigations for key personnel. This prevented eCP 

personnel from having access to “As Is” infrastructure and 

inventory information. On January 10, 2002, eCP had identified a 

risk that if clearances were not in place for Task Order 2 personnel, 

the schedule and completeness of deliverable products would be 

impacted. To work around the clearances, eCP personnel 

developed a higher level of theoretical architecture based on a 

reliance on work performed by MITRE. Also, eCP continued to 

capture “As Is” documentation from various sources and had it 

validated by TAG and MITRE. 

Customs has already previewed the deliverable document and 

verified that it is being written at a very high level, needing more 

work and clarification. Accepting high-level documents will require 

Customs to gather additional detail later when it could be more 

expensive and more difficult to change the documents. 

CMO Web Portal Task 2-7

The CMO Web Portal was not delivered to Customs. While the due 

date was December 20, 2001, Customs has not yet received an 

operational web portal. The failure to establish a portal has 

hampered communications between the various contractors and 

Customs. Specifically, the delay in the installation of the CMO 

Web Portal has hindered ACE collaboration efforts and the sharing 

of information, and has also limited the Program Management 

Reporting System to hardcopy distribution lists. The CMO Web 

Portal slippage is due to backordered products, lack of security 

clearances, support infrastructure, and lack of security policy. 
ACE Program Management (OIG-02-102) Page 18

Also, eCP discovered that their own proposed system would not 

meet Customs systems security requirements for this type of 

information. On February 14, 2002, the Contracting Officer asked 

that eCP provide a schedule supporting the installation of the portal 

to include: dates; interdependencies; Customs roles; assessment 

of the impact of the late delivery to the ACE Program; plus eCP’s 

mitigation plan. The Contracting Officer stated that this 

information was due by February 22, 2002, but as of 

March 8, 2002, a response had not yet been received. 

Desired Business Results Task 3-1

The Desired Business Results (DBR) was due on 

December 21, 2001, and accepted 38 workdays late on 

February 19, 2002. On January 11, 2002, the Customs 

Contracting Officer sent a memo to eCP stating that the overall 

performance in the development of this deliverable was 

unacceptable. Customs spent considerable time reviewing several 

iterations of deliverable documents and collaborating with eCP. 

However, instead of receiving a document of acceptable quality in 

the December 21, 2001 version, Customs had 111 additional 

comments, showing that it still needed substantial work. Customs 

finally accepted the deliverable on February 19, 2002, with a 

requirement that eCP perform a comprehensive and professional 

editing of the document to correct the numerous grammatical and 

style errors. 

Transition & Sequencing & Change Management Plan Task 3-4

The Transition & Sequencing & Change Management Plan was due 

on January 30, 2002, and accepted 23 workdays late on 

March 5, 2002. Customs review of this document resulted in 

several iterations and numerous comments and responses between 

Customs and eCP. Customs conditionally accepted this plan on 

January 25, 2002, deciding that it was sufficient for proceeding 

with Increment 1, Release 1, but still needed more work for 

Increment 1 Release 2. The Plan was accepted by Customs with 

no further comments. However, in response to several Customs 

comments requesting more specific information, eCP responded 

that the deliverable was written at a high level. Further, eCP 
ACE Program Management (OIG-02-102) Page 19

stated that details Customs requested on transition needs, the 

business process, and statutory, regulatory, and policy changes 

could be provided at a later date. 

ACE CBA/Investment Strategy Task 3-7

The CBA Investment Strategy was due on January 11, 2002, and 

accepted 35 workdays late on March 5, 2002. On 

November 27, 2001, the Commissioner of Customs set a 4-year 

goal for the completion of ACE. The CBA/Investment Strategy 

delivered by eCP on November 19, 2001, was based on a 5-year 

program. Accordingly, eCP had to revalidate the pricing of the ACE 

program as a 4-year program. 

To complete this CBA/Investment Strategy deliverable under tight 

CMO time constraints, eCP stated the following actions were 

taken: 

• To collect importer trade data for the CBA, surveys were sent 

to importers on December 17, 2001, with a 5-day turnaround. 

As a result, only 20 responses were received out of the 

250 importers contacted. A CMO risk was that if a sufficient 

number of surveys were not returned, the limited survey 

information might be insufficient to develop meaningful or 

accurate cost and benefit data for the trade. eCP used this 

limited survey data anyway, along with conservative estimates 

to establish significant trade benefits for the ACE Program. Of 

the $24.2 billion in quantified benefits for ACE shown in 

Table 1 of the CBA/Investment Strategy deliverable, trade 

benefits totaled $19.2 billion. 

• The ACE CBA/Investment Strategy document contained a SEI 

Software Validation Checklist. The question on issue 4.4 of the 

checklist is if uncertainties in parameter values had been 

identified and qualified; and the question on issue 4.5 is 

whether a risk analysis had been performed, and risks that 

effect cost or schedule had been identified and documented. 

eCP’s response to both issues was that due to limitations in the 

available schedule, proper analysis and reviews were not 

conducted. Issue 5.1 inquired as to whether Customs 
ACE Program Management (OIG-02-102) Page 20

management reviewed and agreed to the values for all 

descriptive parameters before costs were estimated. eCP 

responded that a detailed review with Program Management 

was not possible due to schedule limitations. 

Top Level Ace Program Plan Task 3-8

The ACE Program Plan deliverable was due on January 30, 2002, 

and accepted 23 days late on March 5, 2002. The eCP response 

to some of the issues raised by Customs comments was that they 

would be resolved at a later date. These issues included: 

inconsistencies across all eCP documents; the organization section 

did not provide compelling evidence that there is a management 

structure in place to effectively manage the ACE Program; the 

Customs interfaces and lines of communication section does not 

provide sufficient information to understand how the eCP plans to 

interface with Customs; and the Automated Targeting System is 

not included in the table from “As-is” to “To-be”. While eCP was 

aware of these issues, they decided to move ahead with the new 

task orders rather than correcting the old ones. 

Recommendations

The Commissioner of Customs should ensure that improvements 

are made in the processing of deliverables. Specifically, 

1. Performance measures based on the deliverable quality issues, 

noted by Customs and this report, should be finalized. The 

measures should have a direct significant effect on the 

determination of the award fee amount. 
ACE Program Management (OIG-02-102) Page 21

Management Comment. Customs concurs with our 

recommendation and is using quality metrics for assessing 

performance in period 1. For period 2, the metrics will be 

reassessed by the Award Fee Working Group on a continuing 

basis. Also, for period 2, Customs has added a fifth outcome, 

Planning. 

OIG Comment. The actions taken by Customs satisfy the intent 

of our recommendation. 

2. The deliverable review process should ensure that products 

accepted are complete, edited, meet acceptable standards and 

written at a level needed by users. 

Management Comment. Customs concurs with our 

recommendation and conducted a joint lessons learned session 

with eCP in late February. The session resulted in 28 action 

items that should directly contribute to better quality 

deliverables. The actions are scheduled to be completed by 

September 30, 2002. 

OIG Comment. The actions taken and planned by Customs 

satisfy the intent of our recommendation. 

3. The CMO should establish a formal system to track items and 

tasks that are not completed on accepted deliverables, but are 

agreed to be performed by eCP at a later date. 

Management Comment. Customs concurs with our 

recommendation and will formalize an administrative procedure 

to track open issues by July 30, 2002. 

OIG Comment. The actions taken and planned by Customs 

satisfy the intent of our recommendation. 
ACE Program Management (OIG-02-102) Page 22

* * * * * * 

We appreciate the courtesies and cooperation provided to our staff 

during the audit. If you wish to discuss this report, you may 

contact me at (202) 927-5171 or Gene Wendt, Audit Manager at 

(713) 706-4611 ext. 242. 

/s/ 

Edward G. Coleman 

Director, Information Technology Audits 

Office of Inspector General 
Appendix 1 

Objective, Scope, and Methodology 

ACE Program Management (OIG-02-102) Page 23

The objective of this audit was to determine whether ACE Program 

Management had been appropriately planned and was in 

compliance with relevant laws, Treasury Directives, and sound 

business practices. To accomplish this objective, our audit work 

included the following: 

• A CMO staffing analysis and review of the ACE development 

schedule. 

• Interviews conducted with various Customs and contractor 

officials at Customs Headquarters, National Data Center in 

Springfield, Virginia, the eCP facility in Springfield, Virginia, and 

Customs Eisenhower facility in Alexandria, Virginia. 

• A review of the following six management control programs to 

evaluate their implementation, effectiveness and integration: 

1. Risk Management 

2. Independent Verification and Validation 

3. Process Improvement 

4. Performance Measures 

5. Quality Assurance 

6. Award and Incentive Fee 

• A judgmental sample of 10 contract deliverables was selected 

to evaluate the adherence to schedules, reviews conducted by 

Customs, acceptance process, and communications between 

Customs and contractors regarding the deliverables. We did not 

conduct a technical assessment of the deliverables. 

• Attended trade support meetings held in Crystal City, Virginia. 

Also, attended various recurring meetings held by CMO, support 

contractors, and eCP related to ACE development. 

• Coordinated our audit efforts with the General Accounting 

Office auditors. 

We conducted the audit from August 2001 through March 2002 in 

accordance with generally accepted government auditing 

standards. 
Appendix 2 

Status of Management Control Programs As of February 28, 2002 

ACE Program Management (OIG-02-102) 

 Plans Status Summary of Program Status 

Programs Approved

Risk Management 3-16-01 Partial Procedures for processes are still being developed. Emphas

contingency plans and the identification of interdependencieneeds to be accessible to other management teams and lesscaptured. 

Independent Verification and Validation 

In Draft Partial Processes for the Independent Verification and Validation pr

documents continue to be modified to improve the review pare in draft detailing processes for software development anshelf software reviews. 

Process Improvement In Draft Partial Process Action Teams chartered and working on identifying, 

processes for seven Key Process Areas. Two people are assperson is working full-time on evaluating institutionalization is working part-time on process development. Process ownmonitor, and review processes. 

Performance Measures 8-29-01 Partial Information on the program was being gathered and analyze

being performed to determine the best way to present the inaccomplished, the remaining three main processes may be im

Quality Assurance In Draft Partial One person was working part-time on implementing this pro

were still in draft. The program processes have not been imdependent on other programs being in place, such as the proassignment of process owners to assist in responsibilities of 

Award and Incentive Fee 1-22-02 Partial Award Fee plans were approved in January 2002, delaying 

processes, specifically interim reviews of contractor performthe first interim review in late March 2002, approximately 2 the first award fee evaluation period, August 13, 2001 to M
Appendix 3 

Capability Maturity Model Overview 

ACE Program Management (OIG-02-102) Page 25

SA-CMM is based on the best practices of organizations that successfully acquire 

software systems and products. SA-CMM is a model that describes the key elements 

of managing and improving the acquisition process in an organization. SA-CMM 

encompasses not just software but the broader system acquisition processes for 

systems, hardware, networks, and software. SEI’s IDEAL model forms an 

infrastructure to guide organizations in planning and implementing an effective process 

improvement program and provides a usable, understandable approach to continuous 

improvement by outlining the steps necessary to establish a successful improvement 

program. The CMM is organized into five levels each of which comprises a set of 

process goals that, when satisfied, stabilize an important component of the software 

process. The five levels can be briefly described as: 

Level 1 Initial

The software process is characterized as ad hoc, and occasionally even chaotic. Over 

commitment is characteristic of Level 1 organizations. During a crisis, projects 

typically abandon planned procedures. 

Level 2 Repeatable

Basic project management processes are established to track cost, schedule, and 

functionality. Projects implement effective processes that are defined, documented, 

practiced, trained, measured, enforced, and improvable. 

Level 3 Defined

The software process for both management and engineering activities is documented, 

standardized, and integrated into a standard software process for the organization. 

Level 4 Managed

Detailed measures of the software process and product quality are collected. Both the 

software process and products are quantitatively understood and controlled.

Level 5 Optimizing

Continuous process improvement is enabled by quantitative feedback from the process 

and from piloting innovative ideas and technologies. The organization has the means 

to identify weaknesses and strengthen the process proactively, with the goal of 

preventing the occurrence of defects. 
Appendix 4 

Tasks Orders 

ACE Program Management (OIG-02-102) Page 26

Task Order 1 

Scope 

• Establish the Program Management structure 

• Develop processes and procedures

• Period of Performance - 18 months

Subtask 1 Perform Program Management 

Subtask 2 Perform Program Planning, Control, and Scheduling 

Subtask 3 Provide Inputs to Cost-Benefit and Financial Analysis 

Subtask 4 Establish/Maintain a Program Management & 

Reporting System 

Subtask 5 Perform Risk Management 

Subtask 6 Perform Configuration Management 

Subtask 7 Perform Quality Assurance 

Subtask 8 Establish a Process Improvement Program 

Subtask 9 Perform Requirements Management 

Subtask 10 Support Organizational Change Management 

Subtask 11 Perform Communications Management 

Subtask 12 Perform Additional Program Level Activities

Task Order 2 

Scope 

• Perform Enterprise Architecture and Engineering 

• Period of Performance - 18 months 

Subtask 1 Maintain Customs ELCM 

Subtask 2 Develop and Maintain the Enterprise Architecture 

Subtask 3 Develop/Maintain Enterprise Transition Sequencing 

Plan 

Subtask 4 Update the Enterprise Infrastructure Plan 

Subtask 5 Maintain Enterprise Technology Insertion 

Subtask 6 Perform/Develop Enterprise Architecture & Engineering 

Subtask 7 Develop/Maintain the Modernization Program Web 

Portal
Appendix 4 

Tasks Orders 

ACE Program Management (OIG-02-102) Page 27

Task Order 3

Scope 

• Plan and define the requirements for the first increment of ACE 

• Customs Subject Matter Experts will work on the Integrated 

Product Teams and contribute their knowledge and expertise to 

this effort 

• Period of Performance - 6 months 

Subtask 1 Document Desired Business Results 

Subtask 2 Gather, Analyze, and Document High Level 

Requirements 

Subtask 3 Define/Document the ACE Architecture 

Subtask 4 Develop the ACS-to-ACE Transition and Sequencing 

Plan 

Subtask 5 Specify Detailed Requirements for ACE Increment 1 

Subtask 6 Prepare Project Plan for ACE Increment 1 Business 

Results 

Subtask 7 Develop Planning Estimate 

Subtask 8 Prepare Top Level Plan - trade Compliance & 

International Trade Data System 

Subtask 9 Support ACE - Specific Organizational Change 

Management 
Appendix 5 

Schedule of Ten Deliverables Selected for Review As Of March 8, 2002 

ACE Program Management (OIG-02-102) 

Deliverables Due Date Final Accepted 

by Customs 

Workdays 

Late 

Dates Deliverables/Drafts 

 Issued & Number of Customs 

Comments 

C

Task 1-2 Integrated Baseline Reviews 

1/18/02 IBR not performed IBR not 

performed 

IBR not performed 

IBR pe

Task 1-3 Cost Benefit Analysis (CBA) 

11/2/01 

Accepted 

11/8/01 

Updated 

12/14/01

Accepted 

incomplete 

deliverable

 09/27/01 27 comments on  draft 11/02/01 Final deliverable  accepted, but  updated on  12/14/01. 

CuPlaadfor 

Task 1-5 Risk Management Process 

10/18/01 11/20/01 

3 comments 

23 

 09/13/01 15 comments 10/18/01 courtesy reply 10/30/01 3 comments 

RiswiPro

Task 1-9 Requirements Management System 

11/13/01 

Conditionally 

Accepted 12/4/01 

With 10 comments

Pending TRC approval. 

12/04/01 10 comments 

Apfro

Task 2-4 Enterprise Infrastructure 

2/13/02 

(Draft) 

4/12/02 

(Final)

Comments on draft being prepared 

Comments in process 

 02/13/02 (Draft) comments  in process 

Cuwrne
Appendix 5 

Schedule of Ten Deliverables Selected for Review As Of March 8, 2002 

ACE Program Management (OIG-02-102) 

Deliverables Due Date Final Accepted 

by Customs 

Workdays 

Late

Dates Deliverables/Drafts 

Issued & Number of Customs 

Comments 

C

T.O. 2-7 CMO Web Portal 12/20/01 

Not completed Not 

completed 

CMO Web Portal not installed The 

be inCust

T.O. 3-1 Desired Business Results 

12/21/01 2/19/02 38 

09/26/01 29 comments 11/19/01 eCP withdrew this  version on 12/7/01  for additional work. 12/21/01 111 comments 01/29/02 2 comments 

Oveunacactio

T.O. 3-4 ACE Transition and Sequencing Plan 

1/30/02 

Conditionally 

1/25/02 

Accepted 

 3/5/02 

23 

01/4/02 134 comments 01/30/02 74 comments 02/12/02 No comments 

DefimethAlso 

T.O. 3-7 ACE CBA/Investment Strategy 

1/11/02 3/5/02 35 

09/26/01 16 comments 11/19/01 51 comments 01/11/02 Rejected by  Customs 1/18/02. 02/12/02 38 comments 

Survfrom resp

T.O. 3-8 Top Level ACE Program Plan 

1/30/02 3/05/02 23 

09/26/01 16 comments 11/19/01 73 comments 01/30/02 50 comments 02/12/02 No comments 

Incodocuorgaacro
Appendix 6 

Management Response 

ACE Program Management (OIG-02-102) Page 30
Appendix 6 

Management Response 

ACE Program Management (OIG-02-102) Page 31
Appendix 6 

Management Response 

ACE Program Management (OIG-02-102) Page 32
Appendix 6 

Management Response 

ACE Program Management (OIG-02-102) Page 33
Appendix 6 

Management Response 

ACE Program Management (OIG-02-102) Page 34
Appendix 6 

Management Response 

ACE Program Management (OIG-02-102) Page 35
Appendix 6 

Management Response 

ACE Program Management (OIG-02-102) Page 36
Appendix 7 

Major Contributors To This Report 

ACE Program Management (OIG-02-102) Page 37

Edward G. Coleman, Director of Information Technology Audits 

George W. Tabb, Regional Inspector General for Audit (Southern) 

Gene Wendt, Audit Manager 

Robert Edwards, Auditor-In-Charge 

Carlos Berrios, Auditor 

Eric Barnett, Referencer 
Appendix 8 

Report Distribution 

ACE Program Management (OIG-02-102) Page 38

The Department of the Treasury

Chief Information Officer 

Office of Accounting and Internal Control 

U.S. Customs Service

Commissioner 

Assistant Commissioner, Office of Information Technology 

Director, Modernization Office 

Director, Office of Planning and Evaluations 

Office of Management and Budget

Budget Examiner