powe 

ACE Release 4 Post-Deployment Problems 

 

 

DEPARTMENT OF HOMELAND SECURITY 

Office of Inspector General 

OIG-07-54 June 2007 
Office of Inspector General 

U.S. Department of Homeland Security 

Washington, DC 20528

June 22, 2007 

Preface

The Department of Homeland Security (DHS) Office of Inspector General (OIG) was established by 

the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector General 

Act of 1978. This is one of a series of audit, inspection, and special reports prepared as part of our 

oversight responsibilities to promote economy, efficiency, and effectiveness within the department. 

This report assesses the extent to which Customs and Border Protection (CBP) is effectively 

detecting and resolving problems with Automated Commercial Environment (ACE) Release 4 after 

deployment to land border ports. It is based on interviews with employees and officials of relevant 

agencies and institutions, direct observations, and a review of applicable documents. 

The recommendations herein have been developed according to the best knowledge available to our 

office, and have been discussed in draft with those responsible for implementation. It is our hope 

that this report will result in more effective, efficient, and economical operations. We express our 

appreciation to all of those who contributed to the preparation of this report. 

 

Richard L. Skinner 

Inspector General 
Table of Contents/Abbreviations 

 ACE Release 4 Post –Deployment Problems

Executive Summary...............................................................................................................................1 

Background............................................................................................................................................2 

Results of Audit .....................................................................................................................................3 

 Recommendations............................................................................................................................6 

 Management Comments and OIG Analysis ....................................................................................6 

 

Appendices 

 Appendix A: Purpose, Scope, and Methodology .........................................................................7 

 Appendix B: Management Comments.........................................................................................8 

 Appendix C: ACE Release 4 Deployment Schedule .................................................................12 

 Appendix D: ACE Help Desk Call Volume ..............................................................................14 

 Appendix E: Flowchart of Process to Detect and Resolve ACE Issues ....................................15 

 Appendix F: Major Contributors to This Report.......................................................................17 

 Appendix G: Report Distribution...............................................................................................18 

Abbreviations 

 ACE Automated Commercial Environment 

 CBP Customs and Border Protection 

 CCB Change Control Board 

 CIO Chief Information Officer 

 CR Change Request 

 DHS Department of Homeland Security 

 GAO Government Accountability Office 

 O&M Operations and Maintenance 

 OIG Office of Inspector General 

 OFO Office of Field Operations 

 PTR Problem Tracking Reports 

 SEV Severity 

 SIT System Integration Testing 

 SWIT Software Integration Testing 

 TT Trouble Ticket 

 
OIG 

Department of Homeland Security 

Office of Inspector General 

ACE Release 4 Post–Deployment Problems

Page 1 

Executive Summary

 

U.S. Customs and Border Protection is developing a new cargo processing 

system to modernize the targeting, inspection, enforcement, border security, 

revenue collection, and trade statistics processes for all cargo entering and 

leaving the United States. The Automated Commercial Environment (ACE) 

will be replacing the current Automated Commercial System. The prime 

contractor started ACE development in August 2001 and plans to complete 

ACE by September 2011 at a total cost of $3 billion. 

The objective of the audit was to determine to what extent Customs and 

Border Protection is effectively detecting and resolving problems at the ports 

after the deployment of ACE Release 4, e-Manifest: Trucks (ACE Release 4). 

Generally, problems referred to the ACE help desk, the principal method used 

by CBP to detect and resolve problems with ACE at the land border ports, 

were effectively resolved. However, Customs and Border Protection did not 

detect and resolve some operational problems that occurred at the ports and 

did not provide adequate communication and guidance to the ports. For 

example, Customs and Border Protection did not develop sufficiently detailed 

standard operating procedures that defined the different types of examinations 

and the proper method for inputting exam results into ACE. This occurred 

because CBP did not adequately monitor port operations and follow-up with 

port personnel after deployment of Release 4. Operational problems with 

ACE 4 and inadequate communication and guidance from Customs and 

Border Protection headquarters to ACE 4 users resulted in resources wasted at 

the ports dealing with operational problems and technical difficulties. We are 

recommending that Customs and Border Protection develop procedures to 

timely monitor post-deployment operations and communicate ACE problems, 

operational fixes, and system changes to officers at the ports. Customs and 

Border Protection concurred with the recommendations and is implementing 

corrective actions. 
ACE Release 4 Post–Deployment Problems

Page 2 

Background 

U.S. Customs and Border Protection (CBP) is responsible for enforcing the 

trade laws of the United States while simultaneously facilitating legitimate 

international trade. One of the critical functions of CBP is to control cargo 

and conveyances entering and leaving the United States to prevent terrorism, 

narcotics smuggling, illegal alien smuggling, enforce trade laws, and collect 

revenue. The ability of CBP to process the growing volume of imports, while 

improving compliance with trade laws, depends heavily on successfully 

improving the trade compliance process and modernizing supporting 

automated systems. The ACE project was initiated to modernize the 

targeting, inspection, enforcement, revenue collection, and trade statistics 

processes for all cargo entering and leaving the United States. Congress 

funded ACE in response to concerns from the trade1 that federal requirements 

were outdated, burdensome, and duplicative. 

ACE is being delivered in a series of eleven incremental releases, seven 

releases for cargo processing and four releases for screening and targeting. 

The first three cargo releases are deployed and operating. CBP first piloted 

ACE Release 4 at Blaine, Washington in December 2004, and has been 

deploying the system to land border ports since April 2005. As of May 2006, 

CBP had deployed ACE Release 4 to 44 ports and plans to complete 

deployment to all 99 land border ports by June 2007. See Appendix C for the 

ACE deployment schedule as of March 2007. Furthermore, the deployment 

included 14 sub-releases to add enhancements or improve functionality for 

Release 4 from January 2006 through May 2006. 

ACE Release 4 provides an electronic truck manifest, a primary officer 

interface (the screens CBP officers use), and expedited importation 

processing. The officer interface consolidates more than seven separate cargo 

release systems to provide CBP officers with a comprehensive view of 

enforcement and transaction history data to enhance enforcement capabilities 

while simultaneously expediting the movement of trade. The release provides 

for the automated manifest processes of determining the admissibility of 

cargo, conveyance, and equipment arriving in the United States. The release 

interfaces with the Automated Targeting System to provide selectivity data to 

CBP primary officers, and other authorized users, to target high-risk 

transactions and to record and track information associated to potential or 

actual noncompliance. Release 4 also provides for carriers to provide an 

electronic manifest for trucks. The filing of an electronic manifest was 

initially voluntary, but CBP will phase in mandatory electronic manifest filing 

from January to December 2007. 

 

1

 “Trade” refers to those companies such as importers, carriers, and brokers that deal in international commerce. 
ACE Release 4 Post–Deployment Problems

Page 3 

CBP established the ACE help desk as the principal method to identify and 

resolve user problems with ACE Release 4 at the land border ports. The 

number of calls made to the help desk by Trade and CBP internal users has 

been increasing as ACE is deployed to additional ports and functionality is 

added. The ACE help desk resolved 1461 problems through telephone calls 

and issued 2642 Trouble Tickets in June 2006. See Appendix D for statistics 

on help desk call volume from November 2005 to June 2006. 

The help desk assigns a Trouble Ticket number for problems and tracks the 

ticket from issuance to resolution. Problems are either resolved immediately 

or escalated to the next level of support depending on the severity and 

complexity of the problem. Problems reported to the ACE help desk may 

result in the creation of a Problem Tracking Report (PTR) or a Change 

Request (CR). CBP uses the PTR process to correct a defect in the ACE 

application code. CBP uses the CR process to add new functionality to the 

ACE application. CBP implements PTRs and CRs about once a month. See 

Appendix E for a flowchart depicting the process for resolving ACE Release 4 

problems called to the help desk. 

CBP also detects and resolves problems through informal communication 

with the ports consisting of telephone calls, conference calls, and e-mail 

between port personnel and various groups within CBP. 

The Cargo Systems Program Office (Program Office) and Office of Field 

Operations (Field Operations) management are responsible for planning, 

organizing, directing, and controlling ACE program operations to ensure that 

ACE achieves its intended goals and objectives. The Program Office is 

responsible for the development, maintenance, and deployment of ACE. Field 

Operations provides centralized management oversight and operational 

assistance to U.S. ports. 

Results of Audit 

CBP’s detection and resolution of problems that occurred at the ports after 

deployment of ACE Release 4 is generally adequate, but we identified an area 

needing improvement. Overall, problems referred to the ACE help desk, the 

principal method used by CBP to identify and resolve problems with ACE at 

the land border ports, were effectively resolved. However, CBP did not 

identify and resolve all operational problems that occurred at the ports and did 

not provide adequate communication and guidance to the ports on system 

usage. This occurred because CBP did not adequately monitor port operations 

and follow-up with port personnel after deployment of the system. 

Operational problems with ACE Release 4 and inadequate communication 

and guidance from CBP headquarters to ACE Release 4 users resulted in 

resources wasted at the ports because of operational problems and technical 

difficulties with the system. 
ACE Release 4 Post–Deployment Problems

Page 4 

According to the Office of Management and Budget Circular A-123, 

Management’s Responsibility for Internal Control, federal managers are 

responsible for establishing and maintaining internal control to achieve 

effective and efficient operations. Internal control includes measuring, 

reporting and monitoring program performance and developing the detailed 

policies, procedures, and guidance for ACE Release 4 operations. Monitoring 

program operations includes reviewing operations to identify and resolve 

problems that affect the efficiency of cargo processing and the efficient use of 

resources. 

We identified examples of operational problems and communication issues at 

the ports of El Paso, Port Huron, and Nogales that the Cargo Systems Program 

Office and Office of Field Operations did not identify and resolve. During our 

audit, CBP conducted two border conferences at which port officials reported 

the same problems to the Program Office and Field Operations that we 

identified as well as additional concerns. Examples of problems noted 

include: 

• Port personnel develop and use unauthorized workarounds. For 

example, one unauthorized workaround used by port personnel to fix a 

problem was to delete truck drivers or shipments to amend or correct a 

manifest. This prevented the proper release of the truck and required 

the help desk to resolve the problem. A trip is the CBP processing of a 

truck crossing the border. Authorized workarounds are temporary, 

written procedures from program management that allow cargo 

processing to proceed before there is a permanent solution to a 

problem. CBP did not compile, maintain or distribute a list of 

authorized workarounds. 

• CBP does not routinely communicate upcoming system enhancements 

and fixes or information on common user errors and how to prevent 

them. CBP occasionally communicated this information to land ports, 

informally. 

• The ports do not receive timely feedback on the status of their 

problems. A GAO Report2

 also found that the ACE help desk did not 

inform users on the status of their unresolved problems. Without 

communication from CBP on the status of their problems, port 

personnel may develop unauthorized workarounds and report fewer 

problems to the help desk. 

 

2 INFORMATION TECHNOLOGY: Customs Has Made Progress on Automated Commercial Environment System, but It 

Faces Long-Standing Management Challenges and New Risks, GAO 06-580, May 2006 
ACE Release 4 Post–Deployment Problems

Page 5 

• CBP did not develop complete or sufficiently detailed Release 4 

standard operating procedures for the land ports. For example, CBP 

did not develop standard operating procedures that defined the 

different types of examinations and the proper method for inputting 

exam results into ACE. Standardized procedures ensure uniform 

processing of cargo and consistency in reporting exam results. Port 

officials at the southern border conference emphasized the need for 

standard operating procedures and guidance, including how to process 

certain shipments. 

• Port personnel do not properly use or understand the port monitoring 

activity function in ACE, which is a near real-time tool for managing 

the specific port and provides a high-level snapshot of truck release 

data for the previous 24 hours and e-manifest data for trucks arriving 

in the next 24 hours. We observed port personnel improperly using 

the port monitoring activity function to generate formal reports instead 

of using the Reports function. CBP officials at the southern border 

conference advised port personnel not to use the port monitoring 

activity function to generate formal reports. 

CBP has not adequately monitored port operations and followed-up with port 

personnel on ACE Release 4 deployment. CBP did not document any ACE 

Release 4 post-deployment reviews from April 2005 through May 2006 and 

did not provide us with documentation of plans for post-deployment reviews. 

As of May 2006, CBP had deployed ACE Release 4 to 44 ports. The first time 

CBP conducted a review at a port was in June 2006 in response to reports that 

ACE was causing significant backups in Laredo. Post-deployment reviews 

include interviews with port managers and staff, observation of processes, and 

a review of reported problems, as well as a written report to management that 

describes the problems affecting the ports and makes recommendations to 

address problems. 

CBP did not hold a formal conference to obtain feedback from the ports and 

identify ACE Release 4 problems with port officials until more than one year 

after ACE deployment. The CBP held a conference for southern border ports 

on June 14, 2006, and another for northern border ports on July 18, 2006. The 

two border conferences to address post-deployment issues were positive steps 

to improve management oversight and resulted in action items to improve the 

ACE Release 4 implementation

The operational problems at the ports and inadequate communication and 

guidance to the ports have resulted in an inefficient use of resources. Port 

personnel and the ACE help desk reported spending considerable time and 

effort resolving ACE Release 4 problems. For example, the three ports we 

visited devoted time and effort attempting to use the ACE reports function 

but, ultimately, the ports continued generating manual reports or reports from 
ACE Release 4 Post–Deployment Problems

Page 6 

legacy systems. The ACE help desk had to repeatedly research and resolve 

problems relating to user errors and unauthorized workarounds as well as to 

follow-up on the status of open problems. CBP officers at the ports took 

additional time to process cargo because they did not have guidance on 

releasing certain types of shipments or they developed unauthorized 

workarounds. One port developed their own standard operating procedures 

for inputting examination results, including new exam definitions. Without 

standard operating procedures from Field Operations, cargo processing and 

examination results may not be uniform across all ports. 

Recommendations 

Recommendation 1:

The Commissioner of CBP should develop plans and procedures to timely and 

routinely monitor post-deployment operations, including post-deployment 

reviews, for Release 4 ports and future ACE releases. 

Recommendation 2:

The Commissioner of CBP should develop plans and procedures to timely 

communicate ACE problems, operational fixes, and system changes to CBP 

officers at the ports. 

Management Comments and OIG Analysis: 

In February 2007, CBP began monthly northern and southern border 

conference calls to identify and resolve operational problems related to ACE. 

CBP plans to establish a template, a schedule and a core group of managers to 

conduct post deployment reviews. CBP plans to complete the first round of 

post deployment reviews by December 31, 2007. 

An internal web site and email box provides CBP officers a quick link to 

procedures and frequently asked questions. Interim field instructions were 

disseminated to ACE ports in September 2006. CBP plans to issue standard 

operating procedures by September 1, 2007. 

The actions taken and planned by CBP satisfy the intent of our 

recommendations. 
Appendix A 

Purpose, Scope, and Methodology 

ACE Release 4 Post–Deployment Problems

Page 7 

Purpose, Scope, and Methodology

The objective of the audit was to determine to what extent CBP is effectively 

detecting and resolving problems at the ports after the deployment of ACE 

Release 4, e-Manifest: Trucks. We performed the audit at various CBP 

locations within the Washington, D.C., area and land border ports of entry at 

El Paso, Texas; Port Huron, Michigan; and Nogales, Arizona. 

To determine the process used by CBP to detect and resolve problems called 

to the help desk, we interviewed CBP officials and reviewed the policies and 

procedures for the help desk, Change Requests, and Problem Tracking 

Reports. We created a flowchart depicting the detection and resolution 

process when a user calls the help desk, found in Appendix C. 

To test the effectiveness of the resolution process for problems reported to the 

help desk, we examined the Trouble Ticket logs for the three ports we visited. 

We judgmentally selected 33 problems based on the description in the Trouble 

Ticket logs. We initially selected at least one Trouble Ticket for each of the 

33 problems. However, we selected additional Trouble Tickets for some 

problems based on the frequency of occurrence, complexity of the problem or 

because interviews with port personnel identified the problem as significant. 

Therefore, we selected 72 Trouble Tickets for the 33 issues. CBP tracks 

Trouble Tickets using a commercial software program. We met with CBP 

officials and observed them trace each of the 72 Trouble Tickets from 

detection through resolution. 

To identify the problems and issues at the ports with ACE Release 4, we 

reviewed the Trouble Ticket logs for the three ports we visited. We conducted 

structured interviews with CBP officers and CBP officials and observed port 

operations. We also obtained and reviewed the Laredo post-deployment 

review report and the minutes of the CBP southern and northern border 

conferences. 

We conducted the audit from April 2006 through July 2006, pursuant to the 

Inspector General Act of 1978, as amended, and according to generally 

accepted government auditing standards. 
Appendix B 

Management Comments 

ACE Release 4 Post–Deployment Problems

Page 8 
Appendix B 

Management Comments 

ACE Release 4 Post–Deployment Problems

Page 9 
Appendix B 

Management Comments 

ACE Release 4 Post–Deployment Problems

Page 10 
Appendix B 

Management Comments 

ACE Release 4 Post–Deployment Problems

Page 11 
Appendix C 

ACE Release 4 Deployment Schedule 

ACE Release 4 Post–Deployment Problems

Page 12 

State Cluster 

Deployment 

Schedule 

as of 06/2006 

Mandatory 

e-Manifest 

Date 

as of 06/2006 

Revised 

Deployment 

Schedule 

as of 03/2007 

Revised 

Mandatory 

e-Manifest 

Date 

as of 04/2007 

Washington 

Blaine, Sumas, Lynden, Point Roberts, 

Oroville, Danville, Ferry, Laurier, 

Boundary, Frontier, Metaline Falls, 

Nighthawk 

4/5/05 8/1/06 4/5/05 – 6/21/05 1/25/07 

Arizona 

Nogales, Douglas, Naco, Sasabe, 

Lukeville 7/25/05 8/1/06 7/18/05 - 7/25/05 1/25/07 

North Dakota 

Pembina, Neche, Walhalla, Maida, 

Hannah, Sarles, Hansboro 8/24/05 9/1/06 8/15/05 – 8/24/05 1/25/07 

Los Indios, Veterans International, Pharr, 

Progreso, Rio Grande City, Roma, Eagle 

Pass, Del Rio 1/17/06 10/1/06 1/18/06 – 1/27/06 4/19/07 

Bridge of the Americas, Ysleta, Presidio, 

Santa Teresa, Columbus 2/27/06-3/25/06 10/1/06 3/3/06 – 4/2/06 4/19/07 

Texas/New 

Mexico 

Colombia Solidarity Bridge, World Trade 

Bridge 4/5/06-4/26/06 10/1/06 4/22/06–5/4/06 4/19/07 

California 

Otay Mesa, Calexico, Andrade, Tecate, 

San Luis 5/1/06-5/19/06 11/1/06 5/10/06 – 5/26/06 4/19/07 

Michigan 

Ambassador Bridge, Windsor Tunnel, 

Barge Transport, Port Huron, Marine City, 

Algonac, Sault Sainte Marie 10/4/05 9/1/06 10/4/05 – 10/10/05 05/2007 

New York 

Champlain, Cannon Corners, Mooers, 

Overton Corners, Rouses Point, Trout 

River, Chateaugay, Fort Covington, 

Churubusco, Jamieson Line, Massena, 

Ogdensburg, Alexandria Bay, Peace 

Bridge, Lewiston Bridge 

6/24/06-7/21/06 9/1/06 6/24/06 – 10/5/06 05/2007 

Vermont/New 

Hampshire 

Highgate Springs, Alburg Springs, 

Alburg, Morses Line, Richford, West 

Berkshire, East Richford, Pinnacle, Derby 

Line I-91, Derby Line Rt 5, North Troy, 

Norton, Canaan, Pittsburgh, Beecher Falls 

8/5/06-9/1/06 11/1/06 10/24/06 – 11/17/06 06/2007 

North Dakota 

St. John, Fortuna, Ambrose, Carbury, 

Noonan, Dunseith, Sherwood, Antler, 

Northgate, Westhope, Portal 11/25/06-12/15/06 3/17/07 11/29/06 – 12/14/06 06/2007 

Idaho/Montana 

Eastport, Roosville, Porthill, Whitlash, 

Del Bonita, Wildhorse, Sweetgrass, 

Piegan, Willow Creek, Turner, Morgan, 

Scobey, Opheim, Raymond, Whitetail 

10/14/06-11/17/06 2/20/07 1/24/07 – 2/22/07 07/2007 
Appendix C 

ACE Release 4 Deployment Schedule 

ACE Release 4 Post–Deployment Problems

Page 13 

State Cluster 

Deployment 

Schedule 

as of 06/2006 

Mandatory 

e-Manifest 

Date 

as of 06/2006 

Revised 

Deployment 

Schedule 

as of 03/2007 

Revised 

Mandatory 

e-Manifest 

Date 

as of 04/2007 

Maine 

Ferry Point, Milltown, Lubec, Vanceboro, 

Forest City, Orient, Houlton, Monticello, 

Van Buren, Hamlin, Madawaska, Fort 

Kent, Estcourt, Limestone, Jackman, 

Coburn Gore, St. Zacharie, St. Aurelie, St. 

Pamphile, St. Juste, Fort Fairfield, Easton, 

Bridgewater 

9/9/06-10/6/06 1/10/07 3/19/07 – 5/10/07 08/2007 

Minnesota 

Roseau, Grand Portage, Pinecreek, 

International Falls, Baudette, Lancaster, 

Warroad 1/8/07-2/4/07 5/17/07 5/22/07 – 6/11/07 08/2007 

Alaska Ports Dalton Cache, Skagway, Alcan 8/12/06-8/26/06 11/30/06 6/20/07 – 6/28/07 09/2007 
Appendix D 

Help Desk Call Volume, November 2005 – June 2006 (From ACE June Program Management 

Report) 

ACE Release 4 Post–Deployment Problems

Page 14 
Appendix E 

Flowchart of Process to Detect and Resolve ACE Problems and Issues 

ACE Release 4 Post–Deployment Problems

Page 15 
Appendix E 

Flowchart of Process to Detect and Resolve ACE Problems and Issues 

ACE Release 4 Post–Deployment Problems

Page 16 
Appendix F 

Major Contributors to the Report 

ACE Release 4 Post–Deployment Problems

Page 17 

George Tabb, Director, Trade Operations and Security Division 

Gene Wendt, Audit Manager 

Carlos Berrios, Auditor In Charge 

David Porter, Auditor 

Michael Lugo, Auditor 

LaTrina McCowin, Auditor
Appendix G 

Report Distribution 

ACE Release 4 Post –Deployment Problems

 Page 18

Department of Homeland Security

Secretary 

Deputy Secretary 

Chief of Staff 

Deputy Chief of Staff 

Executive Secretariat 

General Counsel 

Assistant Secretary for Policy 

Assistant Secretary for Public Affairs 

Assistant Secretary for Legislative and Intergovernmental Affairs 

DHS GAO/OIG Liaison 

Chief Privacy Officer 

Customs and Border Protection

Commissioner, U.S. Customs and Border Protection 

CBP OIG Liaison 

Chief Information Officer 

 

Office of Management and Budget

Chief, Homeland Security Branch 

DHS OIG Budget Examiner 

Congress

Congressional Oversight and Appropriations Committees, as appropriate 

 

 
Appendix G 

Report Distribution 

ACE Release 4 Post –Deployment Problems

 Page 19

 

 

Additional Information and Copies 

To obtain additional copies of this report, call the Office of Inspector General 

(OIG) at (202) 254-4199, fax your request to (202) 254-4305, or visit the OIG web 

site at www.dhs.gov/oig. 

OIG Hotline 

To report alleged fraud, waste, abuse or mismanagement, or any other kind of 

criminal or noncriminal misconduct relative to department programs or 

operations: 

• Call our Hotline at 1-800-323-8603; 

• Fax the complaint directly to us at (202) 254-4292; 

• Email us at DHSOIGHOTLINE@dhs.gov; or 

• Write to us at: 

DHS Office of Inspector General/MAIL STOP 2600, Attention: 

Office of Investigations - Hotline, 245 Murray Drive, SW, Building 410, 

Washington, DC 20528. 

The OIG seeks to protect the identity of each writer and caller.