Customs Service Modernization: Actions Needed to Correct Serious ACE Management and Technical Weaknesses
Published: 1999-04-13T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO discussed the Customs Service's management of its Automated Commercial Environment (ACE) system.
section_title: What GAO Found
paragraphs: GAO noted that: (1) the need to leverage information technology to improve the way that Customs does business in the import arena is undeniable; (2) Customs' existing import processes and supporting systems are simply not responsive to the business needs of either Customs or the trade community, whose members collectively import about $1 trillion in goods annually; (3) these existing processes and systems are paper-intensive, error-prone, and transaction-based, and they are out of step with the just-in-time inventory practices used by the trade; (4) recognizing this, Congress enacted the Customs Modernization and Informed Compliance Act to define legislative requirements for improving import processing through an automated system; (5) Customs fully recognizes the severity of the problems with its approach to managing import trade and is modernizing its import processes and undertaking ACE as its import system solution; (6) begun in 1994, Customs' estimate of the system's 15-year life cycle cost is about $1.05 billion, although this estimate is being increased; (7) in light of ACE's enormous mission importance and price tag, Customs' approach to investing in and engineering ACE demands disciplined and rigorous management practices; (8) such practices are embodied in the Clinger-Cohen Act of 1996 and other legislative and regulatory requirements, as well as accepted industry system/software engineering models, such as those published by the Software Engineering Institute; (9) unfortunately, Customs has not employed such practices to date on ACE; (10) GAO's February 1999 report on ACE describes serious management and technical weaknesses in Customs' management of ACE; and (11) the ACE weaknesses are: (a) building ACE without a complete and enforced enterprise systems architecture; (b) investing in ACE without a firm basis for knowing that it is a cost effective system solution; and (c) building ACE without employing engineering rigor and discipline.

section_title: Letter
paragraphs: Thank you for inviting me to participate in today’s Customs Service  oversight hearing.  My statement will focus on Customs’ Automated  Commercial Environment, better known as ACE.  Through ACE, Customs  intends to implement much needed improvements in the way it currently  enforces import trade laws and regulations and assesses and collects  import duties, taxes, and fees, which total $22 billion annually.
The need to leverage information technology to improve the way that  Customs does business in the import arena is undeniable.  Customs’  existing import processes and supporting systems are simply not  responsive to the business needs of either Customs or the trade  community, whose members collectively import about $1 trillion in goods  annually.  These existing processes and systems are paper-intensive, error- prone, and transaction-based, and they are out of step with the just-in-time  inventory practices used by the trade.  Recognizing this, the Congress  enacted the Customs Modernization and Informed Compliance Act, or  “Mod” Act, to define legislative requirements for improving import  processing through an automated system.
Customs fully recognizes the severity of the problems with its approach to  managing import trade and is modernizing its import processes and  undertaking ACE as its import system solution.  Begun in 1994, Customs’  estimate of the system’s 15-year life cycle cost is about $1.05 billion,  although this estimate is being revised upwards.  In light of ACE’s  enormous mission importance and price tag, Customs’ approach to  investing in and engineering ACE demands disciplined and rigorous  management practices.  Such practices are embodied in the Clinger-Cohen  Act of 1996 and other legislative and regulatory requirements, as well as  accepted industry system/software engineering models, such as those  published by the Software Engineering Institute (SEI).
Unfortunately, Customs has not employed such practices to date on ACE.   Our February 1999 report on ACE, upon which my testimony today is  based, describes serious management and technical weaknesses in  Customs’ management of ACE.  The ACE weaknesses are (1) building ACE  without a complete and enforced enterprise systems architecture, (2) investing in ACE without a firm basis for knowing that it is a cost- effective system solution, and (3) building ACE without employing  engineering rigor and discipline.  My testimony will address each of these  points as well as our recommendations for correcting them.  Customs  agrees with our findings, and it is committed to implementing our  recommendations.
subsections: section_title: ACE: A Brief History
paragraphs: Customs began ACE in 1994, and its early estimate of the cost and time to  develop the system was $150 million over 10 years.   At this time, Customs  also decided to first develop a prototype of ACE, referred to as NCAP  (National Customs Automation Program prototype), and then to complete  the system.  In May 1997, we testified that Customs’ original schedule for  completing the prototype was January 1997, and that Customs did not have  a schedule for completing ACE.  At that time, Customs agreed to develop a  comprehensive project plan for ACE.
In November 1997, Customs estimated that the system would cost $1.05 billion to develop, operate, and maintain throughout its life cycle.   Customs plans to develop and deploy the system in 21 increments from  1998 through 2005, the first four of which would constitute NCAP.
Currently, Customs is well over 2 years behind its original NCAP schedule.   Because Customs experienced problems in developing NCAP software in- house, the first NCAP release was not deployed until May 1998—16 months  late.  In view of the problems it experienced with the first release, Customs  contracted out for the second NCAP release and deployed this release in  October 1998—21 months later than originally planned.  Customs’ most  recent dates for deploying the final two NCAP releases (0.3 and 0.4) are  March 1999 and September 1999, which are 26 and 32 months later than the  original deployment estimates, respectively.  According to Customs, these  dates will slip farther because of funding delays.
Additionally, Customs officials told us that a new ACE life cycle cost  estimate is being developed, but that it was not ready to be shared with us.   At the time of our review, Customs’ $1.05 billion estimate developed in 1997  was the official ACE life cycle cost estimate.  However, a January 1998 ACE  business plan specifies a $1.48 billion life cycle cost estimate.
section_title: Customs Is Developing ACE Without a Complete Enterprise Systems Architecture
paragraphs: Customs is not building ACE within the context of an enterprise systems  architecture, or  “blueprint” of its agencywide future systems environment.   Such an architecture is a fundamental component of any rationale and  logical strategic plan for modernizing an organization’s systems  environment.  As such, the Clinger-Cohen Act requires agency chief  information officers (CIO) to develop, maintain, and implement an  information technology (IT) architecture.  Also, the Office of Management  and Budget (OMB) issued guidance in 1996 that requires agency IT  investments to be architecturally compliant.  These requirements are  consistent with, and in fact based on, IT management practices of leading  private and public sector organizations.
Simply stated, an enterprise systems architecture specifies the system (e.g.,  software, hardware, communications, security, and data) characteristics  that the organization’s target systems environment is to possess.  Its  purpose is to define, through careful analysis of the organization’s strategic  business needs and operations, the future systems configuration that  supports not only the strategic business vision and concept of operations,  but also defines the optimal set of technical standards that should be met to  produce homogeneous systems that can interoperate effectively and be  maintained efficiently.   Our work has shown that in the absence of an  enterprise systems architecture, incompatible systems are produced that require additional time and resources to interconnect and to maintain and  that suboptimize the organization’s ability to perform its mission.
We first reported on Customs’ need for a systems architecture in May 1996  and testified on this subject in May 1997.  In response, Customs developed  and published an architecture in July and August 1997.  We reviewed this  architecture and reported in May 1998 that it was not effective because it  was neither complete nor enforced.   For example, the architecture did not  1.  fully describe Customs’ business functions and their relationships, 2.  define the information needs and flows among these functions, and 3.  establish the technical standards, products, and services that would be  characteristic of its target systems environment on the basis of these  business specifications.
Accordingly, we recommended that Customs complete its enterprise  information systems architecture and establish compliance with the  architecture as a requirement of Customs’ information technology  investment management process.  In response, Customs agreed to develop  a complete architecture and establish a process to ensure compliance.   Customs is in the process of developing the architecture and reports that it  will be completed in May 1999.  Also, in January 1999, Customs reported  that it changed its internal procedures to provide for effective enforcement  of its architecture, once it is completed.  Until the architecture is completed  and enforced, Customs risks spending millions of dollars to develop,  acquire, and maintain information systems, including ACE, that do not  effectively and efficiently support the agency’s mission needs.
section_title: Customs Is Not Managing Its Investment in ACE Effectively
paragraphs: Effective IT investment management is predicated on answering one basic  question:  Is the organization doing the “right thing” by investing specified  time and resources in a given project or system?  The Clinger-Cohen Act  and OMB guidance together provide an effective IT investment  management framework for answering this question.  Among other things,  they set requirements for  1.  identifying and analyzing alternative system solutions,  2.  developing reliable estimates of the alternatives’ respective costs and  benefits and investing in the most cost beneficial alternative, and  3.  to the maximum extent practical, structuring major projects into a series  of increments to ensure that each increment constitutes a wise investment.
Customs did not satisfy any of these requirements for ACE.  First, Customs  did not identify and evaluate a full range of alternatives to its defined ACE  solution before commencing development activities.  For example,  Customs did not consider how ACE would relate to another Treasury- proposed system for processing import trade data, known as the  International Trade Data System (ITDS), including considering the extent  to which ITDS should be used to satisfy needed import processing  functionality.  Initiated in 1995 as a project to develop a coordinated,  governmentwide system for the collection, use, and dissemination of trade  data, the ITDS project is headed by the Treasury Deputy Assistant  Secretary for Regulatory, Tariff and Trade Enforcement. The system is  expected to reduce the burden federal agencies place on organizations by  requiring that they respond to duplicative data requests.  Treasury intends  for the system to serve as the single point for collecting, editing, and  validating trade data as well as collecting and accounting for trade revenue.   At the time of our review of ACE, these functions were also planned for  ACE.
Similarly, Customs did not evaluate different ACE architectural designs,  such as the use of a mainframe-based versus client/server-based hardware  architecture.  Also, Customs did not evaluate alternative development  approaches, such as acquisition versus in-house development.  In short,  Customs committed to and began building ACE without knowing whether  it had chosen the most cost-effective alternative and approach.
Second, Customs did not develop a reliable life cycle cost estimate for the  approach it selected.  SEI has developed a method for project managers to  use to determine the reliability of project cost estimates.  Using SEI’s  method, we found that Customs’ $1.05 billion ACE life cycle cost estimate  was not reliable, and that it did not provide a sound basis for Customs’  decision to invest in ACE.  For example, in developing the cost estimate,  Customs (1) did not use a cost model, (2) did not account for changes in its  approach to building different ACE increments, (3) did not account for  changes to ACE software and hardware architecture, and (4) did not have  historical project cost data upon which to compare its ACE estimate.
Moreover, the $1.05 billion cost estimate used to economically justify ACE  omitted relevant costs.  For instance, the costs of technology refreshment  and system requirements definition were not included (see table 1).    Exacerbating this problem, Customs represented its ACE cost estimate as a  precise point estimate rather than explicitly disclosing to investment  decisionmakers in Treasury, OMB, and Congress the estimate’s inherent  uncertainty.
Customs’ projections of ACE benefits were also unreliable because they  were either overstated or unsupported.  For example, the analysis includes  $203.5 million in savings attributable to 10 years of avoided maintenance  and support costs on the Automated Commercial System (ACS)—the  system ACE is to replace.  However, Customs would not have avoided  maintenance and support costs for 10 years.  At the time of Customs’  analysis, it planned to run both systems in parallel for 4 years, and thus  planned to spend about $53 million on ACS maintenance and support  during this period.  As another example, $650 million in savings was not  supported by verifiable data or analysis, and $644 million was based on  assumptions that were analytically sensitive to slight changes, making this  $644 million a “best case” scenario.
Third, Customs is not making its investment decisions incrementally as  required by the Clinger-Cohen Act and OMB.  Although Customs has  decided to implement ACE as a series of 21 increments, it is not justifying  investing in each increment on the basis of defined costs and benefits and a  positive return on investment for each increment.  Further, once it has  deployed an increment at a pilot site for evaluation, it is not validating the  benefits that the increment actually provides, and it is not accounting for  costs on each increment so that it can demonstrate that a positive return on  investment was actually achieved.  Instead, Customs estimated the costs  and benefits for the entire system--all 21 increments, and used this as  economic justification for ACE.
Mr. Chairman, our work has shown that such estimates of many system  increments to be delivered over many years are impossible to make  accurately because later increments are not well understood or defined.   Also, these estimates are subject to change in light of experiences on  nearer term increments and changing business needs.  By using an  inaccurate, aggregated estimate that is not refined as increments are  developed, Customs is committing enormous resources with no assurance  that it will achieve a reasonable return on its investment.  This “grand  design” approach to managing large system modernization projects has  repeatedly proven to be ineffective across the federal government,  resulting in huge sums invested in systems that do not provide expected  benefits.  Failure of the grand design approach was a major impetus for the  IT management reforms contained in the Clinger-Cohen Act.
section_title: Customs Is Not Managing ACE Software Development/ Acquisition Effectively
paragraphs: Software process maturity is one important and recognized measure of  determining whether an organization is managing a system or project the  “right way,” and thus whether or not the system will be completed on time  and within budget and will deliver promised capabilities.  The Clinger- Cohen Act requires agencies to implement effective IT management  processes, such as processes for managing software development and  acquisition.  SEI has developed criteria for determining an organization’s  software development and acquisition effectiveness or maturity.
Customs lacks the capability to effectively develop or acquire ACE  software.  Using SEI criteria for process maturity at the “repeatable” level,  which is the second level on SEI’s five-level scale and means that an  organization has the software development/acquisition rigor and discipline  to repeat project successes, we evaluated ACE software processes.  In  February 1999,  we reported that the software development processes that  Customs was employing on NCAP 0.1, the first release of ACE, were not  effective.  For example, we reported that Customs lacked effective  software configuration management, which is important for establishing  and maintaining the integrity of the software products during development.   Also, we reported that Customs lacked a software quality assurance  program, which greatly increased the risk of ACE software not meeting  process and product standards.  Further, we reported that Customs lacked  a software process improvement program to effectively address these and  other software process weaknesses.   Our findings concerning ACE  software development maturity are summarized in table 2.
As discussed in our brief history of ACE, after Customs developed NCAP  0.1 in-house, it decided to contract out for the development of NCAP 0.2,  thus changing its role on ACE from being a software developer to being a  software acquirer.  According to SEI, the capabilities needed to effectively  acquire software are different than the capabilities needed to effectively  develop software.  Regardless, we reported later in February 1999 that  the software acquisition processes that Customs was employing on NCAP  0.2 were not effective.  For example, Customs did not have an effective  software acquisition planning process and, as such, could not effectively  establish reasonable plans for performing software engineering and for  managing the software project.  Also, Customs did not have an effective  evaluation process, meaning that it lacked the capability for ensuring that  contractor-developed software satisfied defined requirements. Our findings  concerning ACE software acquisition maturity are summarized in table 3.
section_title: Customs Has Committed to Implementing Our Recommendations for Strengthening ACE Management
paragraphs: To address ACE management weaknesses, we recommended that Customs analyze alternative approaches to satisfying its import automation  needs, including addressing the ITDS/ACE relationship; invest in its defined ACE solution incrementally, meaning for each  system increment (1) rigorously estimate and analyze costs and  benefits, (2) require a favorable return-on-investment and compliance  with Customs’ enterprise systems architecture, and (3) validate actual  costs and benefits once an increment is piloted, compare actuals to  estimates, use the results in deciding on future increments, and report  the results to congressional authorizers and appropriators; establish an effective software process improvement program and  correct the software process weaknesses in our report, thereby bringing  ACE software process maturity to a least an SEI level 2; and require at least SEI level 2 processes of all ACE software contractors.
In his February 16, 1999, comments on a draft of our report, the  Commissioner of Customs agreed with our findings and committed to  implementing our recommendations.  On April 1, 1999, the Commissioner  provided us a status report on Customs efforts to do so.  In brief, the  Commissioner stated that Customs is conducting and will conduct additional analyses to consider  alternative approaches to ACE, and will base these analyses on the  assumption that Customs will use and not duplicate ITDS functionality; is developing the capability to perform cost-benefit analyses of ACE  increments, and is and will conduct postimplementation reviews of ACE  increments; has retained an audit firm to independently validate cost-benefit is developing software process improvement plans to achieve software  process maturity of level 2 and then level 3; and is preparing a directive to require at least level 2 processes of all  Customs software contractors.
Additionally, the Commissioner stated that Customs is developing a plan  for engaging a prime integration contractor that is at least SEI level 3  certified.  Under this approach, the prime contractor would assist Customs  in implementing effective system/software engineering processes and  would engage subcontractors to meet specified system development and  maintenance needs.
section_title: Conclusions
paragraphs: Successful systems modernization is absolutely critical to Customs’ ability  to perform its trade import mission efficiently and effectively in the 21st  century.  Systems modernization success, however, depends on doing the  “right thing, the right way.”  To be “right,” organizations must (1) invest in  and build systems within the context of a complete and enforced enterprise  systems architecture, (2) make informed, data-driven decisions about  investment options based on expected and actual return-on-investment for  system increments, and (3) build system increments using mature software  engineering practices.  Our reviews of agency system modernization efforts  over the last 5 years point to weaknesses in these three areas as the root causes of their not delivering promised system capabilities on time  andwithin budget.
Until Customs corrects its ACE management and technical weaknesses,  the federal government’s troubled experience on other modernization  efforts is a good indicator for ACE.  In fact, although Customs does not  collect data to know whether the first two ACE releases are already falling  short of cost and performance expectations, the data it does collect on  meeting milestones show that the first two releases have taken about 2  years longer than originally planned.  This is precisely the type of  unaffordable outcome that can be avoided by making the management and  technical improvements we recommended.
Fortunately, Customs fully recognizes the seriousness of the situation and  has committed to correcting its ACE management and technical  weaknesses.  We are equally committed to working with Customs as it  strives to do so and with Congress as it oversees this important initiative.
This concludes my statement.  I would be glad to respond to any questions  that you or other Members of the Subcommittee may have at this time.
The first copy of each GAO report and testimony is free. Additional copies are $2 each.  Orders should be sent to the  following address, accompanied by a check or money order made  out to the Superintendent of Documents, when necessary, VISA and  MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address are  discounted 25 percent.
U.S. General Accounting Office P.O. Box 37050 Washington, DC  20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537.
Each day, GAO issues a list of newly available reports and  testimony.  To receive facsimile copies of the daily list or any list  from the past 30 days, please call (202) 512-6000 using a touchtone  phone.  A recorded menu will provide information on how to obtain  these lists.