Customs Service Modernization: Serious Management and Technical Weaknesses Must Be Corrected
Published: 1999-02-26T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO reviewed the Customs Service's management of the Automated Commercial Environment (ACE), focusing on whether Customs has adequately justified ACE cost-effectiveness.
section_title: What GAO Found
paragraphs: GAO noted that: (1) Customs is not managing ACE effectively, and it does not have a firm basis for concluding that ACE is a cost-effective solution to modernizing its commercial environment; (2) GAO found serious weaknesses relating to architectural definition, investment management, and software development and acquisition that must be corrected before further investment in ACE is justified; (3) Customs is not building ACE within the context of a complete and enforced systems architecture; (4) in May 1998, GAO reported that Customs' architecture was incomplete because it was not based on a complete understanding of its enterprisewide functional and information needs; (5) GAO also reported that Customs had not yet instituted effective procedures for ensuring compliance with the architecture once it is completed; (6) until its architecture is completed and effectively enforced, Customs will not have adequate assurance that information systems like ACE will optimally support its needs across all business areas; (7) further, Customs lacks a reliable estimate of what ACE will cost to build, deploy, and maintain; and Customs has neither adequately justified, nor is it effectively monitoring, ACE's cost-effectiveness; (8) specifically, Customs did not use rigorous cost estimating techniques in preparing its cost estimate, and did not disclose the inherent imprecision of the estimate; (9) additionally, Customs omitted costs and inflated benefits in preparing its cost-benefit analysis; (10) moreover, Customs is not using effective incremental investment management practices; (11) while Customs plans to develop/acquire ACE in 21 increments, these increments are not individually cost-benefit justified, and Customs is not determining what benefits each increment, once operational, actually provides; (12) as a result, Customs will not know if ACE's expected return-on-investment is actually being realized until it has already spent hundreds of millions of dollars developing/acquiring the entire system; and (13) GAO found that Customs has neither the capability to effectively develop nor acquire ACE and that its processes for doing both, according to widely accepted and proven software capability maturity models, are ad hoc, immature, and ineffective.

section_title: Letter
paragraphs: This report responds to your requests that we review the U. S. Customs Service’s management of the Automated Commercial Environment (ACE), including whether Customs has adequately justified ACE cost-effectiveness. Customs plans to spend over $1 billion on ACE, which is planned to support modernized import processing. We found that Customs’ is not managing ACE effectively and it does not have a firm basis for concluding that ACE is cost-effective. Accordingly, we are making recommendations to the Commissioner of Customs for strengthening the management and technical weaknesses we identified.
We are sending copies of this report to the Secretary of the Treasury; Commissioner of Customs; Director of the Office of Management and Budget; and Ranking Minority Members of the Subcommittee on Treasury, General Government, and Civil Service of the Senate Committee on Appropriations and the Subcommittee on Treasury, Postal Service, and General Government of the House Committee on Appropriations. We are also sending copies of this report to the Chairmen and Ranking Minority Members of the Senate Committee on Finance and the House Committee on Ways and Means and to other congressional committees. We will also make copies available to other interested parties upon request. If you have questions or wish to discuss the issues in this report, please contact me at (202) 512-6240. Major contributors to this report are listed in appendix V.
subsections: section_title: Introduction
paragraphs: The mission of the Customs Service is to ensure that all goods and persons entering and exiting the United States do so in compliance with all U.S. laws and regulations. It does this by (1) enforcing the laws governing the flow of goods and persons across U.S. borders and (2) assessing and collecting duties, taxes, and fees on imported merchandise. During fiscal year 1997, Customs collected $22.1 billion in revenue at more than 300 ports of entry, and it processed nearly 450 million passengers entering the United States.
subsections: section_title: Customs’ Core Business Operations: A Brief Description
paragraphs: To accomplish its mission, Customs is organized into the following six business areas: trade compliance, outbound, passenger, finance, human resources, and investigations. Each business area is described below.
Trade compliance includes enforcement of laws and regulations associated with the importation of goods into the United States. To do so, Customs (1) works with the trade community to promote understanding of applicable laws and regulations, (2) selectively examines cargo to ensure that only eligible goods enter the country, (3) reviews documentation associated with cargo entries to ensure that they are properly valued and classified, (4) collects billions of dollars annually in duties, taxes, and fees associated with imported cargo, (5) assesses fines and penalties for noncompliance with trade laws and regulation, (6) seizes and accounts for illegal cargo, and (7) manages the collection of these moneys to ensure that all trade-related debts due to Customs are paid and properly accounted for.
Outbound includes Customs enforcement of laws and regulations associated with the movement of merchandise and conveyances from the United States. To do so, Customs (1) selectively inspects cargo at U.S. ports to guard against the exportation of illegal goods, such as protected technologies, stolen vehicles, and illegal currency, (2) collects, disseminates, and uses intelligence to identify high-risk cargo and passengers, (3) assesses and collects fines and penalties associated with the exportation of illegal cargo, and (4) physically examines baggage and cargo at airport facilities for explosive and nuclear materials. In addition, the outbound business includes collecting and disseminating trade data within the federal government. Accurate trade data are crucial to establishing accurate trade statistics on which to base trade policy decisions and negotiate trade agreements with other countries. By the year 2000, Customs estimates that exports will be valued at $1.2 trillion, as compared to a reported $696 billion in 1994.
Passenger includes processing all passengers and crew of arriving and departing air, sea, and land conveyances and pedestrians. In fiscal year 1997, Customs reported that it processed nearly 450 million travelers, and by the year 2000, expects almost 500 million passengers to arrive in the United States annually. Many of Customs’ passenger activities are coordinated with other federal agencies, such as the Immigration and Naturalization Service and the Department of Agriculture’s Animal and Plant Health Inspection Service. Activities include targeting high-risk passengers, which requires timely and accurate information, and physically inspecting selected passengers, baggage, and vehicles to determine compliance with laws and regulations.
Finance includes asset and revenue management activities. Asset management consists of activities to (1) formulate Customs’ budget, (2) properly allocate and distribute funds, and (3) acquire, manage, and account for personnel, goods, and services. Revenue management encompasses all Customs activities to identify and establish amounts owed Customs, collect these amounts, and accurately report the status of revenue from all sources. Sources of revenue include duties, fees, taxes, other user fees, and forfeited currency and property. The revenue management activities interrelate closely with the revenue collection activities in the trade compliance, outbound, and passenger business areas.
Human resources is responsible for filling positions, providing employee benefits and services, training employees, facilitating workforce effectiveness, and processing personnel actions for Customs’ 18,000 employees and managers. Investigations includes activities to detect and eliminate narcotics and money laundering operations. Customs works with other agencies and foreign governments to reduce drug-related activity by interdicting (seizing and destroying) narcotics, investigating organizations involved in drug smuggling, and deterring smuggling efforts through various other methods. Customs also develops and provides information to the trade and carrier communities to assist them in their efforts to prevent smuggling organizations from using cargo containers and commercial conveyances to introduce narcotics into the United States.
To carry out its responsibilities, Customs relies on information systems and processes to assist its staff in (1) documenting, inspecting, and accounting for the movement and disposition of imported goods and (2) collecting and accounting for the related revenues. Customs expects its reliance on information systems to increase as a result of its burgeoning workload. For 1995 through 2001, Customs estimates that the annual volume of import trade between the United States and other countries will increase from $761 billion to $1.1 trillion. This will result in Customs processing an estimated increase of 7.5 million import entries—from 13.1 million to 20.6 million annually—during the same period. Recent trade agreements, such as the North American Free Trade Agreement (NAFTA), have also increased the number and complexity of trade provisions that Customs must enforce.
section_title: Customs Is Legislatively Mandated to Modernize Operations and Systems
paragraphs: Both Customs and the Congress recognize that the ability to process the growing volume of imports while improving compliance with trade laws depends heavily on successfully modernizing Customs’ trade compliance process and its supporting automated systems. To speed the processing of imports and improve compliance with trade laws, the Congress enacted Title VI of the North American Free Trade Agreement Implementation Act in December 1993. Customs refers to this legislation as the “Customs Modernization and Informed Compliance Act” or “Mod” Act.
The primary purpose of the act is to streamline and automate Customs’ commercial operations by eliminating certain legislatively mandated paper requirements and requiring Customs to establish the National Customs Automation Program (NCAP). The legislation also specified certain functions that NCAP must provide, including giving members of the trade community the capability to electronically file import entries at remote locations as well as enabling Customs to electronically process “drawback” claims, which are refunds of duties and taxes paid on imported goods that are subsequently exported or destroyed. According to Customs, the act provides the legal framework to automate commercial operations and thereby streamline the processing, and expedite the movement of merchandise at more than 300 ports of entry.
section_title: ACE Purpose, Plans, and Status
paragraphs: ACE is intended to support the trade compliance business process, implement Mod Act requirements, and replace the existing import system, the Automated Commercial System (ACS). ACS is nearly 15 years old and Customs reports that it is becoming increasingly difficult and expensive to operate, maintain, and enhance due to the system’s antiquated hardware and software and limited processing capacity.
Currently, ACE’s architecture is to include both (1) mainframe-based, centralized processing to support high-volume, repetitive transactions and (2) distributed, client-server processing and desktop PC interfaces to support port office analysis and decision support needs. The transaction-intensive applications are to run on IBM mainframes at Customs’ national data center and are to be written in COBOL and C++. The distributed processing is to occur on UNIX servers and the applications are to be written in C++ and Java. To support this processing environment, Customs plans to design and implement a single, integrated agencywide database. It also plans to connect port offices to the national data center through existing communications networks comprising the Treasury Communications System, and to connect the trade community to the data center through the Internet and a combination of dial-up lines and dedicated lines.
According to Customs, ACE will facilitate increased compliance of individuals, businesses, and governments with the trade laws and regulations of the United States and increased communication with the trade community. It will also provide an integrated, account-based, automated information system for collecting, disseminating, and analyzing trade-related data and ensuring the proper collection and allocation of revenues.
Customs began developing ACE in 1994 and plans to develop and deploy ACE in 21 increments from 1998 through 2005. Each increment consists of the software and hardware necessary to perform a discrete portion of the total ACE functionality. The first four increments, known collectively as the NCAP prototype, will process pre-identified merchandise shipped into the U.S. via truck by three selected importers. The first two increments, NCAP 0.1 and NCAP 0.2, were deployed for operational evaluation at three pilot border port locations (Detroit, Michigan; Laredo, Texas; and Port Huron, Michigan) during May 1998 and October 1998, respectively.
The succeeding 17 increments are intended to build upon the foundational capabilities contained within the four NCAP prototype increments, providing additional functionality including the capability to (1) process merchandise imported via rail, air, sea, and couriers, (2) process collections and refunds associated with imported cargo, (3) process warehouse entry cargo, and (4) process cargo subject to drawbacks. Until all of these deployments are completed, Customs intends to operate and maintain ACS.
Customs’ estimate of ACE costs has escalated since August 1997, when it initially estimated that ACE’s 10-year life-cycle cost would be $895 million for the period 1998 through 2007. In November 1997, Customs revised this 10-year cost estimate to $987.6 million and also estimated that, over 15 years, from 1994 through 2008, it would spend $1.05 billion on ACE. It then used the $1.05 billion estimate along with estimated expected savings of $1.9 billion to justify ACE. Customs is currently reevaluating this estimate.
Figure 1.1 compares planned and actual deployments of the NCAP increments. As the figure illustrates, Customs is well over 2 years behind its original schedule for NCAP.
section_title: Objectives, Scope, and Methodology
paragraphs: The Chairman, Subcommittee on Treasury, Postal Service, and General Government, House Committee on Appropriations, and the Chairman, Subcommittee on Treasury, General Government, and Civil Service, Senate Appropriations Committee, requested that we review ACE. Our objective was to determine whether Customs is effectively managing ACE, including whether Customs has adequately justified ACE cost-effectiveness. In making these determinations, we relied primarily on the following criteria: (1) the Clinger-Cohen Act of 1996 and other legislative reforms that require federal agencies to develop and maintain integrated systems architectures and improve their information technology investment management processes, (2) Office of Management and Budget guidance related to the acquisition and management of information resources, (3) GAO and Treasury systems architecture guidance, and (4) related Software Engineering Institute system engineering standards concerning cost estimating and software development and acquisition maturity.
To address our objective, we reviewed pertinent documentation (e.g., ACE project plan, system/software development process practices/standards, ACE technical architecture descriptions, and ACE project status reports) and interviewed responsible Treasury officials and Customs ACE project officials in order to identify (1) Customs’ management structure for developing and deploying ACE, (2) the ACE system/software development methodology, (3) the planned ACE hardware/software and communications architecture and configuration, and (4) the current ACE system development status.
Additionally, we met with Customs ACE officials to determine the status of the agency’s implementation of recommendations we made in May 1998 to complete its systems architecture. These recommendations included (1) ensuring that the architecture fully described Customs’ business functions, (2) defining the information needs and flows among these functions, (3) establishing the technical standards, products, and services that will be used to build systems that support these business functions and needs, and (4) enforcing compliance with the architecture.
Further, we obtained and reviewed supporting documentation and interviewed Treasury officials, Customs ACE officials, and Customs Investment Review Board (IRB) officials to (1) identify the current ACE project cost and life-cycle cost estimate baselines and determine the current actual expenditures to date, (2) identify the current ACE project schedule baseline and determine the actual progress to date against scheduled milestones, (3) define what ACE is intended to do and how it is expected to benefit Customs’ mission, (4) determine the extent to which ACE mission-related goals/benefits have been achieved to date and how Customs is measuring the accrued benefits, and (5) determine Customs’ strategic approach to managing the development/acquisition, integration, and deployment of ACE. The documentation we analyzed included the ACE project plan and strategic plan, functional/performance requirements, and NCAP technical assessment documents, budget/financial data, cost-benefit policy and guidance, cost-benefit analyses, various life-cycle and project cost estimates, project status reports, testing problem reports, and configuration management documentation.
Also, we reviewed project documentation to determine how the life-cycle cost baseline was estimated and how this estimating approach compared to criteria established by SEI in A Manager’s Checklist for Validating Software Cost and Schedule Estimates. The SEI criteria defines seven primary questions, each supported by more detailed secondary questions, that can be used to determine whether defined and disciplined processes were used to derive a given cost estimate. We also interviewed ACE project officials.
In addition, we assessed whether Customs thoroughly analyzed technical alternatives to ACE including the possibility of (1) enhancing and continuing to use the legacy trade system, ACS, (2) using different architectural designs for ACE, (3) following different development/acquisition strategies, and/or (4) using the Department of the Treasury’s planned multiagency International Trade Data System (ITDS), instead of ACE, for some functions, such as collecting, editing, and validating trade data and collecting and accounting for trade revenue.
Last, we used the Software Engineering Institute’s (SEI’s) Software Development Capability Maturity ModelSM (SW-CMM ), Software Acquisition Capability Maturity ModelSM (SA-CMM ), and its Software Capability Evaluation Method, to evaluate Customs’ ability to manage its NCAP 0.1 software development project and NCAP 0.2 software acquisition effort, respectively. These models and methods provide a logical and widely accepted framework for baselining an organization’s current process capabilities (i.e., strengths and weaknesses) and assessing whether an organization has the necessary process discipline in place to repeat earlier successes on similar projects. Organizations that do not satisfy the requirements for the “repeatable” level are by default judged to be at the “initial” level of maturity, meaning that their processes are ad hoc, sometimes even chaotic, with few of the processes defined and success dependent mainly on the heroic efforts of individuals.
In following the SEI method, GAO staff trained at SEI evaluated Customs’ ACE project software development/maintenance maturity in five of the six key process areas (KPA) that are necessary to attain a “repeatable” level of process maturity. GAO did not evaluate Customs in the sixth repeatable level KPA, software subcontract management, because Customs did not use subcontractors on the ACE project. Additionally, GAO staff trained at SEI evaluated Customs’ ACE project software acquisition maturity in the seven KPAs that are necessary to attain a repeatable level of process capability, and one KPA associated with the “defined” level of process maturity—acquisition risk management. The purpose of acquisition risk management is to formally identify risks as early as possible and adjust the acquisition to mitigate those risks. Many software experts consider acquisition risk management to be an integral part of the solicitation, project performance management, and contract performance management processes.
Customs provided written comments on a draft of this report. These comments are presented in chapter 5, and are reprinted in appendix I. We performed our work at Customs and Department of the Treasury headquarters offices in Washington, D.C., and at the Customs Data Center facility in Newington, Virginia between February 1998 and November 1998, in accordance with generally accepted government auditing standards.
section_title: ACE Is Being Developed Without a Complete Enterprise Systems Architecture
paragraphs: Architectures are critical for designing and developing large and complex information systems. These comprehensive “construction plans” systematically and completely describe an organization’s target business environment, both in logical (e.g., missions, business functions, information flows) terms and technical (e.g., software, hardware, communications) terms. Without a target architecture to guide and constrain IT investment, there is no systematic way to preclude either inconsistent system design and development decisions or the resulting suboptimal performance and added cost associated with incompatible systems.
The Clinger-Cohen Act of 1996 requires agency CIOs to develop and maintain an integrated system architecture. In addition, OMB issued guidance in 1996 that, among other things, requires agency IT investments to be consistent with federal, agency, and bureau architectures.
Customs does not currently have a complete target architecture but has recently established a process for enforcing an architecture once one is completed. Customs has plans for completing its target architecture by May 1999 and ensuring its enforcement. Thus far, however, it has only defined its current architectural environment.
subsections: section_title: A Framework for Effective Systems Architectures
paragraphs: Reflecting the general consensus in the industry that large, complex systems development and acquisition efforts should be guided by explicit architectures, we issued a report in 1992 defining a comprehensive framework for designing and developing systems architectures. This framework, which is consistent with guidance that Treasury has provided to its bureaus, divides systems architectures into (1) a logical or business component, which is developed first, and (2) a technical or systems component, which is based on the first component.
The logical component ensures that the systems meet the business needs of the organization. It provides a high-level description of the organization’s mission and target concept of operations; the business functions being performed and the relationships among functions; the information needed to perform the functions; the users and locations of the functions and information; and the information systems needed to support the agency’s business needs.
The technical component ensures that systems are interoperable, function together efficiently, and are cost-effective over their life-cycles (including maintenance costs). The technical component details specific standards and approaches that will be used to build systems, including hardware, software, communications, data management, security, and performance characteristics.
section_title: Customs Is Developing, but Has Not Yet Completed, Its Target Systems Architecture
paragraphs: Customs does not currently have a complete enterprise systems architecture. In May 1998, we reported that for five of its six business areas (outbound, passenger, finance, human resources, and investigations), Customs’ architecture does not (1) describe all of the agency’s business functions, (2) completely identify the users and locations of the functions, or (3) define the information needed to perform the functions. Further, while the architecture and related documentation describe business functions and users and work locations for the sixth business area (trade compliance), they do not identify all of the information needs and flows for all of the trade functions. Nonetheless, Customs had defined many characteristics of its systems’ hardware, software, communications, data management, and security components. Because these characteristics were not based on a complete understanding of its enterprisewide functional and information needs, as specified in both best practice and Treasury guidance, we concluded that Customs did not have adequate assurance that its information systems will optimally support its needs across all business areas.
We recommended that the Commissioner of Customs direct the Customs CIO, in consultation with the Treasury CIO, to complete the architecture. Specifically, we recommended that, at a minimum, the architecture should (1) describe Customs’ target business operations, (2) fully define Customs’ interrelated business functions to support these target operations, (3) clearly describe information needs and flows among these functions, (4) identify the systems that will provide these functions and support these information needs and flows, and (5) use this information to specify the technical standards and related characteristics that these systems should possess to ensure that they interoperate, function together efficiently, and are cost-effective to maintain.
Customs and Treasury officials have acknowledged the limitations in Customs’ architecture, and agreed to define the agency’s logical architecture (e.g., functions, information needs and flows and users) across its six business areas and, in light of this definition, reevaluate the technical characteristics it has specified for its technical architecture (i.e., systems environment). Thus far, Customs has defined its current (i.e., its “as-is” or “in-state”) architectural environment, including its current business operations, its current supporting business functions and their information needs and flows, the systems currently supporting these functions, and the technical characteristics (e.g., hardware, software, and communications) of these supporting systems. However, Customs has not defined an architecture for its target (i.e., future) business and systems environment. Customs originally planned to have completed its target architecture by September 1998, but that date is now May 1999.
section_title: Customs Has Recently Announced Process for Ensuring Architectural Compliance
paragraphs: In May 1998, we reported that Customs’ investment management processdid not ensure that all new systems would conform to the architecture. In particular, we reported that Customs’ investment review board (IRB) used four criteria in scoring competing investment options and allocating funding among them. The four criteria were (1) risk (e.g., technical, schedule, and cost); (2) strategic alignment (e.g., cross-functional benefits, linkage to Customs’ business plan, and compliance with legislative mandates); (3) mission effectiveness (e.g., contributions to service delivery); and (4) cost-benefit ratio (e.g., tangible and intangible benefits, and costs).
Because compliance with the architecture was considered under the risk criterion but was not required, the process did not preclude funding projects that were inconsistent with the enterprise architecture. Moreover, the process did not require that such deviations from the architecture be rigorously justified.
To ensure that Customs effectively enforced its information systems architecture, we recommended that Customs require that all new projects comply with the architecture unless an exception could be justified by careful, thorough, and documented analysis. Customs agreed and, in January 1999, changed its investment management process to explicitly require that proposed IT investments comply with its architecture, unless an exception is justified and a waiver is granted by the technical review committee.
section_title: Conclusions
paragraphs: Until Customs completes and enforces its enterprise systems architecture, it will not have adequate assurance that ACE and other systems it plans to build and operationally deploy (1) will effectively meet the agency’s business needs and (2) are compatible, efficient, and cost-effective to develop, integrate, and maintain.
section_title: Customs’ ACE Investment Management Practices Are Ineffective
paragraphs: The Clinger-Cohen Act and OMB guidance provide an effective framework for IT investment management. Together, they set requirements for (1) identifying all potential alternative system solutions, (2) developing reliable estimates of project life-cycle costs and benefits, and investing in the most cost-beneficial alternative, and (3) to the maximum extent practical, structuring major projects into a series of increments to ensure that each increment constitutes a wise investment.
Customs has not effectively implemented any of these investment management practices on ACE. Specifically, (1) Customs’ investment analysis did not address alternatives to its chosen ACE system solution, (2) Customs’ did not use rigorous cost estimating techniques in preparing its ACE cost estimate and its cost-benefit analysis for ACE omitted substantial costs and inflated benefits, and (3) Customs is not justifying and validating the costs and benefits for each ACE increment. As a result, Customs lacks a sound basis for making ACE investment decisions.
subsections: section_title: ACE Alternatives Were Not Considered
paragraphs: Before embarking on a major, costly systems initiative such as ACE, agencies should thoroughly assess the full range of technical options. In the case of ACE, Customs had several alternatives to satisfying its mission needs as specified in the Mod Act. For example, it could enhance ACS, use different architectural designs for ACE, follow different development/ acquisition strategies for ACE, and/or use Treasury’s planned governmentwide trade system, International Trade Data System (ITDS), to supplement some ACE functions, such as collecting and disseminating trade data. ITDS was initiated in 1995 as a project to implement the National Performance Review recommendation to develop a coordinated, governmentwide system for the collection, use, and dissemination of trade data. At that time, a multiagency board of directors was established, headed by the Department of the Treasury with Customs and other major trade-related agencies represented. The Department of the Treasury is responsible for designing and developing ITDS, which is expected to reduce the burden that federal agencies place on the international trading organizations by requiring that they respond to duplicative data requests. Treasury intends for the system to serve as the single point for collecting, editing, and validating trade data as well as collecting and accounting for trade revenue.
By thoroughly considering these and other choices, Customs would have ensured that the most cost-effective and beneficial alternative was chosen before deciding to invest $1.05 billion in ACE. In fact, OMB requires that agencies consider alternative system solutions to meet mission needs, including different system architectures, upgrading existing systems, or contracting for development and integration of major systems.
Customs did not identify and evaluate a full range of alternatives to ACE. In fact, Customs considered only (1) enhancing and operating ACS to provide the same functionality of ACE, (2) operating ACS without any enhancement, (3) operating ACS with limited enhancements, and (4) developing and deploying part of ACE’s planned functionality. Customs discarded the last three because none provided for meeting all of the Mod Act requirements. With respect to the first, Customs compared ACS to ACE and decided that ACE was the more cost-effective alternative.
Customs did not evaluate other alternatives to its ACE solution, including (1) adopting a different ACE architectural design (e.g., using a combination mainframe and client-server configuration instead of a mainframe only system), (2) contracting out for ACE development rather than developing ACE in-house, or (3) using ITDS to satisfy part of its functional needs. As a result, Customs committed to and began investing in ACE development without knowing whether it had chosen the most cost-effective alternative.
Since making its decision to develop ACE, Customs has changed ACE’s architectural design, and it has decided to acquire the second ACE increment rather than develop it in-house. However, these decisions are not supported by any verifiable alternatives analysis, meaning that Customs still lacks adequate assurance that it is following the most cost-effective approach to satisfying its mission needs.
section_title: Customs’ $1.05 Billion ACE Cost Estimate Is Not Reliable
paragraphs: Reliable estimates of IT projects’ expected costs are essential to determining a project’s cost-effectiveness. Without such information, the likelihood of poor investment decisions is increased, not only when a project is initiated but also throughout its life-cycle.
To assist management in assessing the credibility of a given project’s cost estimate, SEI developed the following seven questions for decisionmakers to use to assess the reliability of a project’s cost estimate and detailed criteria to assist in evaluating how well a project satisfies each question.(1) Are the objectives of the estimate clear and correct? (2) Has the task been appropriately sized? (3) Is the estimated cost consistent with demonstrated accomplishments on other projects? (4) Have the factors that affect the estimate been identified and explained? (5) Have steps been taken to ensure the integrity of the estimating process? (6) Is the organization’s historical evidence capable of supporting a reliable estimate? (7) Has the situation remained unchanged since the estimate was prepared?
We analyzed the approach that Customs followed in deriving its $1.05 billion ACE life-cycle cost estimate using SEI’s criteria. Among these criteria were several very significant and closely intertwined requirements that are at the core of effective cost estimating. Specifically, embedded in several of the aforementioned seven questions are requirements for using (1) formal cost models, (2) structured and documented processes for determining the software size and reuse inputs to the models, and (3) relevant, measured, and normalized historical cost data (estimated and actual) to calibrate the models.
We found that Customs did not satisfy any of these requirements. In particular, Customs did not use a cost model. Instead, it used an unsophisticated spreadsheet to extrapolate the cost of each ACE increment. Further, Customs’ approach to determining software size and reuse was not documented and was not well supported or convincing. Customs estimated the size of each ACE software increment (most of which were still undefined) by extrapolating from the estimated size of the first increment based on individuals’ undocumented best judgments about increment functionality and complexity. Last, Customs did not have any historical project cost data at the time it developed the $1.05 billion estimate, and it did not account for relevant, measured, and normalized differences among the increments. For example, it did not account for (1) the change in ACE’s architecture from a mainframe system written in COBOL and C++ to a combined mainframe and Internet-based system written in C++ and Java and (2) the change in ACE from an in-house software development project (NCAP 0.1) to a software acquisition (NCAP 0.2). Clearly, such fundamental changes can have a dramatic effect on system costs, and should have been addressed explicitly in Customs’ cost estimates.
Table 3.1 summarizes the complete results of our assessment of Customs’ cost estimating process. For each of SEI’s questions and supporting criteria, Customs was rated as demonstrating a strength, i.e., effectively implementing the criterion; a weakness, i.e., ineffectively implementing the criterion; or, where evidence was inconclusive and could not support characterization as either a strength or a weakness, an observation was noted. (See appendix II for further detail on SEI’s criteria and our findings).
subsections: section_title: Customs’ Cost Estimate Implies an Unjustified Level of Precision
paragraphs: Software and systems development experts agree that early project estimates are by definition imprecise, and that this inherent imprecision decreases during the project’s life-cycle as more information becomes known about the system. These experts emphasize that, to be useful, each cost estimate should include an indication of its degree of uncertainty, possibly as an estimated range or qualified by some factor of confidence. For example, a cost estimate of $1 million could be presented as a range from $750,000 to $1.25 million or as $1 million with a confidence level of 90 percent, indicating that there is a 10 percent chance that costs will exceed this estimate.
Customs did not reveal the degree of uncertainty of its cost estimate for ACE to managers involved in investment decisions. For example, Customs did not disclose that it made the estimate before fully defining ACE functionality. Instead, Customs presented its $1.05 billion ACE life-cycle cost estimate as an unqualified point estimate. This suggests an element of precision that cannot exist for such an undefined system and obscures the investment risk remaining in this project.
section_title: Customs’ Analysis of ACE Costs and Benefits Is Also Unreliable
paragraphs: ACE cost estimates are understated, benefit estimates are overstated, and both are unreliable. Customs’ August 1997 cost-benefit analysis estimated that ACE would produce cumulative savings of $1.9 billion over a 10-year life-cycle beginning in fiscal year 1998 and ending in fiscal year 2007. However, this analysis was unreliable because certain benefits and costs were unsupported, other benefits were overstated, and relevant costs were omitted. For example:
The analysis identified $650 million (35 percent of the total ACE estimated savings) in savings to the trade community from more accurate and timely data, improved customer compliance, reduced support staff, reduced transportation costs, elimination of duplicate recordkeeping systems, and lower processing costs associated with periodic payments. However, Customs could not produce any verifiable data or analysis to support this claim. According to Customs officials, this benefit amount was projected on the basis of undocumented information supplied by one company. The officials stated that other companies considered such data confidential and would not provide them to Customs.
The analysis identified an additional $644 million (33 percent of the total savings) resulting from increased productivity. Because this estimate is driven by Customs’ assumption that every minute “saved” by processing transactions or analyzing data faster using ACE, rather than ACS, would be productively used by all workers, it can be viewed as a “best case,” upper limit on estimated productivity improvements. Given the magnitude of the potential savings, even a small change in the assumption translates into a large reduction in benefits. For example, conservatively assuming that three-fourths of each minute saved would be productively utilized by three-fourths of all workers, the expected benefits would be reduced by about $282 million.
The analysis excluded costs for hardware and system software upgrades (e.g., desktop workstations and operating systems, application and data servers, data base management systems) at each port office. Using Customs’ estimate for acquiring the initial suite of port office hardware and system software, and assuming a technology refreshment cycle of every 3 to 5 years, we estimated this cost to be between $72.9 million and $171.8 million.
The analysis excluded $52.8 million of costs needed to support the data center and maintain ACS through fiscal year 2002. Since Customs intends to operate and maintain ACS in parallel with ACE through 2002, these costs should be included.
The analysis excluded costs to conduct security analysis, project planning and management, and independent verification and validation. Customs estimates that these costs collectively total $23 million.
The analysis excluded other relevant cost items, such as the cost of defining ACE requirements, integrating ACE components, and conducting ACE regression testing and training. Customs did not have an estimated value for these costs.
The analysis included annual telecommunications costs of $60.3 million once ACE is deployed to all sites. However, Customs could not provide us with any supporting data or analysis for this estimate.
section_title: Customs Is Not Investing Incrementally in ACE
paragraphs: Incremental project management involves three fundamental components: (1) developing/acquiring a large system in a series of smaller projects or system increments, (2) individually justifying investment in each separate increment on the basis of return-on-investment, and (3) monitoring actual benefits achieved and costs incurred on completed increments and modifying subsequent increments/investments to reflect lessons learned. By doing so, agencies avoid discovering too late that their system is not cost-beneficial, and they can reduce the enormous risks associated with large, expensive projects. The Clinger-Cohen Act requires that agencies acquire information technology incrementally, to the maximum extent practicable, and have milestones for senior managers to obtain information on the cost, timeliness, quality, and capability of information system investments to meet requirements. Additionally, OMB policy requires that investments in major information systems be implemented in increments with each increment delivering measureable benefits.
Customs is developing and acquiring ACE in a series of 21 increments. It has, to date, defined the functionality of only the first two increments and will define later increments in the future. Nonetheless Customs has estimated costs and benefits for, and has committed to, investing in the entire system (i.e., all 21 increments). It has not estimated the costs and benefits for each increment and does not know whether each increment will produce a reasonable return on investment. Furthermore, once it has deployed an increment at a pilot site for evaluation, Customs is not validating that estimated benefits were actually achieved. As a result, Customs will not even know whether the first ACE increment, now being piloted at three sites, is producing expected benefits or is cost-effective. Instead, Customs will only determine whether the first increment is performing at a level “equal to or better than” ACS.
Customs “grand design” approach to ACE does not constitute wise investment management. Customs will not know whether earlier increments were cost beneficial before it invests in later increments; it does not have reliable cost or benefit data upon which to base investment decisions; and it is committing substantial resources with no assurance that it will achieve a reasonable return-on-investment.
section_title: Conclusions
paragraphs: Because Customs does not have reliable information on ACE costs and benefits and has not analyzed other viable alternatives to the system solution it selected, it does not have adequate assurance that ACE is the optimal approach. In fact, it has no assurance at all that ACE is cost effective. Furthermore, because it is not justifying the return on its investment in each ACE increment, Customs will not be able to demonstrate whether ACE is cost-effective until it has already spent hundreds of millions of dollars to develop/acquire the entire system.
section_title: ACE Software Development and Acquisition Processes Are Not Effective
paragraphs: The Clinger-Cohen Act requires agency CIOs to establish effective IT management processes, such as processes for developing or acquiring software. Customs has developed the first ACE software increment (NCAP 0.1) in-house and is acquiring the second increment (NCAP 0.2) from a contractor. Customs has not yet decided whether it will develop or acquire future ACE software increments. If it is to do either effectively, Customs must use mature software processes.
SEI, recognized for its expertise in software processes, has developed models and methods that define and determine, respectively, software development and software acquisition process maturity. We evaluated both ACE software development and ACE software acquisition processes using SEI’s software development capability maturity model (SW-CMM) and software acquisition capability maturity model (SA-CMM), respectively, and SEI’s software capability evaluation (SCE) method. Our evaluations focused on the key process areas (KPA) necessary to obtain a “repeatable” level of maturity, the second level of SEI’s five-level process maturity models. Organizations that do not satisfy these second-level KPA requirements are by default at the “initial” or first level, meaning that their processes are ad hoc, sometimes even chaotic.
Our evaluations found that Customs lacks the capability to either develop or acquire ACE software effectively, and that it lacks a software process improvement program. Because of the number and severity of the KPA weaknesses found, Customs did not achieve the “repeatable” level of process maturity as either a software developer or acquirer, meaning that any attempts to do so are likely to produce software that does not perform as intended, costs more than expected, and is delivered late. Further, without a software process improvement program, it is unlikely that Customs can effectively address its software process weaknesses.
subsections: section_title: SEI’s Software Capability Models and Method: A Brief Description
paragraphs: The SW-CMM and the SA-CMM rank organizational maturity according to five levels. Maturity levels 2 through 5 require verifiable existence and use of certain key process areas, known as KPAs. The SW-CMM includes six level 2 KPAs. We evaluated Customs’ against five of the six. The sixth level 2 KPA, software subcontract management, was not evaluated because Customs did not use subcontractors on NCAP 0.1 (see table 4.1 for a description of each KPA).
The SA-CMM includes seven level 2 KPAs. We evaluated Customs against all seven level 2 KPAs. We also evaluated Customs against one level 3 KPA—acquisition risk management—because it is considered by software experts to be an integral part of the solicitation, project performance management, and contract performance management processes (see table 4.2 for a description of each KPA).
For both models, each KPA contains five common attributes that indicate whether the implementation and institutionalization of a KPA can be effective, repeatable, and lasting. The five common features are:
Commitment to perform: The actions that the organization must take to establish the process and ensure that it can endure. Commitment to perform typically involves establishing organizational policies and sponsorship.
Ability to perform: The preconditions that must exist in the project or organization to implement the process competently. Ability to perform typically involves allocating resources, establishing effective organizational structures, and ensuring that personnel have the needed training.
Activities performed: The roles and procedures necessary to implement the process. Activities performed typically involve establishing plans and procedures, performing the work, tracking it, and taking appropriate management actions.
Measurement and analysis: The activities performed to measure the process and analyze the measurements. Measurement and analysis typically include defining the measurements to be taken and the analyses to be conducted to determine the status and effectiveness of the activities performed.
Verifying implementation: The steps to ensure that the activities are performed in compliance with the process that has been established. Verification typically includes reviews by management.
In accordance with SEI’s SCE method, for each KPA in level 2 and the one KPA in level 3 (risk management), we evaluated institutional policies and practices and compared project-specific guidance and practices against the five common attributes. This project-specific comparison can result in one of four possible outcomes: (1) project strength—effective implementation of the key practice, (2) project weakness—ineffective implementation of a key practice or failure to implement a key practice, (3) project observation—key practice evaluated but evidence is inconclusive and cannot support characterization as either a strength or a weakness, and (4) not rated—key practice not currently relevant to project, therefore not evaluated.
section_title: Customs Lacks the Capability to Develop ACE Software Effectively
paragraphs: In February 1999, we reported that NCAP 0.1 did not fully satisfy any of the KPAs that SEI’s SW-CMM requires to be CMM level 2 or repeatable. While NCAP 0.1 exhibited many strengths in three KPA’s—requirements management, software project planning, and software project tracking and oversight—it had numerous and significant weaknesses in the remaining two KPA’s—software quality assurance and software configuration management. As a result, Customs’ ACE software development capability is immature.
In our February 1999 report, we also stated that Customs lacked a software process improvement program. Without one, it is unlikely that Customs can effectively address its software process weaknesses. Accordingly, we recommended that, after ensuring that its mission-critical systems are Year 2000 compliant, but before investing in major software development efforts like ACE, Customs (1) assign responsibility and authority for software process improvement, (2) develop and implement a formal plan for software process improvement that, among other things, was based on our software development process maturity findings, (3) ensure that every new software development effort in Customs adopts processes that satisfy at least SW-CMM level 2 requirements, and (4) ensure that process improvement activities be initiated for all ongoing essential software maintenance projects. Customs agreed with our findings and stated its commitment to software process improvement and maturity, including stating its plans for establishing a software process improvement program and addressing the weaknesses that we identified.
Each of the five SW-CMM KPAs, along with examples of how the ACE software development organization compares to each KPA practices, is summarized below. (See appendix III for more detailed information on the KPAs and our findings.)
subsections: section_title: Requirements Management
paragraphs: The purpose of requirements management is to establish a common understanding of the requirements between the customer and the software developers. According to the SW-CMM, a repeatable requirements management process, among other things, includes (1) documenting the system requirements allocated to software, (2) providing adequate resources and funding for managing the allocated requirements, (3) training members of the software engineering group to perform their requirements management activities, (4) using the allocated requirements as the basis for software plans, work products, and activities, (5) following a written organizational policy for requirements management, and (6) having a quality assurance group that reviews the activities and work products for managing allocated requirements and reports the results.
The NCAP 0.1 project exhibited strengths in almost all key practices within the requirements management KPA. For example, (1) the system requirements allocated to software were documented, (2) adequate resources and funding were provided for managing the allocated requirements, (3) members of the software engineering group were trained to perform their requirements management activities, and (4) the software developers used the allocated requirements as the basis for software plans, work products, and activities.
Nevertheless, two very important requirements management key practices were not being performed on the NCAP 0.1 project. Specifically, there was no written organizational policy for requirements management and there was no quality assurance group to review the activities. In the absence of these practices, management is missing important means for ensuring that software requirements are managed in a prescribed manner.
section_title: Software Project Planning
paragraphs: The purpose of software project planning is to establish reasonable plans for performing software engineering and for managing the software project. According to the SW-CMM, a repeatable software project planning process, among other things, includes (1) documenting the software project plan, and preparing plans for software engineering facilities and support tools, (2) identifying the work products needed to establish and maintain control of the software project, (3) making and using measurements to determine the status of planning activities, (4) following a written organizational policy for planning a software project, (5) training personnel in software project planning and estimating, (6) estimating the software project’s efforts, costs, critical computer resources, and schedule according to documented procedures, and (7) having a quality assurance group that reviews the activities and work products for software project planning and reports the results.
The NCAP 0.1 project exhibited strengths in many of the key practices within the software project planning KPA. For example, (1) the software project plan and plans for the software engineering facilities and support tools were documented, (2) software work products that are needed to establish and maintain control of the project were identified in the project plan, and (3) measurements were made and used to determine the status of software planning activities.
However, we also found several significant key practice weaknesses. Specifically, (1) NCAP 0.1 did not follow a written policy for planning a software project, (2) project personnel were not trained in project planning and estimating procedures, (3) estimates for the software project’s effort and costs, critical computer resources, and project schedule were not derived according to documented procedures, and (4) there was no software quality assurance group to review or audit the activities and work products for software project planning. Such project planning weaknesses mean that management has no assurance that it will get the consistent, complete, and reliable information about the project’s expected costs and schedules needed to make expeditious and informed investment decisions.
section_title: Software Project Tracking and Oversight
paragraphs: The purpose of software project tracking and oversight is to provide adequate visibility into the progress of the software development so that management can take effective actions when the software project’s performance deviates significantly from the software plans. According to the SW-CMM, effective software project tracking and oversight, among other things, includes (1) designating a project software manager to be responsible for the project’s software activities and results, (2) having a documented software development plan for tracking software activities and communicating status, (3) conducting periodic internal reviews to track technical progress, plans, performance, and issues against the software development plan, (4) periodically reviewing the activities for software project tracking and oversight with senior management, (5) following a written organizational policy for managing the project, (6) training software project managers in the technical and personnel aspects of the software project, (7) reviewing software project commitments and changes to commitments with senior management, and (8) independently reviewing or auditing the activities and work products for software project tracking and oversight.
NCAP 0.1 exhibited strengths in most of the key practices within the software project tracking and oversight KPA. For example, (1) a project manager had been designated to be responsible for the project’s software activities and results, (2) the project had a documented software development plan, (3) the software engineering group conducted periodic internal reviews to track technical progress, plans, performance, and issues against the software development plan (e.g., the sizes of software work products and the risks associated with software costs, resources, and schedule), and (4) software project tracking and oversight activities were reviewed with senior management on a weekly basis.
However, significant weaknesses also existed. For example, (1) there was no written organizational policy for managing the software project, (2) the software managers were not trained in managing the technical and personnel aspects of the software project, (3) software project commitments and changes to commitments were not reviewed with senior management, and (4) because no software quality assurance group exists, there were no independent reviews or audits of the activities and work products for software project tracking and oversight.
section_title: Software Quality Assurance
paragraphs: The purpose of software quality assurance is to independently review and audit software products and activities to verify that they comply with applicable procedures and standards and to provide the software project and higher level managers with the results of these independent reviews and audits. According to the SW-CMM, a repeatable software quality assurance process, among other things, includes (1) preparing a software quality assurance plan according to a documented procedure, (2) having a written organizational policy for implementing software quality assurance, (3) conducting audits of designated work processes and products to verify compliance, (4) documenting deviations identified in the software activities and software work products and handling them according to a documented procedure, and (5) having experts independent of the software quality assurance group periodically review the activities and work products of the project’s software quality assurance group.
NCAP 0.1 did not satisfy any of the key practices within the software quality assurance KPA. Because its software quality assurance is ineffective, ACE is at risk that (1) process and product standards will not be met and (2) software will not perform as intended, and will cost more and take longer to develop than expected.
section_title: Software Configuration Management
paragraphs: The purpose of software configuration management is to establish and maintain the integrity of the products of the software project throughout the project’s software life-cycle. According to the SW-CMM, a repeatable software configuration management process, among other things, includes (1) preparing a software configuration management plan according to a documented procedure, (2) establishing a configuration management library system as a repository for the software baselines, (3) identifying software work products to be placed under configuration management, (4) controlling the release of products from the software baseline library according to a documented procedure, (5) following a written organizational policy for implementing software configuration management, (6) recording the status of configuration items/units according to a documented procedure, (7) making and using measurements to determine the status of software configuration management activities, and (8) periodically reviewing software configuration management activities with senior management.
NCAP 0.1 had strengths in several of the key practices within the software configuration management KPA. For example, (1) a software configuration management plan was developed according to a documented procedure, (2) a configuration management library system was established as a repository for the software baselines, (3) software work products to be placed under configuration management were identified, and (4) the release of products from the software baseline library was controlled according to a documented procedure.
However, NCAP 0.1 exhibited weaknesses in most of the software configuration management key practices, which collectively jeopardize Customs’ ability to maintain the integrity of the project’s software products. Specifically, (1) there was no organizational policy for implementing software configuration management, (2) the status of configuration management items was not recorded according to a documented procedure, (3) no measurements were taken to determine the status of software configuration management activities, and (4) software configuration management activities were not periodically reviewed with senior management.
section_title: Customs Lacks the Capability to Acquire ACE Software Effectively
paragraphs: NCAP 0.2 did not fully satisfy any of the KPA’s that SEI’s SA-CMM requires to be a CMM level 2 or repeatable. We found extensive weaknesses in all KPA’s except one—requirements development and management. Each of the eight KPA’s, along with examples of how Customs’ software acquisition processes compare to the KPA goals, is summarized below. (See appendix IV for more detailed information on KPAs and our findings.)
subsections: section_title: Software Acquisition Planning
paragraphs: The purpose of software acquisition planning is to ensure that reasonable planning for the software acquisition is conducted and that all aspects of the total software acquisition effort are included in these plans. According to the SA-CMM, a repeatable software acquisition planning process, among other things, includes (1) having a written software acquisition policy, (2) developing and documenting the software acquisition strategy and plan, (3) having management review software acquisition planning activities, and (4) making and using measurements to determine the status of software acquisition planning activities.
NCAP 0.2 had no strengths in this KPA. For example, (1) the project followed no written policy for planning the software acquisition, (2) there was no documented software acquisition strategy or plan, (3) software acquisition planning activities were not being reviewed by management, and (4) software acquisition planning activities were not being measured. As a result, management lacks an effective means to identify problems in project performance and take corrective action expeditiously.
section_title: Solicitation
paragraphs: The purpose of solicitation is to prepare a request for proposal that delineates a project’s software-related requirements, and, consistent with relevant solicitation laws and regulations, select a contractor that can most cost-effectively satisfy these requirements. According to the SA-CMM, specific requirements for a repeatable solicitation process include, among other things, (1) designating responsibility for the software portion of the solicitation, (2) designating a selection official to be responsible for the selection process and decision, (3) preparing cost and schedule estimates for the software products and services being acquired, (4) having a written policy for the conduct of the software portion of the solicitation, (5) having and following a solicitation plan, (6) independently reviewing cost and schedule estimates for the software products and services being acquired, and (7) making and using measurements to determine the status of the solicitation activities and resultant products.
NCAP 0.2 had some strengths in this area, including (1) designating responsibility for the software solicitation, (2) designating a selection official for the selection process and decision, and (3) preparing cost and schedule estimates for the software products and services being acquired.
However, we found many weaknesses, including (1) no written policy for the conduct of the software solicitation and (2) no solicitation plans to guide the performance of solicitation activities. These weaknesses increase the risk of Customs not adequately or uniformly evaluating the offerors’ proposals, and making a suboptimal selection. Other weaknesses included (1) no independent review of software cost and schedule estimates and (2) no measurements to determine the status of solicitation activities and resultant products. As a result, management cannot identify solicitation problems early and resolve them expeditiously.
section_title: Requirements Development and Management
paragraphs: The purpose of requirements development and management is to establish and maintain a common and unambiguous definition of software requirements among the acquisition team, the system users, and the software development contractor. This KPA involves two subprocesses: (1) developing a baseline set of software-related contractual requirements and (2) managing these requirements and changes to these requirements for the duration of the acquisition.
SA-CMM requirements development and management practices necessary to achieve a repeatable maturity include (1) having a group that is responsible for performing requirements development and management activities, (2) ensuring that individuals performing requirements development and management activities have experience or receive training, (3) measuring and reporting on the status of requirements development and management activities and resultant products to management, (4) periodically reviewing requirements development and management activities by acquisition organization management, (5) having a written organizational policy for establishing and managing the software-related contractual requirements, (6) performing requirements development and management activities in accordance with documented plans, (7) appraising the impact of system requirements change requests on the software being acquired, and (8) maintaining traceability between the software-related contractual requirements and the contractor’s software work products.
NCAP 0.2 had many strengths in requirements development and management. For example, (1) there was a group that is responsible for performing requirements development and management, (2) group members had experience or have received training in the activities, (3) measurements of requirements development and management activities were made, and (4) management reviewed the activities periodically.
However, we found weaknesses in important key practices that jeopardize effective control of the requirements baseline and can result in software products that do not meet cost, schedule, or performance objectives. Specifically, (1) there was no written policy for establishing and managing the software-related contractual requirements, (2) there was no documented requirements development and management plan, (3) system requirements change requests were not appraised for their impact on the software being acquired, and (4) traceability between the software-related contractual requirements and the contractor’s software work products was not maintained.
section_title: Project Office Management
paragraphs: The purpose of project office management is to manage the activities of the project office and supporting contractors to ensure a timely, efficient, and effective software acquisition. According to the SA-CMM, effective project office management requires, among other things, (1) designating responsibility for project management, (2) providing project teams with authority to alter either the project’s performance, cost, or schedule baseline, (3) having a written policy for the management of the software project, (4) using a software acquisition management plan, and (5) using a corrective action system for identifying, recording, tracking, and correcting problems.
NCAP 0.2 had several practice strengths in the KPA including (1) designating a project manager as responsible for the project and (2) providing project teams with authority to change either the performance, cost, or schedule software acquisition baseline when necessary. However, we found numerous weaknesses, including (1) no written policy for management of the software project, (2) no software acquisition management plan, and (3) no corrective action system. These weaknesses jeopardize the project’s ability to ensure that important project office and contractor activities are defined, understood, and completed.
section_title: Contract Tracking and Oversight
paragraphs: The purpose of contract tracking and oversight is to ensure that the software development contractor performs according to the terms of the contract; needed contract changes are identified, negotiated, and incorporated into the contract; and contractor performance issues are identified early, when they are easier to address. According to the SA-CMM, a repeatable contract tracking and oversight process, among other things, includes (1) designating responsibility for contract tracking and oversight, (2) providing support for the project team by contracting specialists, (3) ensuring that individuals performing contract tracking and oversight are suitably experienced or trained, (4) having a written organizational policy for contract tracking and oversight, (5) having a documented plan for contract tracking and oversight, (6) reviewing required contractor software planning documents to oversee the contractor’s software engineering effort, and (7) tracking problems or issues found by the project team during contract tracking and oversight in a corrective action system.
NCAP 0.2 had several strengths in this KPA, including (1) designating responsibility for contract tracking and oversight activities, (2) providing contracting specialist support to the project team, and (3) ensuring that personnel responsible for contract tracking and oversight are suitably experienced or trained. However, we found significant weaknesses, including (1) no written organizational policy for contract tracking and oversight, (2) no documented contract tracking and oversight plan, (3) no required contractor software planning documents that could be used to oversee the contractor’s software engineering effort, and (4) no corrective action tracking system for recording problems or issues found by the project team. Because of these weaknesses, Customs’ contractor tracking and oversight activities are undisciplined and unstructured, thereby increasing the chances of its software acquisitions being late, costing more than expected, and not performing as intended.
section_title: Evaluation
paragraphs: The purpose of evaluation is to determine that the acquired software products and services satisfy contract requirements prior to acceptance. According to the SA-CMM, a repeatable evaluation process includes (1) designating responsibility for planning, managing, and performing evaluation activities, (2) ensuring that individuals performing evaluation activities have experience or receive training, (3) managing the evaluation of the acquired software according to a written policy, (4) documenting evaluation plans and conducting evaluation activities in accordance with the plan, (5) developing and managing evaluation requirements in conjunction with developing software technical requirements, (6) tracking contractor performance of evaluation activities for compliance with the contract, and (7) measuring and reporting on the status of evaluation activities to management.
NCAP 0.2 had some evaluation strengths, including (1) designating responsibility for planning, managing, and performing evaluation activities and (2) ensuring that individuals performing evaluation activities have experience or receive training. However, we found many significant weaknesses, including, (1) there was no written policy for managing the evaluation of the acquired software, (2) there was no documented evaluation plan to use as the basis for conducting evaluation activities, (3) evaluation requirements were not developed in conjunction with development of the system or software technical requirements, (4) contractor evaluation activity performance was not tracked for compliance with the contract, and (5) no measurements were made to determine the status of evaluation activities. Because of these pervasive evaluation weaknesses, Customs has no assurance that contractor-developed ACE software will satisfy defined requirements.
section_title: Transition and Support
paragraphs: The purpose of transition and support is to provide for the effective and efficient “hand-off” of the acquired software products to the support organization responsible for software maintenance. According to the SA-CMM, repeatable transition and support processes, among other things, include (1) having a written policy for transitioning software products to the support organization, (2) designating a group that is responsible for coordinating transition and support activities, (3) performing project activities in accordance with a documented transition and support plan, and (4) measuring the status of the transition and support activities.
NCAP 0.2 had no practice strengths in the transition and support KPA. In particular, (1) there was no written policy for the transitioning of software products to the software support organization, (2) there was no designated group responsible for coordinating transition and support activities, (3) there was no transition and support plan, and (4) the status of transition and support activities were not measured. As a result, Customs is not effectively positioned to assume support responsibility for the ACE software that it acquires.
section_title: Acquisition Risk Management
paragraphs: The purpose of acquisition risk management is to formally identify risks as early as possible and adjust the acquisition to mitigate those risks. According to the SA-CMM, an effective risk management process, among other things, includes (1) having a written policy on acquisition risk management, (2) developing a software acquisition risk management plan, (3) conducting software risk management activities in accordance with the plan (e.g., identifying risks, taking mitigation actions, and tracking risk mitigation actions to completion), and (4) measuring and reporting on the status of acquisition risk management activities to management.
NCAP 0.2 was weak in all key practices of acquisition risk management. In particular, (1) there was no written policy on acquisition risk management, (2) there was no software acquisition risk management plan, (3) no risk management activities were being performed, and (4) no measurements were taken to determine the status of risk management activities. Because of these weaknesses, Customs has no assurance that it will identify risks early and effectively mitigate them.
section_title: Conclusions
paragraphs: Customs’ software acquisition and development processes are insufficiently mature to support the effective development or acquisition of ACE. Until these processes are strengthened, there is no basis for confidence that ACE will be delivered on time, within budget, or perform as specified.
section_title: Overall Conclusions, Recommendations, and Agency Comments
paragraphs: Successful systems modernization is critical to Customs’ ability to implement the provisions of its legislative mandates and to achieve its goals of increased compliance with trade laws and speedier processing of imports. However, Customs currently lacks the full management and technical capability needed to successfully modernize its systems. As a result, satisfaction of its legislative mandate and realization of the attendant benefits by both Customs and the trade community are in jeopardy.
With ACE, Customs has not adequately demonstrated that it is “doing the right thing,” i.e., that the system that it is building is the most cost effective system for meeting the needs of Customs and the trade community. Specifically, by not having a complete information systems architecture to guide the development and evolution of its systems, including ACE, Customs runs the risk that its efforts will not provide optimum performance and will be incompatible and expensive to maintain. Also, by using unreliable and incomplete ACE cost and benefit data, Customs does not know if ACE is a cost effective system solution to its needs, or if its investments in ACE are wise.
Customs also has not clearly demonstrated that it is developing ACE “the right way.” In particular, without incrementally managing such a mammoth system as ACE, Customs will not know whether its cost and benefit expectations are being met until it has already spent hundreds of millions of dollars. Such a “grand design” approach has proven repeatedly to be ineffective and has been abandoned by successful IT organizations. Additionally, because its ability to develop or acquire software is immature, Customs can do neither effectively. Therefore, its software acquisition and development processes must be strengthened before it can hope to acquire or develop ACE software on time and within budget, as well as to achieve expected benefits.
subsections: section_title: Recommendations
paragraphs: In addition to previous recommendations to improve Customs’ management of information technology, we recommend that Customs correct the management and technical weaknesses discussed in this report before building ACE. To accomplish this, GAO recommends that the Commissioner of Customs, with the support of Customs’ CIO, ensure that Customs (1) rigorously analyze alternative approaches to building ACE, including ITDS as an alternative to developing ACE entirely within Customs and; (2) make investment decisions incrementally, i.e., for each increment: use disciplined processes to prepare a rigorous life-cycle cost estimate, including an explicit discussion of its inherent uncertainty; prepare realistic and supportable benefit expectations; require a favorable return-on-investment and compliance with Customs’ architecture before making any investment; and validate actual costs and benefits once an increment is piloted, compare these with estimates, use the results in making further decisions on subsequent increments, and report the results to Customs’ House and Senate appropriations and authorizing committees; and (3) strengthen ACE software acquisition management by: establishing an effective process improvement program and correcting the weaknesses in ACE software acquisition processes identified in this report, thereby bringing ACE processes to at least SEI level 2 and requiring at least SEI level 2 processes of all ACE software contractors.
section_title: Agency Comments
paragraphs: In its written comments on a draft of this report, Customs agreed with our conclusions and recommendations and stated that it is committed to addressing the problems discussed in the report. To this end, Customs cited a number of actions that it has underway and planned to improve management of information technology in general, and ACE specifically. For example, Customs stated that it has recruited a CIO with extensive experience in enterprise architecture and major systems acquisition, reorganized its Office of Information and Technology, and revised its System Development Life Cycle process. Also, Customs stated that it has engaged a contractor to update and improve the ACE cost-benefit analysis and plans for another contractor to independently review the new ACE cost-benefit analysis. Additionally, Customs reported that it revised its investment management procedures to provide for effective systems architecture enforcement, as we recommended in our May 1998 report.The report has been modified to reflect this information. To fully correct the management and technical weaknesses discussed in this report, Customs must follow through and effectively implement its plans to address all of our recommendations. Appendix I provides the full text of Customs’ comments.