Customs Service Modernization: Ineffective Software Development Processes Increase Customs System Development Risks
Published: 1999-02-11T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO reviewed the Customs Service's software development maturity and improvement activities, focusing on: (1) the maturity of Customs' software development processes; and (2) whether Customs has an effective software process improvement program.
section_title: What GAO Found
paragraphs: GAO noted that: (1) because of the number and severity of Customs' software development process weaknesses, Customs did not fully satisfy any of the key process areas (KPA) necessary to achieve the repeatable level of process maturity; (2) as a result, its processes for developing software, a complex and expensive component of Customs' systems, are ad hoc, sometimes chaotic, and not repeatable across projects; (3) Customs had some practice strengths in all but one of the five KPAs evaluated (i.e., requirements management, software project planning, software project tracking and oversight, software quality assurance, and software configuration management); however, GAO also found extensive and significant weaknesses in each of these KPAs; (4) some of these weaknesses were systemic, recurring in each of the KPAs; (5) for example, Customs had no written policy for managing or implementing any of the KPAs; (6) none of the projects had: (a) an approved quality assurance plan; (b) documented procedures for determining the project cost, schedule, or effort; or (c) any outside group reviewing or reporting on the project's compliance with defined processes; (7) these weaknesses are some of the reasons for Customs' limited success, for example, in delivering promised Automated Commercial Environment (ACE) capabilities on time; (8) Customs does not have a software development process improvement program, and it has not taken the basic steps to initiate one; (9) these steps, many of which are described in Software Engineering Institute's initiating, diagnosing, establishing, acting, and leveraging model for process improvement, include assigning responsibility and authority for process improvement, establishing a process improvement management structure, defining a plan of action, and committing needed resources; and (10) until Customs establishes an effective process improvement program, its software processes will remain poorly defined and undisciplined, and its software projects are likely to suffer cost, schedule, and performance shortfalls.

section_title: Letter
paragraphs: This report responds to your requests that we review the U. S. Customs Service’s software development maturity and improvement activities. Customs plans to spend hundreds of millions of dollars developing and maintaining software-intensive systems. We found that Customs’ software development processes are immature and are making recommendations to the Commissioner of Customs for strengthening them.
We are sending copies of this report to the Secretary of the Treasury; Commissioner of Customs; Director of the Office of Management and Budget; Ranking Minority Members of the Subcommittee on Treasury and General Government of the Senate Committee on Appropriations and the Subcommittee on Treasury, Postal Service, and General Government of the House Committee on Appropriations; and other congressional committees. We will also make copies available to other interested parties upon request. If you have questions or wish to discuss the issues in this report, please contact me at (202) 512-6240. Major contributors to this report are listed in appendix II.
subsections: section_title: Introduction
paragraphs: The mission of the Customs Service is to ensure that all goods and persons entering and exiting the United States do so in compliance with all U.S. laws and regulations. It does this by (1) enforcing the laws governing the flow of goods and persons across the borders of the United States and (2) assessing and collecting duties, taxes, and fees on imported merchandise. During fiscal year 1997, Customs collected $22.1 billion in revenue at more than 300 ports of entry and reported that it processed nearly 450 million passengers who entered the United States during the year.
To accomplish its mission, Customs is organized into six lines of business—trade compliance, outbound, passenger, finance, human resources, and investigations. Each business area is described below.
Trade compliance includes enforcement of laws and regulations associated with the importation of goods into the United States. To do so, Customs (1) works with the trade community to promote understanding of applicable laws and regulations, (2) selectively examines cargo to ensure that only eligible goods enter the country, (3) reviews documentation associated with cargo entries to ensure that they are properly valued and classified, (4) collects billions of dollars annually in duties, taxes, and fees associated with imported cargo, (5) assesses fines and penalties for noncompliance with trade laws and regulations, (6) seizes and accounts for illegal cargo, and (7) manages the collection of these moneys to ensure that all trade-related debts due to Customs are paid and properly accounted for.
Outbound includes Customs enforcement of laws and regulations associated with the movement of merchandise and conveyances from the United States. To do so, Customs (1) selectively inspects cargo at U.S. ports to guard against the exportation of illegal goods, such as protected technologies, stolen vehicles, and illegal currency, (2) collects, disseminates, and uses intelligence to identify high-risk cargo and passengers, (3) seizes and accounts for illegal cargo, (4) assesses and collects fines and penalties associated with the exportation of illegal cargo, and (5) physically examines baggage and cargo at airport facilities for explosive and nuclear materials. In addition, the outbound business includes collecting and disseminating trade data within the federal government. Accurate trade data are crucial to establishing accurate trade statistics on which to base trade policy decisions and negotiate trade agreements with other countries. By the year 2000, Customs estimates that exports will be valued at $1.2 trillion, compared to a reported $696 million in 1994.
Passenger includes processing all passengers and crew of arriving and departing (1) air and sea conveyances and (2) land vehicles and pedestrians. In fiscal year 1997, Customs reported it processed nearly 450 million travelers and, by the year 2000, expects almost 500 million passengers to arrive in the United States annually. Many of Customs’ passenger activities focus on illegal immigration and drug smuggling and are coordinated with other federal agencies, such as the Immigration and Naturalization Service and the Department of Agriculture’s Animal and Plant Health Inspection Service. Activities include targeting high-risk passengers, which requires timely and accurate information, and physically inspecting selected passengers, baggage, and vehicles to determine compliance with laws and regulations.
Finance includes asset and revenue management activities. Asset management consists of activities to (1) formulate Customs’ budget, (2) properly allocate and distribute funds, and (3) acquire, manage, and account for personnel, goods, and services. Revenue management encompasses all Customs activities to identify and establish amounts owed Customs, collect these amounts, and accurately report the status of revenue from all sources. Sources of revenue include duties, fees, taxes, other user fees, and forfeited currency and property. The revenue management activities interrelate closely with the revenue collection activities in the trade compliance, outbound, and passenger business areas.
Human resources is responsible for filling positions, providing employee benefits and services, training employees, facilitating workforce effectiveness, and processing personnel actions for Customs’ 18,000 employees and managers. Investigations includes activities to detect and eliminate narcotics and money laundering operations. Customs works with other agencies and foreign governments to reduce drug-related activity by interdicting (seizing and destroying) narcotics, investigating organizations involved in drug smuggling, and deterring smuggling efforts through various other methods. Customs also develops and provides information to the trade and carrier communities to assist them in their efforts to prevent smuggling organizations from using cargo containers and commercial conveyances to introduce narcotics into the United States.
To carry out its responsibilities, Customs relies on information systems and processes to assist its staff in (1) documenting, inspecting, and accounting for the movement and disposition of imported goods and (2) collecting and accounting for the related revenues. Customs expects its reliance on information systems to increase as a result of its burgeoning workload. For 1995 through 2001, Customs estimates that the annual volume of import trade between the United States and other countries will increase from $761 billion to $1.1 trillion. This will result in Customs processing an estimated increase of 7.5 million commercial entries—from 13.1 million to 20.6 million annually—during the same period. Recent trade agreements, such as the North American Free Trade Agreement (NAFTA), have also increased the number and complexity of trade provisions that Customs must enforce.
Customs recognizes that its ability to process the growing volume of imports while improving compliance with trade laws depends heavily on successfully modernizing its trade compliance process and its supporting automated systems. To speed the processing of imports and improve compliance with trade laws, the Congress enacted legislation that eliminated certain legislatively mandated paper requirements and required Customs to establish the National Customs Automation Program (NCAP). The legislation also specified certain functions that NCAP must provide, including giving members of the trade community the capability to electronically file import entries at remote locations and enabling Customs to electronically process “drawback” claims. In response to the legislation, Customs began in 1994 to modernize the information systems that support operations.
subsections: section_title: Current Projects: A Brief Description
paragraphs: Customs has several projects underway to develop and acquire new software and evolve (i.e., maintain) existing software to support its six business areas. Customs’ fiscal year 1998 budget for information management and technology activities was about $147 million.
Customs’ major information technology effort is its Automated Commercial Environment (ACE) system. In 1994, Customs began to develop ACE to replace its existing automated import system, the Automated Commercial System (ACS). ACE is intended to provide an integrated, automated information system for collecting, disseminating, and analyzing import-related data and ensuring the proper collection and allocation of revenues, totaling about $19 billion annually. According to Customs, ACE is planned to automate critical functions that the Congress specified when it established NCAP.
Customs reported that it spent $47.8 million on ACE as of the end of fiscal year 1997. In November 1997, Customs estimated it would cost $1.05 billion to develop, operate, and maintain ACE over the 15 years from fiscal year 1994 through fiscal year 2008. Customs plans to deploy ACE to more than 300 ports that handle commercial cargo imports.
Customs plans to develop and deploy ACE in multiple phases. According to Customs, the first phase, known as NCAP, is an ACE prototype. Customs currently plans to deploy NCAP in four releases. The first release was deployed for field evaluation at three locations in May 1998, and the fourth is scheduled for 1999. Customs, however, has not adhered to previous NCAP deployment schedules. Specifically, implementation of the NCAP prototype slipped from January 1997 to August 1997 and then again to a series of four releases beginning in October 1997, with the fourth release starting in June 1998.
Customs also has several other projects underway to modify or enhance existing systems that support its six business areas. For example, in fiscal year 1998, Customs planned to spend about $3.7 million to enhance its Automated Export System (AES), which supports the outbound business area and is designed to improve Customs’ collection and reporting of export statistics and to enforce export regulations. In addition, Customs planned to spend another $4.6 million to maintain its administrative systems supporting its finance and human resource business areas.
section_title: Objectives, Scope, and Methodology
paragraphs: The Chairman, Subcommittee on Treasury and General Government, Senate Committee on Appropriations, and the Chairman, Subcommittee on Treasury, Postal Service and General Government, House Committee on Appropriations, requested that we review Customs’ ability to develop software for its computer systems. Our objectives were to determine (1) the maturity of Customs’ software development processes and (2) the effectiveness of Customs’ software process improvement program.
To determine Customs’ software development process maturity, we applied the Software Engineering Institute’s (SEI) Software Capability ) and its Software Capability Evaluation (SCE) method. SEI’s expertise in software process maturity as well as its capability maturity models and evaluation methods are widely accepted throughout the software industry. All our specialists were SEI-trained.
The SW-CMM ranks organizational maturity according to five levels. (See figure 1.1.) Maturity levels 2 through 5 require the verifiable existence and use of certain software development processes, known as key process areas (KPA). According to SEI, an organization that has these processes in place is in a much better position to successfully develop software than an organization that does not have these processes in place. We evaluated Customs’ software development processes against five of the six level 2 KPAs.
subsections: section_title: Disciplined
paragraphs: Basic project management processes are established to track cost, schedule, and functionality. The necessary process discipline  is in place to repeat earlier successes on  projects with similar applications.
The software process is characterized as ad hoc, and occasionally even chaotic. Few  processes are defined, and success depends  on individual effort.
The sixth level 2 KPA, software subcontract management, was not evaluated because Customs did not use subcontractors on any of the projects that we evaluated. (See table 1.1.)
As established by the model, each KPA contains five common attributes that indicate whether the implementation and institutionalization of a KPA can be effective, repeatable, and lasting. The five common features are:
Commitment to perform: The actions that the organization must take to establish the process and ensure that it can endure. Commitment to perform typically involves establishing organizational policies and senior management sponsorship.
Ability to perform: The preconditions that must exist in the project or organization to implement the software development process competently. Ability to perform typically involves resources, organizational structures, and training.
Activities performed: The roles and procedures necessary to implement a KPA. Activities performed typically involve establishing plans and procedures, performing the work, tracking it, and taking appropriate management actions.
Measurement and analysis: Activities performed to measure the process and analyze the measurements. Measurement and analysis typically includes defining the measurements to be taken and the analyses to be conducted to determine the status and effectiveness of the activities performed.
Verifying implementation: The steps to ensure that the activities are performed in compliance with the process that has been established. Verification typically encompasses reviews by management.
In accordance with SEI’s SCE method and, for five of the six KPAs in level 2, we evaluated Customs’ institutional policies and practices and compared project-specific guidance and practices against the five common attributes. This project-specific comparison can result in one of four possible outcomes: (1) project strength—an effective implementation of the key practice, (2) project weakness—ineffective implementation of a key practice or failure to implement a key practice, (3) project observation—key practice evaluated but evidence inconclusive and cannot be characterized as either strength or weakness, and (4) not rated—key practice not currently relevant to project, therefore, not evaluated.
We performed the project-specific evaluations on three ongoing Customs software development projects, each of which is described below. As requested by the Subcommittee Chairmen, one of the projects evaluated was ACE, which is the largest and most important system that Customs is developing. The other two projects were selected by Customs on the basis of the following GAO specified criteria: (1) each project should be managed by a different software team, (2) at least one project should involve a legacy system, (3) at least one project should involve Year 2000 software conversion, and (4) each project should be relatively large and important to accomplishing Customs’ mission. The projects we evaluated are:
National Customs Automation Program (NCAP 0.1): NCAP 0.1 was the first component of the National Customs Automation Program Prototype (NCAP/P). NCAP/P, in turn, is the first phase of the Automated Commercial Environment (ACE). Customs began developing ACE in 1994 to address the new import processing requirements established by the National Customs Automation Program. ACE is also intended to replace the agency’s legacy automated import system, the Automated Commercial System (ACS). NCAP 0.1 was installed at three field locations in May 1998.
Automated Export System (AES): AES is an export information gathering and processing system, developed through cooperative efforts by Customs, the Bureau of Census, other federal agencies with export missions, and the export trade community. AES is designed to improve the collection of trade statistics; assist in the creation of a paperless export environment; facilitate the release of exports subject to licensing requirements; and consolidate export data required by several government agencies, easing the data filing burden for exporters while streamlining the federal data collection process.
Customs installed AES in all U.S. vessel ports in October 1996, and currently it is operational in all ports, including air, rail, and truck transit ports. Customs and Census officials estimate that they spent approximately $12.9 million to develop and implement AES from fiscal year 1992 to 1997. These costs included, among other things, expenses for contractors, travel, and training. According to Customs’ and Census’ figures, both agencies estimate that together they will spend an additional $32.2 million through fiscal year 2002 on AES implementation and maintenance.
Administrative Security System: The Administrative Security System assists users in requesting access to administrative systems. Users’ requests are electronically submitted to the appropriate official for approvals. In addition, other portions of the Administrative Security System provide functionality to allow the System Administrators the ability to prepare and maintain user profiles, request logs, and electronic approval and disapproval reports.
To assess the effectiveness of Customs’ software process improvement program, we interviewed the Director, Technical Architecture Group, Office of Information and Technology, to determine: (1) process improvements that are planned and underway, (2) the rationale for each initiative, (3) the relative priority of each, (4) progress made on each initiative, and (5) obstacles, if any, impeding progress. We also reviewed past process improvement plans, meeting minutes, and related documentation. Further, we reviewed SEI’s model for software process improvement, known as IDEALSM. IDEAL defines five sequential phases of software process improvement that can be used to develop a long range, integrated plan for initiating and managing a software process improvement program.
Customs provided written comments on a draft of this report. These comments are presented and evaluated in chapter 8, and are reprinted in appendix I. We performed our work at Customs’ Newington, Virginia, Data Center from February 1998 through November 1998, in accordance with generally accepted government auditing standards.
section_title: Requirements Management
paragraphs: The purpose of requirements management is to establish agreement between the customer and the software developers of the customer’s requirements that will be implemented by the software developers. This agreement typically is referred to as the “system requirements allocated to the software.” The agreement covers both technical and nontechnical (e.g., delivery dates) requirements. The agreement forms the basis for estimating, planning, performing, and tracking the software developer’s activities throughout the software life cycle.
According to the SW-CMM, a repeatable requirements management process, among other things, includes (1) documenting the system requirements allocated to software, (2) providing adequate resources and funding for managing the allocated requirements, (3) following a written organizational policy for requirements management, (4) having a quality assurance group that reviews the activities and work products for managing allocated requirements and reports the results, (5) using the allocated requirements as the basis for software plans, work products, and activities, and (6) training members of the software engineering group to perform their requirements management activities.
subsections: section_title: Customs’ Requirements Management Process Does Not Satisfy SEI’s Criteria
paragraphs: All three projects had practice strengths in this KPA. For example, each project documented the system requirements allocated to software and ensured that adequate resources and funding for managing the allocated requirements were provided. One of the projects, NCAP 0.1, had strengths in all but two practices under this KPA; however, each practice weakness is significant.
Collectively, the projects had many weaknesses in this KPA, and thus Customs’ requirements management processes do not meet “repeatable” maturity level criteria. For example, none of the projects had a written organizational policy governing requirements management, and none had a quality assurance group for reviewing and reporting on the activities and work products associated with managing the allocated requirements. In the absence of these two practices, management is missing two means for ensuring that software requirements are managed in a prescribed manner. Also, two of the projects did not use the allocated software requirements as the basis for software plans, work products, and activities, which increases the risk that the software developed will not fully satisfy requirements. Further, members of two projects’ software engineering groups were not trained to perform requirements management activities, thus increasing the chances of mismanagement.
Table 2.1 provides a comprehensive list of the three projects’ strengths and weaknesses for the requirements management KPA. The specific findings supporting the practice ratings cited in table 2.1 are in tables 2.2 through 2.4.
section_title: Conclusions
paragraphs: While Customs’ projects had several practice strengths in this KPA, the number and significance of their practice weaknesses mean that Customs’ ability to manage software requirements is not repeatable. As a result, Customs is at risk of producing systems that fail to provide promised capabilities, and cost more and take longer than necessary.
section_title: Software Project Planning
paragraphs: The purpose of software project planning is to establish reasonable plans for performing the software engineering and for managing the software project. According to the SW-CMM, a repeatable software project planning process, among other things, includes (1) documenting the software project plan, and preparing plans for software engineering facilities and support tools, (2) identifying the work products needed to establish and maintain control of the software project, (3) following a written organizational policy for planning a software project, (4) having a quality assurance group that reviews the activities and work products for software project planning and reports the results, (5) estimating the software project’s efforts and costs, and estimating its critical computer resources according to a documented procedure, (6) making and using measurements to determine the status of planning activities, and (7) training personnel in software project planning and estimating.
subsections: section_title: Customs Is Not Performing Software Project Planning Effectively
paragraphs: All of the projects that we evaluated had key practice strengths in this KPA. For example, all had strengths in (1) documenting a software project plan and preparing plans for the software engineering facilities and support tools needed to develop the software and (2) identifying the work products needed to control the software project. NCAP 0.1, in particular, had many additional practice strengths.
However, many significant practice weaknesses were found in all three projects. None of the projects followed an organizational software project planning policy, and none had a quality assurance group conducting reviews and/or audits. As a result, the projects performed these practices differently and inconsistently, and controls were unreliable. For example, while the NCAP 0.1 project followed a documented procedure for estimating the size of software work products (or changes to the size of work products), and made and used measurements to determine the status of software planning activities, neither of the other two projects performed these practices and none of the projects had personnel trained in software project planning and estimating. Such project planning weaknesses mean that management has no assurance that it will get the consistent, complete, and reliable information about the projects’ expected costs and schedules needed to make expeditious and informed investment decisions.
Table 3.1 provides a comprehensive list of the three projects’ strengths, weaknesses, and observations for the software project planning KPA. The specific findings supporting the practice ratings cited in table 3.1 are in tables 3.2 through 3.4.
section_title: Conclusions
paragraphs: Effective planning is the cornerstone of successful software development project management. While Customs showed some strengths in this KPA, its many weaknesses render its software project planning processes unrepeatable. Therefore, Customs has no assurance that the projects are effectively establishing plans, including reliable projections of costs and schedules, and effectively measuring and monitoring progress and taking needed corrective actions expeditiously.
section_title: Software Project Tracking and Oversight
paragraphs: The purpose of software project tracking and oversight is to provide adequate visibility into the progress of the software development so that management can act effectively when the software project’s performance deviates significantly from the software plans. Software project tracking and oversight involves tracking and reviewing the software accomplishments and results against documented estimates, commitments, and plans, and adjusting these plans based on the actual accomplishments and results.
According to the SW-CMM, effective software project tracking and oversight, among other things, includes (1) designating a project software manager to be responsible for the project’s software activities and results, (2) having a documented software development plan for tracking software activities and communicating status, (3) following a written organizational policy for managing the project, (4) conducting periodic internal reviews to track technical progress, plans, performance, and issues against the software development plan, (5) tracking the software risks associated with the cost, resource, schedule, and technical aspects of the project, (6) explicitly assigning responsibility for software work products and activities, (7) tracking the sizes of the software work products (or sizes of the changes to the software work products) and taking corrective actions as necessary, and (8) periodically reviewing the activities for software project tracking and oversight with senior management.
subsections: section_title: Customs’ Software Project Tracking and Oversight Process Is Immature
paragraphs: The projects evaluated exhibited some software project tracking and oversight practice strengths. For example, all three of the projects had a project software manager designated to be responsible for the project’s software activities and results, and all had a documented software development plan for tracking software activities and communicating status. Also, NCAP 0.1 had strengths in all but five of this KPA’s 24 key practices.
However, the three projects collectively had many weaknesses, and these weaknesses, including the five for NCAP 0.1, were significant and thus preclude Customs from meeting SEI’s repeatable maturity level criteria. For example, none of the projects followed a written organizational policy for managing the software project. With no established policy, Customs increases the risk that key tracking and oversight activities will not be performed effectively. For example, for two of the three projects, the project managers did not (1) conduct periodic internal reviews to track technical progress, plans, performance, and issues against the software development plan, (2) track software risks associated with cost, resource, schedule, and technical aspects of the project, (3) explicitly assign responsibility to individuals for software work products and activities, (4) track the sizes of the software work products (or sizes of the changes to the software work products) and take corrective actions, or (5) periodically review software project tracking and oversight activities with senior management.
Table 4.1 provides a comprehensive list of the three projects’ strengths, weaknesses, and observations for the software project tracking and oversight KPA. The specific findings supporting the practice ratings cited in table 4.1 are in tables 4.2 through 4.4.
section_title: Conclusions
paragraphs: Despite several practice strengths in this KPA, the number and significance of the practice weaknesses that we found mean that Customs’ current process for tracking and overseeing its projects is not repeatable, thereby increasing the chances of its software projects being late, costing more than expected, and not performing as intended.
section_title: Software Quality Assurance
paragraphs: The purpose of software quality assurance is to independently review and audit the software products and activities to verify that they comply with the applicable procedures and standards and to provide the software project and higher-level managers with the results of these independent reviews and audits.
According to the SW-CMM, a repeatable software quality assurance process, among other things, includes (1) preparing a software quality assurance plan for the project according to a documented procedure, (2) having a written organizational policy for implementing software quality assurance, (3) conducting audits of designated work processes and products to verify compliance, (4) documenting deviations identified in the software activities and software work products and handling them according to a documented procedure, and (5) having experts independent of the software quality assurance group periodically review the activities and work products of the project’s software quality assurance group.
subsections: section_title: Customs Lacks a Software Quality Assurance Process
paragraphs: All of the projects evaluated had extensive and significant software quality assurance practice weaknesses. For example, two of the projects did not have a software quality assurance plan; and none of the projects (1) had a written organizational policy for implementing software quality assurance, (2) conducted audits of designated work products to verify compliance, (3) documented deviations identified in the software activities and software work products and handled them according to a documented procedure, or (4) had experts independent of the software quality assurance group periodically review the group’s work products. In fact, only one of the projects, AES, had any software quality assurance practice strengths, and these strengths were limited to only a few practices. In this case, the project had assigned responsibility for software quality assurance to a single individual and, for example, a software quality assurance plan had been drafted, although not according to a documented procedure. This virtual absence of software quality assurance on Customs’ software projects increases greatly the risk of software process and product standards not being met, which in turn increases the risk of software not performing as intended, and costing more and taking longer to develop than necessary.
Table 5.1 provides a comprehensive list of the three projects’ strengths, weaknesses, and observations for the software quality assurance KPA. The specific findings supporting the practice ratings cited in table 5.1 are in tables 5.2 through 5.4.
section_title: Conclusions
paragraphs: Customs’ software quality assurance process has many weaknesses and is, therefore, undefined and undisciplined. As a result, Customs cannot provide management with independent information about adherence to software process and product standards. To develop and maintain software effectively, Customs must adopt a structured and rigorous approach to software quality assurance.
section_title: Software Configuration Management
paragraphs: The purpose of software configuration management is to establish and maintain the integrity of the products of the software project throughout the project’s software life-cycle. Software configuration management involves establishing product baselines and systematically controlling changes to them.
According to the SW-CMM, a repeatable software configuration management process, among other things, includes (1) preparing a software configuration management plan according to a documented procedure, (2) establishing a configuration management library system as a repository for the software baselines, (3) identifying software work products to be placed under configuration management, (4) controlling the release of products from the software baseline library according to a documented procedure, (5) following a written organizational policy for implementing software configuration management, (6) recording the status of configuration items/units according to a documented procedure, (7) making and using measurements to determine the status of the software configuration management activities, and (8) reviewing software configuration management activities with senior management on a periodic basis.
subsections: section_title: Customs’ Software Configuration Management Process Is Immature
paragraphs: Customs’ processes for software configuration management show strengths in several activities. For example, all three projects had developed software configuration management plans according to a documented procedure. Also, two of the projects (NCAP 0.1 and AES) established configuration management library systems as repositories for the software baselines, identified software work products to be placed under configuration management, and controlled the release of products from the software baseline library according to a documented procedure.
However, the projects had many practice weakness that collectively jeopardize Customs’ ability to maintain the integrity of the projects’ software products. For example, none of the projects had a written organizational policy for implementing software configuration management, and none had documented procedures for recording the status of configuration items (e.g., code, documents). Moreover, none of the projects made or used measurements to determine the status of the software configuration management activities, or reviewed software configuration management activities with senior management on a periodic basis.
Table 6.1 provides a comprehensive list of the three projects’ strengths and weaknesses for the software configuration management KPA. The specific findings supporting the practice ratings cited in table 6.1 are in tables 6.2 through 6.4.
section_title: Conclusions
paragraphs: Customs has many configuration management process weaknesses, and thus its capability to establish and maintain the integrity of the wide range of software products is nonrepeatable and ineffective. Without a mature configuration management process, Customs can lose control of the current software product baseline, potentially producing and using inconsistent product versions and creating operational problems.
section_title: Customs Lacks a Software Process Improvement Program
paragraphs: To consistently develop software with specified functionality on time and within budget, Customs must improve its software development processes. According to SEI, an effective process improvement program includes (1) establishing a process improvement management structure, (2) developing a process improvement plan, (3) determining the organization’s baseline capability and using this as a basis for targeting process initiatives, and (4) dedicating adequate resources for implementing the plan.
Although it has attempted in the past to initiate and sustain process improvement activities, these activities were terminated without having improved Customs processes. Currently, Customs has no software process improvement program.
subsections: section_title: SEI Has Defined a Five-Phase Model for Software Process Improvement
paragraphs: In 1996, SEI published a software process improvement model, called IDEAL. This model has five phases: Initiating, Diagnosing, Establishing, Acting, and Leveraging—IDEAL. Each of the phases is summarized below.
Initiating phase: During this phase, an organization establishes the management structure of the process improvement program, defines and assigns roles and responsibilities, allocates initial resources, develops a plan to guide the organization through the first three phases of the program, and obtains management approval and funding for the program. Two key organizational components of the program management structure established during this phase are a management steering group and a software engineering process group (SEPG). Responsibility for this phase rests with senior management.
Diagnosing phase: During this phase, the SEPG appraises the current level of software process maturity to establish a baseline of the organization’s process capability, and identifies any ongoing process improvement initiatives. The SEPG then uses the baseline to identify weaknesses and target process improvement activities. It also compares these targeted activities with any ongoing process improvement activities and reconciles any differences. Responsibility for this phase rests primarily with line managers and practitioners.
Establishing phase: During this phase, the SEPG prioritizes the software process improvement activities and develops strategies for pursuing them. It then develops a process improvement action plan that details the activities and strategies and includes measurable goals for the activities and metrics for monitoring progress against the goals. Also during this phase, the resources needed to implement the plan are committed and training is provided for SEPG’s technical working groups, who will be responsible for developing and testing new or improved processes. Responsibility for this phase resides primarily with line managers and practitioners.
Acting phase: In this phase, the work groups create and evaluate new and improved processes. Evaluation of the processes is based on pilot tests that are formally planned and executed. If the pilots are successful, the work groups develop plans for organization-wide adoption and institutionalization, and once approved, execute them. Responsibility for this phase resides primarily with line managers and practitioners.
Leveraging phase: During this phase, results and lessons learned from earlier phases are assessed and applied, as appropriate, to enhance the process improvement program’s structure and plans. Responsibility for this phase rests primarily with senior management.
section_title: Customs Does Not Currently Have a Software Process Improvement Program
paragraphs: In 1996, Customs initiated some limited software process improvement activities. Specifically, it hired a contractor to develop a process improvement plan, which was completed in September 1996. According to the plan, Customs was to reach CMM level 2 process maturity (the repeatable level) by 1998 and CMM level 3 (the defined level) by 2002. Customs began limited implementation of the plan in May of 1997, when it established process improvement teams for two KPAs—software project planning and project tracking and oversight. Generally, the teams were tasked with defining, implementing, and maintaining CMM-based processes for their respective KPAs. Customs did not staff or fund any other KPA improvement activities at this time. In August 1997, Customs discontinued all process improvement activities. Customs officials stated that this decision was based on the need to focus staff and resources on the agency’s Year 2000 conversion program.
Currently, Customs does not have a software development process improvement program, and it has not taken steps to initiate one. Although it has assigned two people part-time to process improvement, it has not assigned organizational responsibility and authority, established a program management structure, developed a plan of action, and committed resources needed (trained staff and funding) to execute the plan.
section_title: Conclusions
paragraphs: Customs does not have an effective software development process improvement program. As a result, it cannot expect to improve its immature software development processes.
section_title: Overall Conclusions, Recommendations, and Agency Comments
paragraphs: Customs develops and maintains software for systems that are critical to its ability to fulfill its mission. However, its software development processes are ad hoc and sometimes chaotic, and are not repeatable even on a project-by-project basis. As a result, Customs’ success or failure in developing software depends largely on specific individuals, rather than on well-defined and disciplined software management practices. This greatly reduces the probability that its software projects, whether new developments or maintenance of existing software, will consistently perform as intended and be delivered on schedule and within budget. For Customs software projects to mature beyond this initial level, the agency must implement basic management controls and instill self-discipline in its software projects.
Customs acknowledges the importance of software process maturity and the need to improve its software development processes. However, it does not have a program for improving its software development processes and has not begun to establish one. Until it does, Customs has no assurance that its large investment in software development and maintenance will produce systems that perform needed functions, on time, and within budget.
subsections: section_title: Recommendations
paragraphs: We recommend that, after ensuring that its mission-critical systems are Year 2000 compliant but before investing in major software development efforts like ACE, the Commissioner of Customs direct the Chief Information Officer to assign responsibility and authority for software development process develop and implement a formal plan for software development process improvement that is based on the software capability evaluation results contained in this report and specifies measurable goals and time frames, prioritizes initiatives, estimates resource requirements (trained staff and funding) and defines a process improvement management structure; ensure that every new software development effort in Customs adopts processes that satisfy at least SW-CMM level 2 requirements; and ensure that process improvement activities are initiated for all ongoing essential software maintenance projects.
section_title: Agency Comments and Our Evaluation
paragraphs: In its written comments on a draft of this report, Customs acknowledged the importance of software process improvement and maturity. Also, it agreed with GAO’s overall findings, including that Customs’ software development processes have not attained SW-CMM level 2 maturity.
To address these weaknesses, Customs stated that it has taken the first step toward implementing our recommendations by assigning responsibility and authority for software process improvement as part of a reorganization of its Office of Information and Technology, which Customs stated will be implemented in early 1999. Customs further stated that once the reorganization is implemented, a formal software process improvement program will be established, and that this program will include definition of an action plan, commitment of resources, and specification of goals for achieving CMM levels 2 and 3. According to Customs, these improvement activities are in their early stages. When they are successfully implemented, they should address many of our recommendations.
Customs also stated that because its legacy systems are aging and need to be enhanced and replaced, software process improvement must occur in parallel with continued software development investments. History has shown that attempting to modernize without first instituting disciplined software processes has been a characteristic of failed modernization programs. Until it implements disciplined software processes (i.e., at least level 2 process maturity), Customs cannot prudently manage major system investments, such as ACE with an estimated life cycle cost exceeding $1 billion.
Customs’ comments also included a request to meet with us to discuss system-specific KPA practice strength and weakness determinations. We met prior to requesting comments on a draft of this report and then again on January 12, 1999, to discuss SEI’s SW-CMM requirements and the basis for our determinations. We are prepared to continue assisting Customs as it improves its software processes.
Appendix I provides the full text of Customs’ comments and our responses to additional Customs comments not discussed above.