Information Technology: Customs Has Made Progress on Automated Commercial Environment System, but It Faces Long-Standing Management Challenges and New Risks
Published: 2006-05-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Homeland Security (DHS) is conducting a multiyear, multibillion-dollar acquisition of a new trade processing system, planned to support the movement of legitimate imports and exports and strengthen border security. By congressional mandate, plans for expenditure of appropriated funds on this system, the Automated Commercial Environment (ACE), must meet certain conditions, including GAO review. This study addresses whether the fiscal year 2006 plan satisfies these conditions; it also describes the status of DHS's efforts to implement prior GAO recommendations for improving ACE management, and provides observations about the plan and DHS's management of the program.
section_title: What GAO Found
paragraphs: The fiscal year 2006 ACE expenditure plan, including related program documentation and program officials' statements, either satisfied or partially satisfied the legislative conditions imposed by the Congress; however, more can be done to better address several aspects of these conditions. In addition, DHS has addressed some recommendations that GAO has previously made, but progress has been slow in addressing several recommendations aimed at strengthening ACE management. For example, DHS has more to do to implement the recommendation that it establish an ACE accountability framework that, among other things, ensures that expenditure plans report progress against commitments made in prior plans. Implementing a performance and accountability framework is important for ensuring that promised capabilities and benefits are delivered on time and within budget. In addition, describing progress against past commitments is essential to permit meaningful congressional oversight. Among GAO's observations about the ACE program and its management are several related to the need to effectively set and use performance goals and measures. Although the program set performance goals, these targets were not always realistic. For example, in fiscal year 2005, the program set a target that 11 percent of all Customs and Border Protection (CBP) employees would use ACE. However, this target does not reflect the fact that many CBP employees will never need to use the system. Additionally, the program has established 6 program goals, 11 business results, 23 benefits, and 17 performance measures, but the relationships among these are not fully defined or adequately aligned with each other. For example, not every goal has defined benefits, and not every benefit has an associated performance measure. Without realistic ACE performance measures and targets that are aligned with the overall program goals and desired results, DHS will be challenged in its efforts to establish an accountability framework for ACE that will help to ensure that the program delivers its expected benefits. In addition, DHS plans to develop several increments, referred to as "releases," concurrently; in the past, such concurrency has led to cost overruns and schedule delays because releases contended for the same resources, and resources that were to be used on later releases were diverted to earlier ones. However, because of DHS's belief that such concurrent development will allow it to deliver ACE functionality sooner, it is reintroducing the same problems that resulted in past shortfalls.

section_title: Letter
paragraphs: In February 2006, U.S. Customs and Border Protection (CBP), within the  Department of Homeland Security (DHS), submitted to the Congress its  fiscal year 2006 expenditure plan for the Automated Commercial  Environment (ACE) program. ACE is to be CBP’s new import and export  processing system. The program’s goals include    supporting border security by enhancing analysis and information sharing  with other government agencies and   streamlining time-consuming and labor-intensive tasks for CBP personnel  and the trade community through the provision of a single Web-based  interface.
DHS currently plans to acquire and deploy ACE in 11 increments, referred  to as “releases,” in approximately 8.5 years. The first three releases are  deployed and operating, and the fourth release is being deployed. Other  releases are in various states of definition and development. For the ACE  life-cycle, the risk-adjusted cost estimate is about $2.8 billion; through  fiscal year 2005, ACE-appropriated funding has been about $1.6 billion.
The Department of Homeland Security Appropriations Act of 2006   appropriates about $320 million for ACE. The act states that DHS may not  obligate any funds for ACE until it submits for approval to the House and  Senate Committees on Appropriations a plan for expenditure that  1.  meets the capital planning and investment control review requirements  established by the Office of Management and Budget (OMB), including  Circular A-11, part 7;  2.  complies with DHS’s enterprise architecture;  3.  complies with the acquisition rules, requirements, guidelines, and  systems-acquisition management practices of the federal government;  4. includes a certification by the DHS Chief Information Officer that an  independent verification and validation agent is currently under  contract;  5. is reviewed and approved by the DHS Investment Review Board  (IRB), the Secretary of Homeland Security, and OMB; and  6. is reviewed by GAO.
As required by the act, we reviewed the ACE fiscal year 2006 expenditure  plan. Our objectives were to (1) determine whether the expenditure plan  satisfies certain legislative conditions, (2) determine the status of our open  ACE recommendations, and (3) provide any other observations about the  expenditure plan and DHS’s management of the ACE program.
On March 10, 2006, we briefed your offices on the results of this review.  This report transmits the results of our work. The full briefing, including  our scope and methodology, can be found in appendix I.
subsections: section_title: Compliance with Legislative Conditions
paragraphs: The legislative conditions that the Congress placed on the use of fiscal  year 2006 ACE appropriated funds have been either partially or fully  satisfied by the latest expenditure plan and related program  documentation and activities. However, more can be done to better  address several aspects of these conditions. For example:    One legislative condition states that the plan should meet OMB’s capital  planning and investment control review requirements, which include  addressing security and privacy issues. However, a privacy impact  assessment for ACE has been in draft for several months and is not yet  approved. Another capital planning and investment control review  requirement is that performance goals and measures be provided in the  business case for ACE. Although CBP describes selected performance  goals and measures, the goals (i.e., targets) are not always realistic (we  provide further discussion of this issue later in this report).
According to another legislative condition, the expenditure plan must  comply with DHS’s enterprise architecture. However, DHS does not have a  documented methodology for evaluating programs for compliance with its  enterprise architecture, other than relying on the professional expertise of  its staff.
According to a third legislative condition, the DHS Chief Information  Officer is to certify that an independent verification and validation (IV&V)  agent is under contract. Although DHS satisfied this condition, the scope  of the IV&V contractor’s activities is not consistent with the operative  industry standard, which states that IV&V should extend to key system  products and development processes.
section_title: Status of Our Open Recommendations
paragraphs: CBP has addressed some recommendations, while progress has been slow  on others. Each recommendation, along with the status of actions to  address it, is summarized below.
Ensure that future expenditure plans are based on cost estimates that are  reconciled with independent cost estimates.
Complete. In October 2005, CBP, with contractor support, compared the  program plan cost estimate with the independent cost estimate. According  to the analysis performed, the two estimates are consistent.
Develop and implement a rigorous and analytically verifiable cost  estimating program that embodies the tenets of effective estimating, as  defined in the institutional and project-specific estimating models  developed by the Software Engineering Institute (SEI).
In progress. CBP has taken steps such as (1) hiring a contractor to  develop cost estimates (including contract task order cost estimates) that  are independent of CBP’s estimates, and (2) tasking a support contractor  with evaluating both the independent and CBP estimates against the  criteria defined by SEI. According to the results of the support  contractor’s evaluation, the independent estimates satisfied the SEI  criteria; CBP’s estimates largely satisfied the criteria.
However, according to the support contractor, CBP’s cost estimating had  limitations. First, the CBP estimate did not adequately consider past  projects in its cost and schedule estimates. In addition, the CBP estimate  was an aggregation of estimates developed separately for three ACE  components, each according to a different cost estimating methodology;  the support contractor advised against this approach, recommending that  component estimates be based on the same methodology.
Immediately develop and implement a human capital management strategy  that provides both near- and long-term solutions to program office human  capital capacity limitations, and report quarterly to the Appropriations  Committees on the progress of efforts to do so.
In progress. CBP has expanded its contractor and government workforce  dedicated to the ACE program by merging staff assigned to trade-related  legacy systems with the ACE program staff. In addition, it is beginning to  use subject matter experts from existing field operations advisory boards  to help program officials define requirements for future releases. However,  it does not have a documented human capital strategy covering its ACE  program.
Have future ACE expenditure plans specifically address any proposals or  plans, whether tentative or approved, for extending and using ACE  infrastructure to support other homeland security applications, including  any impact on ACE of such proposals and plans.
In progress. The expenditure plan describes steps both planned and under  way to ensure that ACE infrastructure supports both ACE and other  homeland security applications. For example, it states that both ACE and  the United States Visitor and Immigrant Status Indicator Technology (US- VISIT) program should conform to the DHS enterprise architecture, which  is to define standard shared services that the two systems can request.  Such a services oriented architecture is intended to promote reuse, as well  as reducing overlap and duplication.
Define measures, and collect and use associated metrics, for determining  whether prior and future program management improvements are  successful.
In progress. CBP continues to make changes that are intended to improve  overall program management, but it has not consistently defined measures  to determine whether the changes are successful. For example, CBP has  reorganized its Office of Information Technology; this reorganization is  intended to improve program management by providing (1) enhanced  government oversight of ACE development, (2) better definition of  requirements for future ACE releases, and (3) faster and cheaper delivery  of ACE capabilities. However, program officials told us that they have not  established measures or targets for determining whether the  reorganization is providing these benefits.
Define and implement an ACE accountability framework that fulfills  1.  The framework should cover all program commitment areas, including  key expected or estimated system (a) capabilities, use, and quality;  (b) benefits and mission value; (c) costs; and (d) milestones and  schedules.
In progress. CBP has prepared an initial version of an accountability  framework that it intends to improve as it proceeds. The framework is  built around measuring progress against costs, milestones, schedules,  and risks for select releases; however, the benefit measurement has  not been well defined, and the performance targets are not always  realistic.  2.  The framework should ensure currency, relevance, and completeness  of all program commitments made to the Congress in expenditure  plans.
In progress. The fiscal year 2006 expenditure plan includes inaccurate,  dated, and incomplete information and omits other relevant  information. For example, the plan did not include information  regarding CBP’s decision to eliminate the dependencies among the  screening and targeting releases and the cargo releases, and to take  advantage of the capabilities of its existing Automated Targeting  System.3.  The framework should ensure reliable data that are relevant to  measuring progress against commitments.
In progress. The data that CBP uses to measure progress against  commitments are not consistently reliable. For example, data in the  defect tracking system show that defects in Release 4 (which is now  operational) have not been closed; however, program officials told us  that many of these defects have been resolved.  4.  The framework should ensure that future expenditure plans report  progress against commitments contained in prior expenditure plans.
In progress. The current expenditure plan does not adequately  describe progress against commitments made in previous plans. For  example, the plan provides a summary of the funding requested in  each of the previous six expenditure plans, but it does not provide  information on whether these funding amounts were actually  expended or obligated as planned.  5.  The framework should ensure that criteria for exiting key readiness  milestones adequately consider indicators of system maturity, such as  the severity of open defects.
In progress. ACE milestone exit criteria provide for addressing the risk  associated with severe defects that are unresolved. Using these  criteria, CBP passed several release milestones with severe defects  still open. However, CBP officials were unable to provide us with any  documentation on how they assessed the inherent risks of passing  these milestones with open severe defects.  6.  The framework should ensure clear and unambiguous delineation of  the respective roles and responsibilities of the government and the  prime contractor.
Complete. The current ACE program plan describes general roles and  responsibilities for the government and the prime contractor. More  detailed roles have been documented in a roles and responsibilities  matrix that assigns primary responsibility for each activity.
Report quarterly to the House and Senate Appropriations Committees on  efforts to address our open recommendations.
In progress. CBP submitted quarterly reports to both Committees on its  efforts to address our open recommendations; however, progress in  addressing our recommendations was not always reported accurately.
section_title: Observations about ACE Management
paragraphs: We have several observations about the development of ACE releases, as  well as several more concerning the performance of ACE releases that are  deployed and operating.
ACE development: Steps have been taken to address a past pattern of  ACE release shortfalls, but new release management weaknesses are  emerging.
As we have previously observed, CBP established a pattern of borrowing  resources from future releases to address problems with the quality of  earlier releases; this led to schedule delays and cost overruns. This pattern  has continued with the most recently deployed cargo release, which  developed problems that caused delays with a subsequent screening and  targeting release. CBP took steps to mitigate this problem by eliminating  the dependencies between the cargo releases and the screening and  targeting releases.
However, CBP’s planned schedule for developing additional releases  includes a significant level of concurrence, because of CBP’s interest in  delivering ACE functionality sooner. Such concurrence between ACE  release activities has led to cost overruns and schedule delays in the past.  Thus, the revised ACE plans and actions are potentially reintroducing the  same problems that produced past shortfalls.
We made several specific observations related to these weaknesses,  including the following:    On two recent releases, key milestones were passed despite unresolved  severe defects. Officials told us that the risk of proceeding did not  outweigh the need to get the releases to users, and thereby gaining user  acceptance and feedback. However, the risks were not documented  and formally managed.
Concurrence in developing early ACE releases caused schedule slips  and cost overruns. Despite these experiences, CBP has established a  risky plan that involves considerable overlap across the development  schedules for three future releases.
Although the use of earned value management (EVM) is an OMB  requirement, it was not being used to manage the development of two  recent releases. For example, CBP discontinued use of EVM on one  release because this method was not familiar to staff who were  transferred to work on the program.
ACE operations: Operational performance has been mixed, and mission  impact is unclear.
ACE releases one through four are in operation. To date, these releases’  operational performance has been uneven. For example, ACE has largely  been meeting its goals for being available and responsive in processing  virtually all daily transactions, and has decreased truck processing times  at some ports. However, ACE is not being used by as many CBP and trade  personnel as was expected, and truck processing times at other ports have  increased. Moreover, overall user satisfaction has been low.
In addition, ACE goals, expected mission benefits, and performance  measures are not fully defined and adequately aligned with each other. For  example, not every goal has defined benefits, every benefit is defined only  in terms of efficiency gains, not every benefit has an associated business  result, and not every benefit and business result has associated  performance measures.
Further, where performance measures have been defined, the associated  targets are not always realistic. For example, the performance target in  fiscal year 2005 for ACE usage was that 11 percent of all CBP employees  would use ACE. However, that many CBP employees will never need to  use the system. This performance target does not reflect that.
Because performance measures are not always realistic or aligned with  program goals and benefits, it is unclear whether ACE has realized—or  will realize—the mission value that it was intended to bring to CBP’s and  other agencies’ trade- and border security-related operations.
section_title: Conclusions
paragraphs: The legislative conditions that the Congress placed on the use of fiscal  year 2006 ACE appropriated funds have been either partially or fully  satisfied by the latest expenditure plan and related program  documentation and activities. Nevertheless, more can be done to better  address several aspects of these conditions, such as ensuring that the  program’s privacy impact assessment is approved, measuring ACE  performance and results, ensuring architectural alignment, and employing  effective IV&V practices. Given that the legislative conditions are  collectively intended to promote accountability and increase the chances  of program success, it is important that each receives DHS’s full attention.
Also important to ACE’s success is the swift and complete implementation  of the recommendations that we have previously made to complement the  legislative conditions and improve program management, performance,  and accountability. In this regard, some recommendations have been  addressed, while progress has been slow on others, such as    accurately reporting to the Appropriations Committees on CBP’s progress  in implementing our prior recommendations;   developing and implementing a strategic approach to meeting the  program’s human capital needs;    using criteria for exiting key milestones that adequately consider  indicators of system maturity, such as severity of open defects and the  associated risks; and   developing and implementing a performance and accountability  framework for ensuring that promised capabilities and benefits are  delivered on time and within budget.
To its credit, CBP has taken several steps to stem the pattern of cost,  schedule, and performance shortfalls that it experienced on early ACE  releases. However, future releases are unlikely to realize the impact of  these steps because revised ACE plans and actions are reintroducing the  same pattern that led to early release shortfalls. This pattern includes not  formally and transparently considering, and proactively addressing, the  risks associated with passing key release milestones with known severe  defects; building considerable overlap and concurrence in the  development schedules of releases that will contend for the same  resources; and not performing EVM on all releases. If this pattern  continues, the prospects for a successful program will be diminished.
Although availability and responsiveness targets are largely being met and  long-standing help desk limitations are being addressed, the prospects for  a successful program nevertheless remain unclear. The true measure of  ACE’s success is arguably the mission value that it brings to CBP’s and  other agencies’ trade- and border security-related operations and users.  Such value depends both on the operational performance of ACE and on  CBP’s ability to demonstrate that this performance is achieving program  goals, delivering expected benefits, and producing desired business  results. At this juncture, however, neither the system’s performance nor its  value is clear because of several factors: the operational performance of  deployed releases has been mixed; users’ satisfaction has been low; the  relationships among goals, benefits, and desired business outcomes are  not evident; and the range of measures needed to create a complete and  realistic picture of ACE’s performance is missing.
In summary, a number of ACE activities have been and are being done  well; these have contributed to the program’s progress to date and will go  a long way in determining the program’s ultimate success. However, it will  be important for CBP to effectively address long-standing ACE  management challenges along with emerging problems. Until it does so,  ACE will remain a risky program.
section_title: Recommendations for Executive Action
paragraphs: To assist CBP in managing ACE—and increasing the chances that it will  deliver required capabilities on time and within budget, demonstrating  promised mission benefits and results—we recommend that the Secretary  of Homeland Security direct the appropriate departmental officials to fully  address those legislative conditions associated with having an approved  privacy impact assessment and ensuring architectural alignment.
We also recommend that the Secretary, through CBP’s Acting  Commissioner, direct the Assistant Commissioner for Information and  Technology to  fully address those legislative conditions associated with measuring ACE  performance and results and employing effective IV&V practices;   accurately report to the appropriations committees on CBP’s progress in  implementing our prior recommendations; include in the June 30, 2006, quarterly update report to the appropriations  committees a strategy for managing ACE human capital needs and the  ACE framework for managing performance and ensuring accountability;   document key milestone decisions in a way that reflects the risks  associated with proceeding with unresolved severe defects and provides  for mitigating these risks;    minimize the degree of overlap and concurrence across ongoing and  future ACE releases, and capture and mitigate the associated risks of any  residual concurrence;    use EVM in the development of all existing and future releases;    develop the range of realistic ACE performance measures and targets  needed to support an outcome-based, results-oriented accountability  framework, including user satisfaction with ACE; and   explicitly align ACE program goals, benefits, desired business outcomes,  and performance measures.
section_title: Agency Comments
paragraphs: In written comments on a draft of this report signed by the Director,  Departmental GAO/OIG Liaison, DHS agreed with our findings concerning  progress in addressing our prior recommendations, and it agreed with the  recommendations in this report. DHS also described actions that it has  under way and planned to address the recommendations. The  department’s comments are reprinted in appendix II.
We are sending copies of this report to the Chairmen and Ranking  Minority Members of other Senate and House committees and  subcommittees that have authorization and oversight responsibilities for  homeland security. We are also sending copies to the DHS Secretary, the  CBP Commissioner and, upon their request, to other interested parties. In  addition, the report will be available at no charge on the GAO Web site at  http://www.gao.gov.
Should you or your offices have any questions on matters discussed in this  report, please contact me at (202) 512-3459 or at hiter@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. Other contacts and key  contributors to this report are listed in appendix III.
section_title: Appendix I: Briefing to Subcommittees on Homeland Security, House and Senate Committees on Appropriations
subsections: section_title: Information Technology: Customs Has Made Progress on Automated Commercial Environment System, but it Faces Long-Standing Management Challenges and New Risks
subsections: section_title: March 10, 2006
section_title: Attachment 1: Scope and Methodology
section_title: The Department of Homeland Security’s (DHS) Customs and Border Protection (CBP)1 is about 5 years into its trade processing modernization program, known as the Automated Commercial Environment (ACE). The goals of ACE are as follows:
paragraphs: Support border security by enhancing analysis and information sharing with other  government agencies to deal with increasing new security threats to our nation.  Provide CBP personnel with the technology and information needed to decide, before  a shipment reaches the border, what should be targeted because it is a security  threat, and what should be expedited because it complies with U.S. laws.   Provide an integrated, fully automated information system to enable the efficient  collection, processing, and analysis of commercial import and export data.  Streamline time-consuming and labor-intensive tasks for CBP personnel and the  trade community, through a single, Web-based interface, reducing costs for the  government and the trade community.  Enable users to process, view, and manage their accounts nationally, and obtain  historical information on cargo, conveyances, and crew, based on screening and  targeting rules.  Enable CBP to comply with legislative mandates to improve efficiency/effectiveness  and provide better customer service to U.S. citizens.
CBP was formed from the former U.S. Customs Service and other entities with border protection responsibility.
OMB Circular A-11 establishes policy for planning, budgeting, acquisition, and management of federal capital assets.
section_title: 5. is reviewed and approved by the DHS Investment Review Board (IRB),1 the
paragraphs: Secretary of Homeland Security, and OMB; and  6. is reviewed by GAO.
On February 2, 2006, DHS submitted its fiscal year 2006 expenditure plan for  $316.8 million to the House and Senate Appropriations Subcommittees on  Homeland Security.
DHS currently plans to acquire and deploy ACE in 11 increments, referred to as  releases. The first three releases are fully deployed and operating, and the fourth  release is being deployed. Other releases are in various stages of definition and  development.
The purpose of the Investment Review Board is to integrate capital planning and investment control, budgeting, acquisition,  and management of investments. It is also to ensure that spending on investments directly supports and furthers the mission  and that this spending provides optimal benefits and capabilities to stakeholders and customers.
section_title: Objectives
section_title: As agreed, our objectives were to
paragraphs: determine whether the ACE fiscal year 2006 expenditure plan satisfies the   determine the status of our open recommendations on ACE, and  provide any other observations about the expenditure plan and DHS’s management of the ACE program.
We conducted our work at CBP headquarters and contractor facilities in the  Washington, D.C., metropolitan area, as well as at the port in Blaine, Washington, from July 2005 through March 2006, in accordance with generally accepted  government auditing standards. Details of our scope and methodology are provided  in attachment 1.
section_title: The legislative conditions that the Congress placed on the use of fiscal year 2006 ACE appropriated funds have been either partially or fully satisfied by the latest expenditure plan and related program documentation and activities. Nevertheless, more can be done to better address several aspects of these conditions, such as having an approved privacy impact assessment, measuring ACE performance and results, ensuring architectural alignment, and employing effective independent verification and validation (IV&V) practices. The following table summarizes the status of each of the legislative conditions.
subsections: section_title: 1. Meets the capital planning and investment control review requirements
paragraphs: established by OMB, including OMB Circular A-11, part 7.  2. Complies with DHS’s enterprise architecture.  3. Complies with the acquisition rules, requirements, guidelines, and  systems acquisition management practices of the federal government.  4. Includes a certification by the Chief Information Officer of DHS that an  independent verification and validation agent is currently under contract.  5. Is reviewed and approved by the DHS Investment Review Board,  Secretary of Homeland Security, and OMB.  6. Is reviewed by GAO.
section_title: Some recommendations have been addressed, while progress has been slow on others, such as
paragraphs: accurately reporting to the Appropriations Committees on CBP’s progress in  implementing our prior recommendations;   developing and implementing a strategic approach to meeting the program’s   using criteria for exiting key milestones that adequately consider indicators of  system maturity, such as severity of open defects and the associated risks;  and  developing and implementing a performance and accountability framework for  ensuring that promised capabilities and benefits are delivered on time and  within budget.
The following table summarizes the status of each of the open recommendations.
subsections: section_title: 1. Ensure that future expenditure plans are based on cost
paragraphs: estimates that are reconciled with independent cost  estimates.  2. Develop and implement a rigorous and analytically  verifiable cost estimating program.  3. Immediately develop and implement a human capital  management strategy that provides both near- and  long-term solutions; develop and implement missing  human capital practices.
Number of  months open  Status  4. Have future ACE expenditure plans specifically address any  proposals or plans for extending and using ACE infrastructure  to support other homeland security applications.  5. Define measures, and collect and use associated metrics, for  determining whether prior and future program management  improvements are successful.  6. Define and implement an ACE accountability framework that  a. coverage of all program commitment areas, including key  expected or estimated system (1) capabilities, use, and  quality; (2) benefits and mission value; (3) costs; and  (4) milestones and schedules.  b. currency, relevance, and completeness of all such  commitments made to the Congress in expenditure plans.
11 c.  reliable data relevant to measuring progress against  commitments.  d. reporting in future expenditure plans progress against  commitments contained in prior expenditure plans.  e. use of criteria for exiting key readiness milestones that  adequately consider indicators of system maturity,  such as severity of open defects.   f.  clear and unambiguous delineation of the respective  roles and responsibilities of the government and the  prime contractor.  7. Report quarterly to the House and Senate Appropriations  Committees on efforts to address open GAO  recommendations.
section_title: Objective 3: Observations
section_title: Steps have been taken to address past pattern of ACE release shortfalls, but new release management weaknesses are emerging.
paragraphs: Release 4 pilot revealed performance problems that caused the pilot period to  be extended and the pilot scope to be reduced.
Release 4 operational readiness review was passed despite unresolved  severe defects, and Release 4 is now being deployed.
Release 4 quality problems and enhancement needs have led to changes in  how ACE release requirements are defined.
Release 4 problems delayed Screening 1 and led to a revised strategy for  delivering all screening and targeting releases.
Screening 1 key milestones were passed despite unresolved severe defects.
section_title: Past pattern of cost, schedule, and performance shortfalls with Releases 2, 3,
paragraphs: and 4 makes new strategy of concurrently developing Releases 5, 6, and 7  risky.
Earned value management (EVM), a technique for measuring progress toward  meeting deliverables, is not being used to manage Screening 1 and Release  5.
ACE’s operational performance has been mixed, and mission impact is unclear.
Availability and responsiveness performance targets are largely being met.
Processing times for trucks crossing the border at key ports vary.
Long-standing help desk limitations are being addressed.
Usage by CBP and the trade is lower than expected.
User satisfaction was reported as low.
Performance targets are not always realistic.
Goals, expected mission benefits, and performance measures are not  adequately aligned.
section_title: To assist CBP in managing ACE and increasing the chances that it will deliver required capabilities on time and within budget and demonstrate promised mission benefits and results, we are recommending that DHS
paragraphs: minimize the degree of overlap and concurrency across ongoing and future  ACE releases, and capture and mitigate the associated risks of any residual  concurrency;  use EVM in the development of all existing and future releases;   develop the range of realistic ACE performance measures and targets needed  to support an outcome-based, results-oriented accountability framework,  including user satisfaction with ACE; and  explicitly align ACE program goals, benefits, desired business outcomes, and  performance measures.
section_title: In their oral comments on a draft of this briefing, DHS and CBP officials, including the Executive Director of Cargo Management Systems, CBP, generally agreed with our findings, conclusions, and recommendations and stated that the presentation was fair and balanced. They also provided clarifying information that we incorporated as appropriate in this briefing.
paragraphs: ACE is to support eight major CBP business areas. 1. Release1 Processing: Processing of cargo for import or export; tracking of  conveyances, cargo, and crew; and processing of in-bond, warehouse,  Foreign Trade Zone, and special import and export entries.  2. Entry2 Processing: Liquidation and closeout of entries and entry summaries  related to imports, and processing of protests and decisions. 3. Finance: Recording of revenue, performance of fund accounting, and  maintenance of the general ledger. 4. Account Relationships: Maintenance of trade accounts, their bonds and  CBP-issued licenses, and their activity. 5. Legal and Policy: Management of import and export legal, regulatory,  policies and procedures, and rulings issues.
A release is the act of CBP permitting imported merchandise to enter the United States.
An entry is the documentation required to be submitted to CBP in order for it to permit imported merchandise to enter the  United States.
section_title: 6. Enforcement: Enforcement of laws, regulations, policies and procedures,
paragraphs: Screening is the method of determining high-risk people or shipments before their arrival at a port.
Targeting is the risk-based determination of whether a shipment should undergo additional documentary review or  physical inspection.
section_title: The ACE technical architecture is to consist of layers or tiers of computer technology:
paragraphs: The Client Tier includes user workstations and external system interfaces.
The Presentation Tier provides the mechanisms for the user workstations  and external systems to access ACE.
The Integration Services Tier provides the middleware for integrating and  routing information between ACE software applications and legacy systems.
The Applications Tier includes the ACE software applications comprising  commercial products (e.g., SAP1) and custom-developed software that provide  the functionality supporting CBP business processes.
The Data Tier provides the data management and warehousing services for  ACE, including database backup, restore, recovery, and space management.
Security and data privacy are to be embedded in all five layers.
SAP is a commercial enterprise resource planning software product that has multiple modules, each performing separate  but integrated business functions. ACE will use SAP to support many of its business processes and functions. CBP’s  Modernization Office is also using SAP as part of a joint project with its Office of Finance to support financial management,  procurement, property management, cost accounting, and general ledger processes.
section_title: Simplified View of ACE Technical Architecture
section_title: CBP currently plans to acquire the 11 ACE releases in about 8.5 years. Screening 2 is to be acquired in two “drops,” or subreleases; Release 5 is to be acquired in two drops; Release 6 is to be acquired in three drops; and Release 7 is to be acquired in two drops.
paragraphs: The following presents the functionality provided by the 11 ACE releases, their  status, and associated plans.
Release 1 (ACE Foundation): Provide information technology (IT) infrastructure— computer hardware and system software—to support subsequent system releases.  This release was deployed in October 2003 and is operating.
Release 2 (Account Creation): Give initial group of CBP national account  managers1 and importers access to account information, such as trade activity.  This release was deployed in October 2003 and is operating.
Release 3 (Periodic Payment): Provide additional account managers and  importers, as well as brokers and carriers,2 access to account information; provide  initial financial transaction processing and CBP revenue collection capability,  allowing importers and their brokers to make monthly payments of duties and fees.  This release was deployed in July 2004 and is operating.
CBP national account managers work with the largest importers.
Brokers obtain licenses from CBP to conduct business on behalf of the importers by filling out paperwork and obtaining a  bond; carriers are individuals or organizations engaged in transporting goods for hire.
Background Summary of ACE Releases Release 4 (e-Manifest: Trucks): Provide electronic truck manifest1 processing and  interfacing to legacy enforcement systems and databases. As discussed later, this  release is operating at 39 truck border crossings as of March 8, 2006. Additional  enhancement releases for Release 4 have been deployed since May 2005.
Screening 1 (Screening Foundation): Establish the foundation for screening  cargo and conveyances by centralizing criteria and results into a single standard  database; allow users to define and maintain data sources and business rules. This  release is scheduled for deployment beginning in March 2006.
Screening 2 (Targeting Foundation): Establish the foundation for advanced  targeting capabilities by enabling CBP’s National Targeting Center to search  multiple databases for relevant facts and actionable intelligence. This release is  scheduled for deployment in two drops:  Screening 2 Targeting Platform (TP): Provide a platform to collect and  search relevant data and other information from multiple databases. This drop  is scheduled for deployment beginning in June 2006.
Manifests are lists of passengers or invoices of cargo for a vehicle, such as a truck, ship, or plane.
section_title: Screening 2 Targeting Foundation (TF): Build on the targeting platform to
paragraphs: add new data sources, enhance screening business rules, and provide  reporting capabilities. This drop is scheduled for deployment beginning in  October 2006; however, CBP deployed a prototype to the National Targeting  Center as part of an effort to gather detailed requirements.
Release 5 (Entry Summary, Accounts, and Revenue): Leverage SAP  technologies to enhance and expand accounts management, financial  management, and entry summary functionality. This release is being developed in  two drops:   Master Data and Enhanced Accounts (Drop A1): Use SAP to deliver  enhanced account creation and maintenance functionality and expand the  types of accounts managed in ACE. This drop is scheduled for deployment  beginning in May 2007.
Entry Summary and Revenue (Drop A2): Expand ACE to encompass entry  summary, interfaces with participating government agencies, calculation of  duties and fees, reconciliation processing, and refunds. This drop is scheduled  for deployment beginning in July 2008.
Background Summary of ACE Releases Screening 3 (Advanced Targeting Capabilities): Provide enhanced screening for  reconciliation, intermodal manifest, Food and Drug Administration data, and in- bond, warehouse, and Foreign Trade Zone authorized movements; integrate  additional data sources into targeting capability; and provide risk management  capability. This release is scheduled for deployment beginning in February 2007.
Screening 4 (Full Screening and Targeting): Provide full screening and targeting  functionality supporting all modes of transportation and all transactions within the  cargo management life cycle, including enhanced screening and targeting  capability with additional technologies. This release is scheduled for deployment  beginning in December 2008.
The multimodal manifest involves the processing and tracking of cargo as it transfers between different modes of  transportation, such as cargo that arrives by ship, is transferred to a truck, and then is loaded onto an airplane.
section_title: E-Manifest: Rail and Sea (Drop M1): Extend the electronic manifest
paragraphs: functionality to rail and sea shipments; convert rail, sea, and truck electronic  manifests into the multimodal manifest. Drop M1 is scheduled for deployment  beginning in July 2008.
E-Manifest: Air (Drop M2): Provide the electronic manifest capability to air  shipments, and bring all modes of transportation into the multimodal manifest.  Drop M2 is scheduled for deployment beginning in October 2007.
E-Manifest: Enhanced Tracking (Drop M3): Provide the capability to track  cargo, conveyances, individuals, and equipment, providing more timely and  accurate shipment status information. Drop M3 is scheduled for deployment  beginning in June 2009.
Background Summary of ACE Releases Release 7 (Exports and Cargo Control): Implement the remaining accounts  management, revenue, manifest, and release and export functionality. This release  is planned for development in two drops:   ESAR: Drawback, Protest, and IASS (Drop A3): Provide the import activity  summary statement (IASS),1 drawback functionality, and enhanced protest;  provide on-line processing for trade account applications. Drop A3 is  scheduled for deployment beginning in December 2009.
E-Manifest: Final Exports and Manifest (Drop M4): Extend the electronic  manifest for mail, pipeline, and hand carry; provide for electronic export  processing. Drop M4 is scheduled for deployment beginning in December  2009.
An import activity summary statement is a summary of an importer’s shipment activities over a specific period of time that is  transmitted electronically to CBP on a periodic basis by importers and brokers.
section_title: As of March 8, 2006, ACE, Releases 1 through 4 capabilities have been deployed to 39 of the 99 truck ports (see table).
paragraphs: ACE Satisfaction of Modernization Act Requirements  ACE is intended to support CBP satisfaction of the provisions of Title VI of the  North American Free Trade Agreement, commonly known as the Modernization  Act. Subtitle B of the Modernization Act contains the various automation provisions  that were intended to enable the government to modernize international trade  processes and permit CBP to adopt an informed compliance approach with  industry. The following table illustrates how each ACE release is to fulfill the  requirements of Subtitle B.
section_title: Status and Description of Completed Task Orders
subsections: section_title: No. Name
paragraphs: Initial program and project management; continued by  task 009.
Initial enterprise architecture and system engineering;  continued by task 010.
Initial requirements development and program planning  effort; continued by tasks for specific  increments/releases.
Design, development, testing, and deployment of  Releases 1 and 2 (initially intended to build Increment  1, which was subsequently divided into four releases)
March 2004  Development of Release 5 project plan, documentation  of ACE business processes, and development of an  ACE implementation strategy.
Design, development, and testing of Releases 3 and 4,  and deployment of Release 3.
Follow-on to task 001 to continue program and project  management activities.
Follow-on to task 002 to continue enterprise  architecture and system engineering activities;  continued by task 017.
March 2003  Acquisition and setup of the necessary infrastructure  and facilities for the contractor to design, develop, and  test releases.
Establishment of the infrastructure to operate and  maintain releases.
Conversion of scripts for interfacing desktop  applications (MS Word and Excel) and mainframe  computer applications.
March 2004  Development, demonstration, and delivery of a  prototype to provide CBP insight into whether  knowledge-based risk management should be used in  ACE.
section_title: Status and Description of Active Task Orders
subsections: section_title: No. Name
paragraphs: Enterprise process improvement integration.
International  Trade Data  System (ITDS)
Assistance for participating government agencies to  define requirements for an integrated ACE/ITDS  system.
Development and demonstration of technology  prototypes to provide CBP insight into whether the  technologies should be used in ACE.
Program management and support to organizational  change management through activities such as  impact assessments, end user training,  communication, and outreach.
Coordination of program activities and alignment of  enterprise objectives and technical plans through  architecture and engineering activities.
Application of the CBP Enterprise Life Cycle  Methodology to integrate multiple projects and other  ongoing Customs operations into CBPMO.
Follow-on to task 012 includes establishment,  integration, configuration, and maintenance of the  infrastructure to support Releases 2, 3, and 4.
Design, develop, test, and deploy the Screening and  Targeting (S&T) operational capability.
Project definition and initial design for Release 5; initial  project authorization and definition for Release 6.
Chronology of Seven ACE Expenditure Plans  Since March 2001, seven ACE expenditure plans have been submitted.1  Collectively, the seven plans have identified a total of $1,698.1 million in funding.
On March 26, 2001, CBP submitted to its appropriations committees the first  expenditure plan seeking $45 million for the modernization contract to sustain  Customs and Border Protection Modernization Office (CBPMO) operations,  including contractor support. The appropriations committees subsequently  approved the use of $45 million, bringing the total ACE funding to $50 million.
On February 1, 2002, the second expenditure plan sought $206.9 million to  sustain CBPMO operations; define, design, develop, and deploy Increment 1,  Release 1 (now Releases 1 and 2); and identify requirements for Increment 2  (now part of Releases 5, 6, and 7 and Screenings 1 and 2). The  appropriations committees subsequently approved the use of $188.6 million,  bringing total ACE funding to $238.6 million.
In March 2001, appropriations committees approved the use of $5 million in stopgap funding to fund program management  office operations.
Chronology of Seven ACE Expenditure Plans   On May 24, 2002, the third expenditure plan sought $190.2 million to define,  design, develop, and implement Increment 1, Release 2 (now Releases 3 and  4). The appropriations committees subsequently approved the use of $190.2  million, bringing the total ACE funding to $428.8 million.
On November 22, 2002, the fourth expenditure plan sought $314 million to  operate and maintain Increment 1 (now Releases 1, 2, 3, and 4); to design  and develop Increment 2, Release 1 (now part of Releases 5, 6, and 7 and  Screening 1); and to define requirements and plan Increment 3 (now part of  Releases 5, 6, and 7 and Screenings 2, 3, and 4). The appropriations  committees subsequently approved the use of $314 million, bringing total ACE  funding to $742.8 million.
Chronology of Seven ACE Expenditure Plans   On January 21, 2004, the fifth expenditure plan sought $318.7 million to  implement ACE infrastructure; to support, operate, and maintain ACE; and to  define and design Release 6 (now part of Releases 5, 6, and 7) and  Selectivity 2 (now Screening 2 and 3). The appropriations committees  subsequently approved the use of $316.8 million, bringing total ACE funding  to $1,059.6 million.
On November 8, 2004, the sixth expenditure plan sought $321.7 million for  design and development of Release 5 and Screening 2, definition of  Screening 3, ACE program management, architecture and engineering, and  operations and maintenance. The appropriations committees subsequently  approved the use of $321.7 million, bringing total ACE funding to $1,381.3  million.
Chronology of Seven ACE Expenditure Plans   On February 02, 2006, CBP submitted its seventh expenditure plan, seeking  $316.8 million for detailed design and development of Release 5,  development of Release 6, deployment of Screening 2, development and  deployment of Screening 3, program management and operations, and ACE  operations, maintenance, and infrastructure implementation.
section_title: Summary of the ACE Fiscal Year 2006 Expenditure Plan
subsections: section_title: Plan activity Entry Summary, Accounts, and Revenue (Release 5), Design and Development e-Manifest: All Modes and Cargo Security (Release 6), Design and Development Screening and Targeting, Development International Trade Data System Foundation Architecture and Engineering Operations and Maintenance Implementation Infrastructure and Support Communications, Training, and Outcomes ACE Foundation Program Management Program Office Operations Total
section_title: The plan does not include management reserve funding. As of December 15, 2005, the program had about $33.8 million in unused management reserve funding from prior years.
paragraphs: Background ACE Testing and Related Milestones Development of each ACE release includes system integration and system  acceptance testing, followed by a pilot period that includes user acceptance testing.  Generally, the purpose of these tests is to ensure that the system meets defined  system requirements or satisfies user needs. The associated readiness reviews are  to ensure that the system is ready to proceed to the next stage of testing or  operation.
Tests and their related milestones are described in the following table.
Verify that related system, subsystem, or  module components are capable of integrating  and interfacing with each other.
Verify that the developed system, subsystem,  or module operates in accordance with  requirements.
Verify that the functional scope of the release  meets the business functions for the users.
section_title: Defects identified during testing and operation of the system are classified into one of four severity categories, as described below.
section_title: Category
section_title: Critical (Severity 1)
paragraphs: Defect prevents or precludes the performance of an operational or mission- essential capability, jeopardizes safety or security, or causes the system,  application, process, or function to fail to respond or to end abnormally.
Severe  (Severity 2)
Defect prevents or precludes system from working as specified and/or  produces an error that degrades or impacts the system or user functionality.
Moderate  (Severity 3)
Defect prevents or precludes system from working as specified and/or  produces an error that degrades or impacts the system or user functionality.  An acceptable (reasonable and effective) work-around is in place that  rectifies the defect until a permanent fix can be made.
Minor  (Severity 4)
Defect is inconsequential, cosmetic, or inconvenient and does not prevent  users from using the system to accomplish their tasks.
section_title: Appendix II: Comments from the Department of Homeland Security
section_title: Appendix III: Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the person named above, Justin Booth, Barbara Collier,  William Cook, Neil Doherty, Michael Marshlick, Shannin O’Neill, Tomas  Ramirez, and Jennifer Vitalbo made key contributions to this report.