Information Technology: Improvements for Acquisition of Customs Trade Processing System Continue, but Further Efforts Needed to Avoid More Cost and Schedule Shortfalls
Released: 2007-10-25T00:00:00

Letter

Compliance with Legislative Conditions

The ACE expenditure plan—including related program documentation and program officials’ statements—satisfies or partially satisfies the six legislative conditions specified in the appropriations act. Specifically, the plan satisfies the conditions that it (1) comply with the acquisition rules, requirements, guidelines, and systems acquisition management practices of the federal government; (2) include a certification by the DHS Chief Information Officer that an independent verification and validation agent is currently under contract for the program; (3) be reviewed by the DHS Investment Review Board, the Secretary of DHS, and Office of Management and Budget (OMB); and (4) be reviewed by us. The plan and its supporting information partially satisfies the conditions that it (5) meet the capital planning and investment control review requirements established by OMB (including Circular A-11, part 7) and that it (6) comply with the DHS enterprise architecture. For example, although DHS determined that ACE was aligned with the DHS enterprise architecture, the agency’s analysis did not address key aspects of an architectural alignment, such as alignment with the architecture’s data reference model.

Status of GAO Recommendations

DHS has implemented some, but not all, of the recommendations pertaining to ACE that we have made since 2003. These recommendations, along with their status, are summarized here.

Eight recommendations have been implemented. For example, DHS has developed a range of ACE performance measures and targets needed to support an outcome-based, results-oriented accountability framework, including user satisfaction with ACE; aligned ACE program goals, benefits, desired business outcomes, and addressed those legislative conditions associated with measuring ACE performance and results and employing effective independent verification and validation practices; and ensured that future expenditure plans are based on cost estimates that are reconciled with independent cost estimates.

The remaining seven recommendations are in the process of being implemented, as described here.

Recommendation: Define and implement an ACE accountability framework that fulfills several conditions, to include ensuring the currency, relevance, and completeness of commitments made to Congress in expenditure plans and reporting in future expenditure plans the progress against commitments that were contained in prior expenditure plans.

In progress. The program has established an accountability framework that is providing input for both the annual expenditure plan and quarterly congressional reports. However, as with prior expenditure plans, the fiscal year 2007 expenditure plan did not reflect the most current program information. For example, information on milestones, earned value management, and risks were about 4 months old when the plan was submitted to the appropriations committees in February 2007, and program commitments were no longer current. Further, while program officials continue to use the quarterly ACE status reports to provide the appropriations committees with more detailed information, these reports are generally submitted to Congress 3 to 4 months after the end of each quarter, thus limiting their currency and relevance as well. CBP and DHS officials told us that the delays were due to the DHS review and approval process and that they are exploring ways to accelerate the process.

The 2007 expenditure plan did not adequately report on progress against previous plan commitments. For example, the plan did not (1) report progress against milestones in the fiscal year 2006 plan or explain why these milestones were not achieved, (2) report actual obligation or expenditure of funds relative to the planned uses of these funds in prior expenditure plans, or (3) address progress against the milestone dates for each stage of a release that was included in the prior year’s plan. According to program officials, the quarterly congressional reports provide more current information on the program’s progress against prior commitments. However, these reports have also not fully addressed the commitments made in prior expenditure plans. Program officials stated that they intend to start providing this information in the last quarterly report of each year.

Recommendation: Define measures and collect and use associated metrics for determining whether prior and future program management improvements are successful.

Planned. The program office has made changes that are to improve overall program management and, according to its December 2006 quarterly report to Congress, the program office plans to measure the impact of future management improvements. Moreover, this report stated that the program anticipates more changes, including creation of a cargo requirement management board to decide the disposition of all change requests to production systems; establishment of a new invoice review policy; and colocation of personnel within a given business area. However, program officials told us that they have yet to define measures to determine the impact of such changes and thus are not yet positioned to determine their success.

Recommendation: Minimize the degree of overlap and concurrency across ongoing and future ACE releases and capture and mitigate the associated risks of any residual concurrence.

In progress. Since May 2006, the program office has reduced overlap and concurrence of ACE releases and has taken actions to reduce potential contention for limited resources by, for instance, decoupling (i.e., reducing dependencies among) certain program components; dividing releases into smaller subreleases to provide more flexibility in scheduling; improving planning for development, integration, testing, training activities, and milestones to better schedule use of development and test environments; and centralizing management of shared software services. Further, the program office conducts regular integration meetings with the teams supporting each release to discuss concerns, decisions, and schedules associated with resource availability and is using a software tool to track and mitigate release-specific concurrency risks. However, this tool contains vague or incomplete data relative to mitigating these risks. These data limitations make it difficult to determine the status or the effectiveness of the efforts to reduce the risks associated with overlap and concurrence among releases.

Recommendation: Direct the appropriate departmental officials to fully address those legislative conditions associated with having an approved privacy impact assessment (PIA) and ensuring architectural alignment.

In progress. One legislative condition states that the plan should meet OMB’s capital planning and investment control review requirements, which include addressing security and privacy issues. The program office has developed a PIA for ACE release 4 (e-Manifest: Trucks), which is currently operational. DHS approved this PIA on July 14, 2006. This PIA addressed the major elements of DHS’s guidance, and program officials stated that they would update the assessment for each ACE release. However, this PIA did not cover other recently completed screening releases. Program officials told us that these screening releases were considered to be part of the Automated Targeting System and were covered by the Automated Targeting System’s PIA. However, this PIA does not specifically identify or address ACE screening releases.

With respect to architectural alignment, DHS determined in May 2007 that all required ACE products and technologies were aligned with the DHS technical reference model and that ACE was thus aligned with the DHS enterprise architecture. However, we have yet to receive sufficient documentation describing the criteria and methodology used to make these determinations or verifiable analysis supporting the determinations. Moreover, the determinations were based on technical alignment and did not address other relevant aspects of program alignment to an enterprise architecture, such as data alignment.

Recommendation: Develop and implement key human capital management practices.

In progress. In June 2006, the Office of Information Technology Strategic Human Capital Management Plan, which included an ACE-specific appendix as a road map for effective management of human capital resources, was approved. However, the plan does not address the basic tenets of effective human capital management, such as defining the positions needed (including core competencies) to perform core program functions, assessing and inventorying current workforce skills and abilities, assessing any gaps between needed and existing workforce levels and capabilities, and filling identified gaps. Officials acknowledged these limitations in the plans and stated that they are developing an implementation plan to address these shortfalls.

Recommendation: Include in the June 30, 2006, quarterly update report to the appropriations committees a strategy for managing ACE human capital needs and the ACE framework for managing performance and ensuring accountability.

In progress. The June 30, 2006, quarterly report to the House and Senate Appropriations Committees included the ACE program’s strategy for meeting its human capital needs and its accountability framework; however, the human capital strategy does not meet the basic tenets of strategic human capital management, as previously explained.

Recommendation: Accurately report to the appropriations committees on progress in implementing our prior recommendations.

In progress. Quarterly reports to the House and Senate Appropriations Committees have contained information on the status of our open recommendations since November 2002, but recent reports remain outdated due to a 2- to 7-month time lapse between when the reports are produced and when they are provided to the appropriations committees. DHS and program officials stated that they are exploring ways to accelerate the review process and thereby improve the timeliness and accuracy of their reports to Congress.

Observations on the Expenditure Plan and Management of ACE

We have three observations about ACE requirements, commercial product selection, and risk management.

Requirements: Redefinition of requirements for several ACE releases is now under way to address limitations in completeness of originally defined requirements, and this redefinition is likely to introduce program schedule delays and cost increases.

In defining ACE requirements, the program office discovered that its original approach did not adequately engage all key stakeholders, such as software programmers and subject matter experts, for the legacy system ACE is intended to replace. To address this, key stakeholders are now collaborating and decomposing legacy code. However, this effort is expected to significantly delay some system releases and drops. Program officials have taken several actions that they say will minimize the impact of the delays, such as prioritizing shared functionality, dividing releases/drops into smaller increments, and changing release deployment strategies. However, these changes have yet to be approved, and the full extent of the cost and schedule implications is not yet known. Moreover, neither the fiscal year 2007 expenditure plan nor the ACE quarterly reports have disclosed the requirements redefinition and its impact, and neither has addressed any changes to release deployment strategies.

Commercial product selection: Significant changes to ACE requirements have led to reevaluation and replacement of a key commercial off-the-shelf (COTS) product previously selected and being prepared for use.

The program office conducted several analyses to determine which COTS products would best meet ACE system requirements, including a general review of various packages in 2002 to select a provider and a more detailed analysis in 2004 to define and allocate ACE requirements and select a specific product—the SAP Enterprise Portal. In December 2006, however, the ACE Chief System Architect determined through additional analysis that all planned SAP functionality could be provided by Internet Transaction Server (ITS) technology and recommended that ITS be adopted. The program office subsequently stopped work on SAP Enterprise Portal design and configuration efforts and reported that ITS would be used for release 5/drop A2 instead of the SAP Enterprise Portal. This decision is expected to have some near-term schedule impacts because much of the completed work for A2 had been based on the planned use of SAP Enterprise Portal. Further, use of ITS raises the risk of inadequate user response time, which would, in turn, negatively impact user productivity and introduce a high probability of significant cost and schedule impacts. Program officials reported that actions are under way to mitigate the risk through performance modeling and test planning. However, neither the fiscal year 2007 expenditure plan nor the quarterly reports to Congress disclose this COTS product change, its impact on release schedules and cost estimates, or the risk to future system performance.

Risk management: All program risks are not being effectively managed.

The program office has developed a process guide and implemented an automated tool (database) for managing ACE risks in accordance with relevant guidance and best practices. Although the database contains fields to provide a description, level (high, medium, or low), and mitigation strategy (including start and end dates, exit criteria, and implementation status) for each risk, the completeness and quality of this information varies. Because of such database limitations, we could not determine the status of and mitigation progress on 17 risks. Moreover, these database limitations were not reflected in the documentation used at key program events, indicating that the program does not have the risk-related information that it needs to inform its program decisions and to reduce the chances of potential problems becoming actual problems. Program officials stated that they are taking steps to improve risk management, including establishing a group to ensure the quality and completeness of the database, holding regular group meetings with contract staff and team leads to discuss risks and their impacts, and conducting risk management training. To date, however, program risks have not been communicated to oversight organizations through the 2007 expenditure plan or recent quarterly reports to the House and Senate Appropriations Committees.

Conclusions

Over the past 7 years, CBP and DHS have worked to fulfill legislatively mandated annual expenditure plan requirements and to implement dozens of our recommendations related to these plans and management of the program. Among other things, these requirements and recommendations have promoted effective program management and accountability for performance and results. As a result of these years of effort, the ACE program is better positioned today for delivering promised capabilities and benefits than it has been in the past.

Nevertheless, key program management practices relating to, for example, human capital management, requirements management, and risk management continue to remain a challenge, and other management areas, such as information security and architecture alignment, continue to require attention. As a result, avoiding major program schedule delays and cost overruns remains a challenge as more of each appears to be on the horizon. To further improve ACE management and minimize its exposure to risk, it is important for CBP and DHS to remain vigilant in their efforts to satisfy ACE legislative requirements and to fully implement our prior recommendations. Moreover, it is important that they keep Congress fully informed on where the program stands and what changes are planned to address emerging cost overruns and schedule delays.

Recommendations for Executive Action

To further strengthen ACE management and promote accountability for ACE performance and results, we are recommending that the Secretary of Homeland Security direct the CBP Commissioner to ensure that future quarterly reports to the House and Senate Appropriations Committees disclose the risks and associated mitigation strategies of not having fully satisfied the expenditure plan legislative conditions and not having completed implementation of all our prior recommendations; the status and impacts on the program’s estimated cost and schedule and lessons learned from ongoing efforts to redefine requirements and to implement a different COTS product than originally selected; and the program’s plans and actions for improving ACE risk management and its current inventory of program risks, including their associated mitigation strategies and the status of the strategies’ implementation.

Agency Comments and Our Evaluation

In written comments on a draft of this report signed by the Director, Departmental GAO/OIG Liaison, and reprinted in appendix II, DHS agreed with our findings and stated that it is committed to addressing them. Further, the department agreed with our recommendations and described actions that it said are under way or planned to address them. While most of the department's stated actions are consistent with our recommendations, in one case they may not be sufficient. Specifically, the department stated that it would ensure that future expenditure plans meet all OMB capital planning and investment control review requirements by attaching the required OMB budget submission for ACE, referred to as an Exhibit 300, to all future plans. However, we reviewed the fiscal year 2007 ACE Exhibit 300 as part of our review of the fiscal year 2007 ACE expenditure plan and found that it did not meet the OMB requirements cited above. Therefore, unless DHS improves the quality of future ACE Exhibit 300s by addressing the weaknesses we have identified, this action alone may not fully address our recommendation.

We are sending copies of this report to the Chairmen and Ranking Minority Members of the other Senate and House committees and subcommittees that have authorization and oversight responsibilities for homeland security. We are also sending copies to the DHS Secretary, the CBP Commissioner and, on their request, to other interested parties. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

Should you or your offices have any questions on matters discussed in this report, please contact me at (202) 512-3459 or at hiter@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Other contacts and key contributors to this report are listed in appendix III.

Briefing to Subcommittees on Homeland Security, House and Senate Committees on Appropriations

Furthermore, program officials told us that although the prime contractor was contractually required to have an SEI CMM® Level 34 capability, they are no longer required to do so because funding contractor program management-related activities is now a lower priority relative to other competing demands for ACE funding.

GAO-06-580; GAO-05-267. As with our prior reviews of ACE expenditure plans, the evaluation does not include compliance with federal acquisition regulations or other federal rules and requirements beyond those encompassed by SEI’s Capability Maturity Models. The SA-CMM® is consistent with the acquisition guidelines and systems acquisition management practices of the federal government, and provides a management framework that defines acquisition practices for such process areas as acquisition planning, solicitation, requirements development and management, project management, contract tracking and oversight, and evaluation. Level 3 capability is more advanced than level 2 and indicates that acquisition management processes have been defined throughout the organization.

The program office refers to these plans, processes, and procedures as assets. required activities for program management processes and disseminating them to ACE staff.

Objective 1: Legislative Conditions Condition 4. The plan satisfies the condition that it include certification by the DHS CIO that an IV&V agent is currently under contract.

On October 24, 2006, the DHS Deputy CIO certified in writing that an IV&V agent is under contract for ACE and that the agent met applicable requirements and standards. (See open recommendations section of this briefing for more information on IV&V.)

Objective 1: Legislative Conditions Condition 5. The plan, including related program documentation and program officials’ statements, satisfies the requirement that it be reviewed and approved by the DHS IRB, the Secretary of Homeland Security, and OMB.

The DHS IRB reviewed the program and approved the expenditure plan on December 12, 2006.

The DHS Under Secretary for Management approved the expenditure plan on behalf of the Secretary of Homeland Security on February 6, 2007.

OMB approved the expenditure plan on January 22, 2007.

Objective 1: Legislative Conditions Condition 6. The plan satisfies the requirement that it be reviewed by GAO.

Our review was completed on July 26, 2007.

Open Recommendation 1: Define and implement an ACE accountability framework that fulfills these conditions: a. Covers all program commitment areas, including key expected or estimated system (1) capabilities, use, and quality; (2) benefits and mission value; (3) costs; and (4) milestones and schedules.

Effective program management includes defining and measuring progress against program commitments and being held accountable for results. Such commitments generally cover expected or estimated (1) capabilities and their associated use and quality, (2) benefits and mission value, (3) costs, and (4) milestones and schedules.

Since 2003, we have reported that such commitments for ACE have not always been defined, although improvements have been made. Most recently, we reported1 that the program office had developed an initial version of an accountability framework for measuring several program commitments, such as capabilities and milestones, but that other commitments, such as benefits, had not been as well defined.

Objective 2: Open Recommendations Accountability Framework During the last year, the program office has continued to improve its coverage of program commitments, as evidenced by the content of key program documents— ACE accountability framework, ACE Program Plan (August 2006), the CBP/ACE performance reference model, periodic Program Management Review (PMR) reports, and the fiscal year 2007 expenditure plan. For example, The accountability framework now captures and integrates data on ACE functional and performance capabilities, benefits, and mission value, estimated and actual costs, milestones and schedule, accomplishments, and earned value management status. Further, the framework provides visibility into each ACE release at several levels of detail, and it is being used by the ACE Executive Director and others as the means for monitoring program progress, issues, and decisions.

The PMR reports have included the accountability framework data.

The fiscal year 2007 ACE expenditure plan included an example of the accountability framework commitments from the September 2006 PMR report.

Objective 2: Open Recommendations b. Ensures currency, relevance, and completeness of such commitments made to Congress in expenditure plans.

We have previously reported that commitments made in expenditure plans relative to program capabilities, benefits, costs, and schedules need to be current, relevant, and complete. To the extent that they are not, the currency and relevance of the plan and its utility to Congress as an accountability mechanism are limited.

ACE expenditure plans have not always included such information. To address this limitation, we reported last year1 that CBP was relying on quarterly reports to Congress to provide the appropriations committees with more current, relevant, and complete information about the program than could be provided in the expenditure plan. However, we also noted that the quarterly reports were generally submitted to Congress 3 to 4 months after the end of each quarter, thus limiting their currency and relevance.

Objective 2: Open Recommendations The fiscal year 2007 expenditure plan continues to include information about program commitments that is not current. Specifically, The latest expenditure plan was provided to the appropriations committees on February 6, 2007. However, information from the ACE accountability framework that was included in the plan, such as milestones, EVM values, and risks, were as of October 2006, making it about 4 months old. For example, the expenditure plan listed the milestone for the Production Readiness Review (PRR) for a key release, Release 5/Drop A1, as March 1, 2007. However, this milestone had already slipped by more than 4 months--to July 12, 2007--by the December 2006 PMR (held on January 5, 2007).

Objective 2: Open Recommendations According to program officials, they continue to use the quarterly ACE status reports to provide the appropriations committees with more current information.1 However, recent quarterly reports have not been current. For example, The December 2006 quarterly report was not provided until February 26, 2007, and it contained the same outdated PRR milestone as the expenditure plan did for the release previously mentioned.

Other quarterly reports have been submitted to the appropriations committees as many as 7 months after the end of the quarter.

CBP and DHS officials told us that the delays were due to the DHS review and approval process and that incorporating the most current program information would delay the reports even longer. According to the ACE and DHS officials, they are exploring ways to accelerate the review process.

Quarterly reports should be received by Congress approximately 60 days after the end of the quarter.

Objective 2: Open Recommendations c. Ensures reliable data relevant to measuring progress against commitments.

The quality of the capabilities that a program is to deliver is a relevant program commitment. One measure of the quality of system capabilities is the trend in the number and severity of unresolved system defects or problems. Reliable data about these defects are needed so that system maturity can be understood and informed investment decisions can be made.

We previously reported1 that ACE defect data were not always consistent because the two tools that the program used to track defects were not integrated and reconciled. As a result, the true status of ACE defects, and thus an important measure of system quality, was not known.

Since then, the program office has integrated the two tools using a cross- referencing function and has instituted manual processes for reconciling the data in each tool. The December 2006 quarterly congressional report stated that these efforts have improved the ability to record, assess, and report on system quality and performance.

Objective 2: Open Recommendations d. Ensures reporting in future expenditure plans progress against commitments contained in prior expenditure plans.

The value and utility of an expenditure plan for congressional oversight and agency accountability depends in large part on whether the plan reports on progress against commitments--system capabilities, benefits, costs, and schedules--made in prior plans.

Historically, the ACE expenditure plans have not reported adequately on progress against previous plan commitments. Most recently, we reported1 that the fiscal year 2006 expenditure plan contained several such reporting gaps, such as whether funding amounts were actually obligated and expended as planned and whether releases met schedules. In particular, we reported that the plan did not address whether the design and development for Release 5 was actually accomplished as planned.

Objective 2: Open Recommendations The fiscal year 2007 expenditure plan still does not adequately describe the program’s progress against the commitments that were made in the fiscal year 2006 plan. For example, The plan does not report progress against milestones in the fiscal year 2006 plan or explain why these milestones were not achieved.

The plan does not report actual obligation or expenditure of funds relative to the planned uses of these funds in prior expenditure plans.

The plan did not address progress against the milestone dates for each stage of a release that was included in the prior year’s plan. As a case in point, the fiscal year 2006 expenditure plan described specific functionality planned for Release 5/Drop A1 (Master Data and Enhanced Accounts), such as online registration for trade representatives. However, the fiscal year 2007 expenditure plan does not include information on whether these functions were delivered, stating only that Release 5 functionality would be deployed in two phases.

Objective 2: Open Recommendations According to program officials, the quarterly congressional reports provide more current information on the program’s progress against prior commitments. However, we found that these reports have also not fully addressed the commitments made in prior expenditure plans. For example, recent quarterly reports describe progress against developmental milestones for each release, but do not report whether key functionality associated with each release was delivered. In particular, the report for the first quarter of fiscal year 2007 identified planned future capabilities and associated milestones for Release 5/Drop A1 but did not address the status of the key functions associated with this release.

Program officials stated that they intend to start providing this information in the last quarterly report of each year.

Objective 2: Open Recommendations e. Ensures use of criteria for exiting key readiness milestones that adequately consider indicators of system maturity, such as severity of open defects, and document the milestone decisions in a way that reflects the associated risks and plans for mitigating them.1 As noted earlier, one measure of the quality of system capabilities being delivered is the number and severity of unresolved system defects or problems. As such, information on such defects is an important consideration when program decisions, such as key milestone decisions, are made.

We previously reported2 that several key milestones were passed that had severe open defects and that program officials were unable to provide documentation regarding how the risks associated with these defects were assessed. We also reported that these risks were not being tracked in the program’s risk database.

Objective 2: Open Recommendations Since then, the system life cycle gate review process for Office of Information and Technology projects, including ACE, has been amended to include milestone readiness decisions based on an assessment and acceptance of risks (including the risks related to unresolved defects). Further, the ACE Risk and Issue Management Process guidance provides for the systematic identification, analysis, prioritization, planning, execution, evaluation, and documentation of program risks and issues and requires that any risks associated with going forward should be identified as part of each life cycle gate review.

This process has been applied in the following two major gate reviews: Release A1 (ESAR) Critical Design Review (May 2006) and Release M1 (E-Manifest: Rail and Sea Manifest) Critical Design Review (August 2006).

Objective 2: Open Recommendations In the case of Release 5/Drop A1, program officials reported that no severe defects existed, but that all known risks associated with proceeding past the milestone had been entered into the program’s risk database and documented in the certification package submitted to the DHS CIO.

The quarterly reports to Congress similarly state that risks related to gate review decisions and the associated impacts are entered into a database to ensure visibility and mitigation and are included in the package submitted to the DHS CIO for review and certification to pass a given milestone.

Going forward, officials stated that similar assessments will be part of the following planned gate reviews for Release 5/Drop A1: Operational Readiness Review (planned for August 23, 2007) and Live operations (planned for August 25, 2007).

Objective 2: Open Recommendations Open recommendation 2: Develop the range of realistic ACE performance measures and targets needed to support an outcome-based, results-oriented accountability framework, including user satisfaction with ACE.

We have previously reported on both the absence of meaningful performance measures to understand ACE progress, quality, and results, and have raised concerns about the practicality and applicability of some measures that have been defined. Most recently, we reported1 that defined ACE performance targets are not always realistic and that goals, expected mission benefits, and performance measures are not fully defined and adequately aligned.

In response, the program office has established an initial set of performance measures that are tied to program objectives and related performance goals, and it has established processes for the collecting, analyzing, and integrating performance data for each measure. Among other things, these performance measures address user satisfaction, efficiency, and productivity. Moreover, these measures have been made an integral part of the ACE accountability framework and they were approved by the CBP Commissioner and the DHS CIO on June 30 and July 6, 2006, respectively.

Objective 2: Open Recommendations According to program officials, these initial measures will continue to be revised and augmented, and thus will evolve over time based on lessons learned, changes to existing release capabilities, and refinements of requirements for future releases. For example, the program office reported in March 2007 that new measures are still being developed, reviewed, and evaluated for Releases 2, 3, and 4 at the same time that performance data is being collected and reported for existing measures for these releases. According to program officials, the new measures will be combined with existing performance measures.

In addition, CBP’s December 2006 quarterly congressional report stated that performance measures for future releases, such as Release 5, are being scheduled for identification and development and that changes are being made to existing measures when they are determined to be inappropriate. For example, the program office learned that the performance measure “percentage of truck manifests being filed electronically” did not recognize that empty trucks are not required to file manifests, and thus it revised the metric used to divide the number of electronically filed truck manifests by the total number of required truck manifests instead of the total number of trucks in order to get a more meaningful reflection of ACE performance.

To assist in managing these performance measures, the program has established a life cycle process for performance measures as well as a database tool to record and manage modifications.

Objective 2: Open Recommendations Performance Measures Alignment Open recommendation 3: Explicitly align ACE program goals, benefits, desired business outcomes, and performance measures.

Use advanced passenger and cargo information (NTC, ATS-Air, ATS, Screening and Targeting- ACE) to pre-screen, target, and identify potential terrorists and terrorist shipments and any related activity.

Increased use of targeting.

Further, this model links the measures to specific ACE releases. For example, the above performance measure applies to Screening Foundation (S1) and Targeting Foundation (S2).

Objective 2: Open Recommendations Management Improvement Measures Open recommendation 4: Define measures and collect and use associated metrics for determining whether prior and future program management improvements are successful.

As we have previously reported,1 investments in program management improvements should include defined measures of progress and results. To date, the program office has implemented a number of such improvements; however, it has not had measures or metrics to determine the success of the improvements. Moreover, the December 2006 quarterly congressional report stated that the program anticipates more changes, including creation of a cargo requirement management board to decide the disposition of all change requests to production systems; establishment of a new invoice review policy; and co-location of personnel within a given business area.

Objective 2: Open Recommendations Management Improvement Measures According to its December 2006 quarterly reports to Congress, the program office plans to measure the impact of future management improvements. However, program officials told us that they have yet to define them and thus are not yet using such measures.

Objective 2: Open Recommendations Legislative Conditions Open recommendation 5: Fully address those legislative conditions associated with measuring ACE performance and results and employing effective IV&V practices.

Among the legislative conditions that ACE expenditure plans have been required to meet are satisfaction of OMB guidance, including that associated with measuring program performance and results and use of effective IV&V practices. These conditions reflect good program management practices and, if implemented properly, can reduce program risks.

As previously discussed in this briefing, the program office has developed a range of ACE performance measures and taken steps to align them with program, CBP, and DHS strategic goals and outcomes. According to program officials, they plan to continue to evolve and refine the performance measures and include the measures in the ACE accountability framework.

Objective 2: Open Recommendations In January 2006, the DHS CIO certified that an IV&V agent was under contract for the ACE program, but noted several issues, including these: mechanisms were needed to ensure that products were complete, of sufficient quality, and met the needs of the user and a more explicit technical approach, describing when and how certain activities should (or should not be) performed, was needed.

Moreover, we subsequently reported1 that the scope of the contractor’s activities did not extend to both IV&V of key system products and development processes.

Objective 2: Open Recommendations On October 24, 2006, the Deputy CIO again certified that an IV&V agent was under contract and stated that the previous issues had been addressed. Since then, the program office has also developed an IV&V Implementation Management Plan that addresses the concerns raised by the DHS CIO and us. In particular, the plan requires an IV&V program consistent with the industry standard; provides a set of objectives, guidelines, and expectations for IV&V activities, including periodic independent reports on status, observations, recommendations, and activities; and addresses satisfaction of quality standards for ACE products and user needs.

Program officials told us that IV&V has allowed early identification and correction of program process and product weakness.

Objective 2: Open Recommendations Reconciliation of Cost Estimates Open recommendation 6: Ensure that future expenditure plans are based on cost estimates that are reconciled with independent cost estimates.

It is important that expenditure plans be based on reliable estimates of costs, to include reconciling differences between government and independent cost estimates. We recently reported1 that the cost estimate in the fiscal year 2006 expenditure plan was based on government and independent cost estimates that had been compared and found to be consistent.

For the fiscal year 2007 expenditure plan, our analysis showed that the government and independent cost estimates differed by about 15 percent. According to program officials, they reconciled these differences in January 2007, and concluded that the results did not warrant changes to the expenditure plan because the government estimates used in the expenditure plan were more accurate.

Objective 2: Open Recommendations Reconciliation of Cost Estimates According to program officials, two primary factors account for the difference in estimates. Specifically, The estimates assumed different timelines for completing development of all releases. The independent estimate assumed development completion by fiscal year 2010, while the government estimate assumed fiscal year 2011. According to program office officials, this accounts for about five percent of the difference.

The government estimate included a number of items that the independent estimator did not. For example, the independent estimator did not include training and outreach items, such as the cost of conferences with trade associations, training materials, and associated travel. As a result, the government estimated training and outreach costs over the life of the program to be about $90 million, while the independent estimate put these costs at about $39 million. The independent estimate has since been amended to include the missing items, and the reconciliation process is continuing.

Objective 2: Open Recommendations Rigorous Cost Estimation Open recommendation 7: Develop and implement a rigorous and analytically verifiable cost estimating program that embodies the tenets of effective estimating.1 The reliability of cost estimates is largely a function of the quality of the estimating process used to derive them. We previously reported2 that the program did not have a well-defined cost estimating process, but that it has since made progress in strengthening its cost estimating program. Specifically, the program office has: defined and documented processes for estimating program costs (including management reserve costs) and hired a contractor to develop costs estimates that were independent of the government estimates.

In September 2006, an ACE support contractor reported that both the government and independent cost estimation processes demonstrated significant conformance to effective estimating practices and concluded that the program is using a rigorous and verifiable cost estimating approach.

See, for example, models developed by the Carnegie Mellon University SEI, Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of Software Organizations, CMU/SEI-95-SR-005 (Pittsburgh, Pa.: Carnegie Mellon University, 1995) and A Manager's Checklist for Validating Software Cost and Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburgh, Pa.: Carnegie Mellon University,1995).

Open recommendation 8: Use EVM in developing all existing and future releases.

EVM is a program management tool to measure progress by comparing, during a given period of time, the value of work accomplished with the amount of work expected to be accomplished. This comparison permits performance to be evaluated based on calculated variances from the planned (baselined) cost and schedule. EVM is both an industry accepted practice and an OMB requirement.

We recently reported1 that the program office was not using EVM for all releases (e.g., Release 5) due to changes to release baselines and the lack of familiarity on the part of program staff with EVM practices.

Since then, the program office has established the performance baselines needed to implement EVM for Release 5/Drops A1 and A2, Screening 3, and Release 6/Drop M1.

Objective 2: Open Recommendations Earned Value Management For these releases, the program office reports that it has also applied EVM standards in establishing baselines and included EVM data in the accountability framework for management and decision-making purposes. Program officials also reported that they plan to implement EVM on future releases and task orders.

Objective 2: Open Recommendations Open recommendation 9: Have future ACE expenditure plans specifically address any proposals or plans, whether tentative or approved, for extending and using ACE infrastructure to support other homeland security applications, including any impact on ACE of such proposals and plans.

Together, the fiscal year 2007 expenditure plan and the quarterly reports to Congress address ACE’s relationships with other trade processing and DHS applications. For example, the plan discusses efforts to work with participating government agencies in defining and deploying the International Trade Data System (ITDS). The stated goal is to deliver ACE/ITDS in an integrated and coordinated manner. In this regard, the plan discusses workshops to gather requirements from participating agencies for Release 6/Drop M2 (first held on July 19, 2006), a working group to address these agencies’ HAZMAT issues (established on July 17, 2006), and efforts to develop data element inputs for ACE from other government agencies.

Objective 2: Open Recommendations In addition, the quarterly reports to Congress cite coordination efforts with related homeland security programs. For example, the report for the first quarter of 2007 states that Container Security Initiative will be supported by ACE Release 6 and Customs-Trade Partnership Against Terrorism is coordinating with ACE Release 5/Drop A1 capabilities to provide both CBP and trade representatives with the ability to view the status of CBP programs; and U.S.-Mexico Border Partnership Plan will coordinate with ACE to implement any cargo screening standards derived from partnership plan agreements.

In addition, CBP reports that ACE and other CBP applications1 share infrastructure (desktops, clients, and local area networks) at the ports of entry. This infrastructure is purchased and maintained by CBP’s OIT Program Integration Division. Each application, including ACE, is responsible for complying with OIT’s infrastructure standards.

CBP applications include IT systems that provide tools and information to help front-line officers ensure the security of our nation. This includes applications for the United States Visitor and Immigrant Status Indicator Technology (US-VISIT), Container Security Initiative and Automated Targeting System.

Objective 2: Open Recommendations Overlap and Concurrence Open recommendation 10: Minimize the degree of overlap and concurrency across ongoing and future ACE releases and capture and mitigate the associated risks of any residual concurrence.

Significant overlap and concurrency among major program activities, such as releases, introduce considerable cost and schedule risks as they can create contention for limited resources among the releases. We have continued to report on extensive overlap and concurrence in ACE releases and the cost overruns and schedule delays that have resulted. Most recently, we reported1 that the ACE schedule continued to provide for such overlap and concurrency and that the risks associated with doing so were not being effectively addressed.

Since then, the program office has reduced this overlap and concurrence. For example, it has decoupled (i.e., reduced dependencies among) certain ACE program components by separating Screening 1-3 from Releases 4-6 and aligned the development and delivery of functionality for different releases with the availability of required hardware environments.

Recent quarterly congressional reports stated that CBP has taken actions to reduce potential contention for limited resources. Examples include dividing releases into smaller subreleases, called drops, to provide more improving planning for development, integration, testing, and training activities and milestones to better schedule use of development and test environments; and centralizing management of shared software services to address, among other things, allocation of resources and responsiveness to workload peaks and use of consistent technical management approaches across releases.

Moreover, the program’s risk database identifies overlap and concurrency-related risks. However, the mitigation strategies for these risks contain vague or incomplete data. These data limitations make it difficult to determine the status or the effectiveness of the efforts to reduce the risks associated with overlap and concurrence among releases. (See the observations section of this briefing for additional information on program risk management.)

Objective 2: Open Recommendations The program office is also conducting regular integration meetings with the teams supporting each release to discuss concerns, decisions, and schedules associated with resource availability and is using a software tool to track and mitigate release- specific concurrency risks.

Notwithstanding these steps, the program continues to face challenges in managing dependencies among releases, which any associated concurrency in the program’s development exacerbates. For example, the schedule slips expected with Release 5/Drop A2 will affect Release 6/Drop M1, as resources have been shifted from M1 to address the A2 delay. In addition, further delays in A1 and A2 will impact the implementation of M1.

Objective 2: Open Recommendations Open recommendation 11: Direct the appropriate departmental officials to fully address those legislative conditions associated with having an approved privacy impact assessment (PIA)1 and ensuring architectural alignment.

The department approved the ACE PIA for Release 4 in July 2006, however, this assessment does not cover other completed releases. Further, DHS has determined that ACE is aligned with DHS architecture, but we have yet to receive documentation to adequately understand and verify the determination.

PIA (In progress)

OMB, Guidance for Implementing the Privacy Provisions of the EGovernment Act of 2002, OMB M-03-22 (Sept. 26, 2003).

Objective 2: Open Recommendations The program office has developed a PIA for Release 4 of the ACE e-Manifest: Trucks and the International Trade Data System, and DHS approved it on July 14, 2006. As noted earlier, our analysis shows that this PIA addresses the major elements of DHS’s guidance. Program officials stated that they would update the assessment for each ACE release. This assessment does not cover other recently completed screening releases (i.e., S1 and S2). However, program officials told us that S1 and S2 are considered to be part of the Automated Targeting System (ATS), and are therefore covered by the ATS PIA. However, our analysis of the ATS PIA showed that while it addresses screening and targeting functions, it does not specifically identify or address releases S1 and S2.

Architectural Alignment (in progress)

As discussed in the legislative conditions section of this briefing, DHS has determined that ACE is aligned with the DHS EA. For example, In December 2006, DHS’s EA Center of Excellence determined that ACE was conditionally compliant with the DHS architecture, but that the program needed to take actions to address four conditions.

Objective 2: Open Recommendations Privacy Assessment and Architectural Alignment On March 2007, the DHS EA Board reported that ACE had satisfied three of the conditions and recommended approval of ACE’s request for program alignment with one remaining condition—alignment with the DHS Technical Reference Model.

In May 2007, DHS EA officials reported that all required products and technologies were aligned with the Technical Reference Model and that ACE was thus aligned with the DHS EA.

However, as was also discussed earlier in this briefing, we have yet to receive sufficient documentation describing the criteria and methodology used to make these determinations or verifiable analysis supporting the determinations. Moreover, the determinations were based on technical alignment and did not address other relevant aspects of program alignment to an EA, such as data alignment. Thus, the program is at risk of being designed and implemented in a way that is not consistent with all relevant aspects of the DHS EA, such as data structures and standards, and thus does not support optimized DHS-wide operations, performance, and results.

Open recommendation 12: Develop and implement key human capital management practices.

As we have previously reported,1 effective strategic management of program human capital includes, among other things, defining the skill sets needed to perform program functions, assessing and inventorying the skill sets of the program’s current workforce and assessing any associated skill gaps in meeting future needs, and developing strategies to fill any identified gaps. Moreover, it includes having a well-defined plan that provides for effective implementation of these processes.

In June 2006, CBP executives approved the OIT Strategic Human Capital Management Plan. The plan is intended to be a road map for supporting CBP’s workforce vision and ensuring effective management of human capital resources across OIT to address enterprise-wide priorities. An ACE-specific plan was included as an appendix to the CBP plan. This appendix was intended to provide better near- and long-term human capital management practices for the OIT offices involved in ACE development.

Objective 2: Open Recommendations Human Capital Practices Neither the OIT nor the ACE-specific plan addresses the basic tenets of effective human capital management. For example, neither provides for: defining the positions needed (including core competencies) to perform core assessing and inventorying current workforce skills and abilities; assessing any gaps between needed and existing workforce levels and filling identified gaps via such means as hiring new staff, training existing staff, and augmenting staff with contractor support.

OIT officials acknowledged these limitations in the plans and stated that they are developing an implementation plan to address these shortfalls. The officials stated that the implementation plan will include accountability, timeframes, and metrics to carry out the larger OIT Strategic Human Capital Management Plan.

According to these officials, this implementation plan was to be completed in January 2007, following presentation to the OIT Deputy Director's Council and approval by the OIT Office Directors and Assistant Commissioner. However, as of July 2007, it had not yet been approved and thus was not available for our review.

In the interim, OIT and program officials reported taking various steps to address ACE human capital needs. These include, for example, Using various methods to fill staff vacancies. As of May 2007, the program reports that it has 62 full-time employees and one vacancy and that additional staff are working on ACE either full- or part-time.

Lobbying OPM for more flexibility in recruiting, to include higher salaries and direct hiring authority.

Reorganizing OIT into six program offices aligned to major mission areas so that the number of government personnel responsible for ACE development activities can be augmented by IT functional program management expertise.

Offering training to improve the skills of staff.

Nevertheless, these steps have not been guided by a well-defined plan and thus represent activities that are not tied to strategic goals and outcomes and thus cannot be measured against them. Without a plan, it is unlikely that the program will be able to adequately ensure that it has the right people at the right time to deliver ACE successfully.

Objective 2: Open Recommendations Open recommendation 13: Include in the June 30, 2006, quarterly update report to the appropriations committees a strategy for managing ACE human capital needs and the ACE framework for managing performance and ensuring accountability.

The June 30, 2006 quarterly report to the House and Senate Appropriations Committees included the program’s strategy for meeting its human capital needs and its accountability framework.

Human Capital Strategy (In progress)

The June 30, 2006, quarterly report to Congress included a description of the previously described ACE-specific appendix to the OIT Strategic Human Capital Management Plan, including the plan’s five goals and strategies. Moreover, the report states that this ACE-specific plan is aligned with OPM’s Human Capital Assessment and Accountability Framework.1 As previously stated, however, this ACE-specific plan does not meet the basic tenets of strategic human capital management, as defined in this framework and other relevant guidance.

As revised by OPM in 2005, this framework reflects guidance from the collaboration of OMB, OPM, and GAO.

Objective 2: Open Recommendations According to program officials, they are developing a new OIT human capital strategy and implementation plan, but they have yet to provide us with a date when it will be completed.

Accountability Framework (Complete)

The June 30, 2006, quarterly report included a description of the ACE accountability framework. As previously discussed, this framework is the means by which the program measures performance relative to promised ACE capabilities, costs, schedules, earned value, risks, and mission values and benefits. The report also included an appendix illustrating the format and content of the accountability framework tool. At that time, CBP reported that it would continue to work with stakeholders to further enhance the format, readability, and utility of the framework as a program management and reporting tool.

Since June 2006, the program office has refined and implemented the accountability framework. The framework covers all program commitment areas and key system aspects to support executive decision making and provides external program stakeholders with information on the program.

Objective 2: Open Recommendations Quarterly Report on Open Recommendations Open recommendation 14: Report quarterly to the House and Senate Appropriations Committees on efforts to address open GAO recommendations.

CBP has submitted quarterly reports on ACE to both the House and the Senate Appropriations Subcommittees since November 2002, including reports for each quarter of fiscal year 2006 and the first and second quarters of fiscal year 2007. These reports have addressed CBP’s efforts to address open GAO recommendations.

Objective 2: Open Recommendations Accurately Report on Open Recommendations Open recommendation 15: Accurately report to the appropriations committees on CBP's progress in implementing our prior recommendations.

As previously stated, CBP has included information on progress in meeting GAO’s recommendations in its quarterly reports1 to the appropriations committees since November 2002. However, some of this information is dated and thus inaccurate due to the time lapse between when the reports are produced and when they are provided to the appropriations committees. Recently, this time lapse has been between about two and seven months. According to program officials, they are exploring ways to accelerate the review process and thereby improve the timeliness and accuracy of their reports to Congress. DHS officials also told us that they have ongoing efforts to improve the review process; however, the process has not improved consistently or significantly.

The reports are due to Congress approximately 60 days after the completion of a fiscal quarter. ACE quarterly reports are due to DHS approximately 30 days after the end of the quarter, and then DHS has 30 days to review the report and submit it to Congress.

Objective 3: Observations Requirements Limitations Likely to Cause Major Delays Observation 1: Redefinition of requirements for several ACE releases is now under way to address limitations in completeness of originally defined requirements, and this redefinition is likely to introduce program schedule delays and cost increases.

A key aspect of successful system acquisition programs is having well-defined requirements. Among other things, requirements should be complete, and to accomplish this, best practices advocate engaging all key stakeholders in the requirements definition and management process.

In defining ACE requirements, the program office discovered that its original requirements definition approach did not adequately engage all key stakeholders, and to its credit, has since taken steps to address this. Specifically, In the spring of 2004, the program office and the ACE support team conducted more than 300 business process workshops with the ACE user community to help define ACE requirements for the future releases. This requirements definition approach was referred to as the Global Business Blueprint (GBB) effort.

Objective 3: Observations Requirements Limitations Likely to Cause Delays In June and July of 2005, a key group of ACE stakeholders that were not originally involved in the GBB (application programmers familiar with the legacy system that ACE is to replace) raised questions about the completeness of the requirements. To address these questions, the program office examined the GBB’s completeness by first reverse-engineering the legacy software for a small number of legacy system programs and then comparing the reverse-engineered requirements to the GBB-derived requirements. Based on this, which is referred to as legacy code decomposition, the program office found that the GBB-derived requirements were missing about 20 percent of needed ACE functionality.

In August 2005, the program office decided that it needed to decompose all of the legacy system code in order to completely capture the requirements for all ACE releases. Work began with decomposition of the legacy code related to ACE Release 5/Drop A2 (Entry Summary Accounts and Revenue).

Objective 3: Observations Requirements Limitations Likely to Cause Delays In September 2006, the program office determined that the legacy code decomposition approach alone was not sufficient to gain a full understanding of the requirements given the size and complexity of A2. As a result, the requirements definition process was expanded to engage another key stakeholder group (business process subject matter experts). Under the expanded approach, referred to as legacy code decomposition and collaboration, legacy system software programmers and subject matter experts were to examine the code line by line in defining ACE requirements.

In November 2006, the legacy system code decomposition and collaboration effort for Release 5/Drop A2 fell behind schedule because of lack of personnel with legacy system expertise and experience. The schedule for A2 was tentatively revised, but at the time of our review, the schedule was still under review, had not yet been approved, and thus was not yet available for our review.

Objective 3: Observations Requirements Limitations Likely to Cause Delays The program office has identified the A2 legacy code decomposition and collaboration process as a high risk item that will significantly impact the A2 schedule. For example, the decomposition and collaboration process for part of the functionality on A2’s critical path—the Authorized Broker Interface Entry Summary—is not expected to be completed before early 2008, at the earliest.

According to program officials, delays for other releases/drops, such as Release 5/Drop A1 and Release 6/Drop M1, will not be as significant. They also said that, while they have not yet estimated how long A2 will be delayed and what the associated cost implications are, they do not expect the cost increases to breach the current acquisition program baseline of $3.3 billion, which translates into a cost increase of less than $200 million. Moreover, they said that several actions have been taken to minimize the impact of the delays. For example, A2 functionality necessary for M1 has been given priority in order to support M1 deployment as originally planned and A2 scope is being divided into increments to allow some functionality to be delivered sooner and to minimize the impact on other drops.

Objective 3: Observations Requirements Limitations Likely to Cause Delays However, these actions carry consequences, such as missed opportunities to combine field training and an increase in the number of legacy interfaces, thus increasing the potential for introducing software problems.

In addition, program officials told us that they are considering changes to the A2 and M1 deployment strategies to address stakeholder concerns, and they said that these changes could also minimize the magnitude of the A2 and M1 delays. Specifically, they said that they had planned to deploy on a national basis, meaning that A2 and M1 functionality would be deployed to all ports at the same time and concurrently adopted by all users nationwide. However, deployment may change to a filer-by-filer basis, meaning that A2 and M1 functionality would be deployed to all ports at the same time, but not all filers would begin using it at the same time.

Objective 3: Observations Requirements Limitations Likely to Cause Delays According to program officials, they believe that the change in the deployment strategy would both address stakeholder concerns and minimize any A2 and M1 schedule delays caused by the redefinition of requirements. However, they added that these changes have yet to be approved, and the full extent of the cost and schedule implications are not yet known. Moreover, neither the fiscal year 2007 expenditure plan nor the ACE quarterly reports have disclosed the A2 and M1 requirements redefinition and its impact, and neither has addressed any changes to their deployment strategies.

Objective 3: Observations Key COTS Product Being Replaced Observation 2: Significant changes to ACE requirements have, in turn, led to reevaluation and replacement of a key commercial off-the-shelf product (COTS) previously selected and being prepared for use.

When acquiring commercial component-based systems, like ACE, best practices advocate basing decisions on whether to employ a given COTS product on thorough, rigorous, and continuous analysis of a number of factors, including how well competing products satisfy defined system requirements.

To the program office’s credit, it reports having followed the CBP system life cycle methodology to determine which COTS product would best meet the requirements for Release 5/Drop A2. These analyses include: In 2002, the program office reviewed, in general terms, various COTS packages and determined that a solution using SAP (formerly Systems Application and Products), a COTS provider, combined with other commercial solutions and customized development, provided the best combination of capability, performance, and cost for ACE.

Objective 3: Observations Key COTS Product Being Replaced In 2004, a more detailed analysis was conducted as part of the previously mentioned GBB process, which was intended to define and allocate ACE requirements to future ACE releases and provide the basis for, among other things, selecting a specific SAP product. At that time, the SAP Enterprise Portal product was selected for Release 5/Drop A2.

In December 2006, the ACE Chief System Architect recommended that the Internet Transaction Server (ITS) technology already used by CBP should be adopted instead of the SAP Enterprise Portal, based on the determination that all currently planned SAP functionality could be presented using the ITS technology. This decision was based on improved understanding of the requirements and previous analyses of ITS.

Objective 3: Observations Key COTS Product Being Replaced On the basis of these analyses and the Chief System Architect’s recommendation, the program office subsequently stopped work on SAP Enterprise Portal design and configuration efforts and, in March 2007, the program reported that the SAP Internet Transaction Server would be used for Release 5/Drop A2 instead of the SAP Enterprise Portal. While this decision was expected to have some near-term schedule impact because much of the completed work for A2 had been based on the planned use of SAP Enterprise Portal, the program office reports that these impacts are offset by the cost advantages of other ACE releases already using the Internet Transaction Server technology.

However, in March 2007, program officials identified a risk of inadequate response time for the Internet Transaction Server—thus negatively impacting user productivity—and that there was high probability of significant cost and schedule impacts. Actions are underway to mitigate the risk through performance modeling and test planning.

Neither the fiscal year 2007 expenditure plan nor the quarterly reports to Congress disclose this COTS product change, its impact on release schedules and cost estimates, or risk to future system performance.

Observation 3: All program risks are not being effectively managed.

Risk management is a continuous, forward-looking process that is intended to either prevent program cost, schedule, and performance problems from occurring or to minimize the impact if they occur. According to relevant guidance and best practices, effective risk management involves proactively identifying, assessing, and disclosing risks; defining and implementing cost-effective strategies for mitigating these risks; and measuring and disclosing progress in doing so.

To its credit, the program office has developed a process guide and implemented an automated tool (database) for managing ACE risks in accordance with relevant guidance and best practices. Among other things, the database contains the description, level (high, medium, or low), and mitigation strategy (including start and end dates, exit criteria, and implementation status) for each risk.

Objective 3: Observations Risks Not Being Effectively Managed However, the completeness and quality of the information on each of the risks in the risk database1 vary. For example, many risks were missing information on the status of efforts to implement the many risks (18) were missing criteria for completing mitigation steps, clear descriptions of what the risk entailed, and start and end dates for planned mitigation activities.

Because of such database limitations, we could not determine the status of and mitigation progress on 17 risks. Moreover, these database limitations were reflected in the documentation used at key program events, such as PMRs. This means that the program does not have the risk-related information that it needs to inform its program decisions and to reduce the chances of potential problems becoming actual problems.

As of May 23, 2007 the risk database contained 46 risks.

Objective 3: Observations Risks Not Being Effectively Managed Program officials, including the official responsible for risk management, stated that risk management is immature and needs to be strengthened. The reasons they gave for these risk management weaknesses are due to all staff not being trained on how to use the tool (last training was provided in 2003, and the since then a number of people have joined the program); the tool is unique to the ACE program and thus no CBP guidance exists on its use); and each ACE group addresses risk differently in its weekly meetings.

To improve ACE risk management, program officials told us that they are: establishing a group to ensure the quality and completeness of the database; holding regular group meetings with contract staff and team leads to discuss risks and their impacts; and conducting risk management training.

Objective 3: Observations Risks Not Being Effectively Managed If implemented effectively, such steps should result in more meaningful information about program risks that can be useful to DHS in managing the program and to Congress in overseeing it. To date, however, ACE program risks have not been communicated to oversight organizations through the 2007 expenditure plan or recent quarterly reports to the House and Senate Appropriations Committees.

Nevertheless, key program management practices relating to, for example, human capital management, requirements management, and risk management continue to remain a challenge, and other management areas, such as information security and architecture alignment, continue to require attention. As a result, avoiding major program schedule delays and cost overruns remains a challenge as more of each appear to be on the horizon.

To further improve ACE management and minimize its exposure to risk, it is important for CBP and DHS to remain vigilant in their efforts to satisfy ACE legislative requirements and to fully implement our prior recommendations. Moreover, it is important that they keep the Congress fully informed on where the program stands and what changes are planned to address emerging cost overruns and schedule delays.

To further strengthen ACE management and promote accountability for ACE performance and results, we are making the following recommendation to the Secretary of Homeland Security to direct the CBP Commissioner to ensure that future quarterly reports to the House and Senate Appropriations Committees disclose

(1)the risks and associated mitigation strategies of not having fully satisfied the expenditure plan legislative conditions and not having completed implementation of all our prior recommendations; (2)the status and impacts on the program’s estimated cost and schedule and lessons learned from ongoing efforts to redefine requirements and to implement a different COTS product than originally selected; and (3)the program’s plans and actions for improving ACE risk management and its current inventory of program risks, including their associated mitigation strategies and the status of the strategies’ implementation.

In oral comments on a draft of this briefing, DHS and CBP officials agreed with our conclusions and recommendations, and provided clarifying information and technical comments that we incorporated in the briefing, as appropriate.

Scope and Methodology

To accomplish our objectives, we analyzed the ACE fiscal year 2007 expenditure plan and supporting documentation, and compared them to relevant federal requirements and guidance, applicable best practices, and our prior recommendations. We also interviewed DHS and CBP officials and ACE program contractors. In particular, we reviewed

SEI’s institutional and project-specific estimating guidelines are defined in Robert E. Park, Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of Software Organizations, CMU/SEI-95-SR-005 (Pittsburgh, Pa.: Carnegie Mellon University Software Engineering Institute, 1995) and A Manager’s Checklist for Validating Software Cost and Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburg, Pa: 1995), respectively.

Attachment 1 Scope and Methodology independent verification and validation (IV&V) activities using the Institute of Electrical and Electronics Engineers standard for software verification and validation;2 CBP actions to coordinate ACE with related programs; CBP’s reorganization documentation, including the organizational charts and roles and responsibilities matrix; ACE’s accountability framework; and cost and schedule data and program commitments from program management documentation.

Attachment 1 Scope and Methodology For DHS-, CBP-, and contractor-provided data that we did not substantiate, we have made appropriate attribution indicating the data's source.

We conducted our work at CBP headquarters and contractor facilities in the Washington, D.C., metropolitan area from December 2006 through July 2007 in accordance with generally accepted government auditing standards.

Information Technology: Customs Has Made Progress on Automated Commercial Environment System, but IT Faces Long-Standing Management Challenges and New Risks. GAO-06-580. Washington, D.C.: May 31, 2006.

Information Technology: Customs Automated Commercial Environment Program Progressing, but Need for Management Improvements Continues. GAO-05-267. Washington, D.C.: March 14, 2005.

Information Technology: Early Releases of Customs Trade System Operating, but Pattern of Cost and Schedule Problems Needs to Be Addressed. GAO-04- 719. Washington, D.C.: May 14, 2004.

Customs Service Modernization: Automated Commercial Environment Progressing, but Further Acquisition Management Improvements Needed. GAO- 03-406. Washington, D.C.: February 28, 2003.

Customs Service Modernization: Third Expenditure Plan Meets Legislative Conditions, but Cost Estimating Improvements Needed. GAO-02-908. Washington, D.C.: August 9, 2002.

Customs Service Modernization: Management Improvements Needed on High- Risk Automated Commercial Environment Project. GAO-02-545. Washington, D.C.: May 13, 2002.

Customs Service Modernization: Results of Review of First Automated Commercial Environment Expenditure Plan. GAO-01-696. Washington, D.C.: June 5, 2001.

Information Security: Homeland Security Needs to Enhance Effectiveness of Its Program. GAO-07-1003T. Washington, D.C.: June 20, 2007.

Intellectual Property: Better Data Analysis and Integration Could Help U.S. Customs and Border Protection Improve Border Enforcement Efforts. GAO-07- 735. Washington, D.C.: April 26, 2007.

Cargo Container Inspections: Preliminary Observations on the Status of Efforts to Improve the Automated Targeting System. GAO-06-591T. Washington, D.C.: March 30, 2006.

Comments from the Department of Homeland Security

Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the person named above, Daniel Castro, Dawn Day, Neil Doherty, Nancy Glover, Paula Moore, Jamelyn Payan, Nik Rapelje, and Karen Talley made key contributions to this report.