Information Technology: Improvements for Acquisition of Customs Trade Processing System Continue, but Further Efforts Needed to Avoid More Cost and Schedule Shortfalls
Published: 2007-10-25T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Homeland Security (DHS) established the Automated Commercial Environment (ACE) program to replace and supplement existing cargo processing technology. According to the fiscal year 2007 DHS appropriations act, DHS is to develop and submit an expenditure plan for ACE that satisfies certain conditions, including being reviewed by GAO. GAO reviewed the plan to (1) determine whether the expenditure plan satisfies the legislative conditions, (2) determine the status of 15 open GAO recommendations, and (3) provide observations about the expenditure plan and DHS's management of the program. To address the mandate, GAO assessed plans and related documentation against federal guidelines and industry standards and interviewed the appropriate DHS officials.
section_title: What GAO Found
paragraphs: The ACE expenditure plan satisfies many--but not all--of the legislative conditions specified in the fiscal year 2007 DHS appropriations act. Specifically, the plan (with related program documentation and officials' statements) complies with acquisition rules, requirements, guidelines, and management practices of the federal government; includes a DHS certification that an independent verification and validation agent is under contract; was reviewed and approved by DHS and the Office of Management and Budget (OMB); and was reviewed by GAO. In addition, it partially satisfies conditions for meeting the capital planning and investment control review requirements established by OMB in Circular A-11 (part 7), including information security, and for complying with the DHS enterprise architecture. DHS has implemented eight open GAO recommendations made during the past 4 years, including those related to performance measures and targets, independent verification and validation, cost estimation, and program reporting. Seven other recommendations made during this time are in the process of being implemented. With respect to these seven, DHS has taken steps to satisfy each, such as establishing an accountability framework, reducing overlap and concurrence among ACE releases, and completing a privacy impact assessment, and actions are under way or planned to more fully address them. GAO is making three new observations about the expenditure plan and the management of ACE. First, the program is taking needed steps to redefine requirements for several ACE releases because of limitations in the completeness of original requirements, but this redefinition is likely to introduce significant program schedule delays and cost increases. Second, the changes to ACE requirements have led to replacement of a key commercial product, but the new product carries the risk of negatively impacting user productivity. Third, the automated database used for managing ACE risks is incomplete and does not contain information needed to adequately inform program decisions. All told, DHS has continued to make progress on ACE, and the program is better positioned today for delivering promised capabilities and benefits than it has been in the past. Nevertheless, key program management practices relating to, for example, human capital management, requirements management, and risk management remain a challenge, and other management areas, such as information security and architecture alignment, continue to require attention. As a result, GAO sees major program schedule delays and cost overruns on the horizon. To improve ACE management and minimize exposure to risk, it is important for DHS to remain vigilant in its efforts to satisfy ACE legislative requirements, fully implement prior GAO recommendations, and keep Congress fully informed about the program's status, plans, and risk.

section_title: Letter
paragraphs: The U.S. Customs and Border Protection (CBP), as part of the Department  of Homeland Security (DHS), submitted to Congress in February 2007 its  fiscal year 2007 expenditure plan for the Automated Commercial  Environment (ACE) program pursuant to the Department of Homeland  Security Appropriations Act, 2007.
Begun in 2001, ACE is to replace and supplement existing cargo processing  technology. The system is being developed and deployed in a series of  increments to about 300 ports of entry and is expected to be fully deployed  by the end of 2011. The goals of ACE include (1) supporting border security  by enhancing analysis and information sharing with other government  agencies and providing CBP with the means to decide before a shipment  reaches the border what should be targeted because it is a security threat  and what should be expedited because it complies with U.S. laws and (2)  streamlining time-consuming and labor-intensive tasks for CBP personnel  and the trade community through a national trade account and a single  Web-based interface.
As required by the appropriations act, we reviewed ACE’s fiscal year 2007  expenditure plan. Our objectives were to (1) determine whether the  expenditure plan satisfies the legislative conditions, (2) determine the  status of 15 open GAO recommendations for ACE, and (3) provide  observations about the expenditure plan and DHS’s management of the  program.
On July 26, 2007, we provided a briefing to the staffs of the Subcommittees  on Homeland Security, Senate and House Committees on Appropriations  on the results of our review. This report transmits those results. The full  briefing, including our scope and methodology, is reprinted in appendix I.
subsections: section_title: Compliance with Legislative Conditions
paragraphs: The ACE expenditure plan—including related program documentation and  program officials’ statements—satisfies or partially satisfies the six  legislative conditions specified in the appropriations act. Specifically, the  plan satisfies the conditions that it (1) comply with the acquisition rules,  requirements, guidelines, and systems acquisition management practices of  the federal government; (2) include a certification by the DHS Chief  Information Officer that an independent verification and validation agent is  currently under contract for the program; (3) be reviewed by the DHS  Investment Review Board, the Secretary of DHS, and Office of Management  and Budget (OMB); and (4) be reviewed by us. The plan and its supporting  information partially satisfies the conditions that it (5) meet the capital  planning and investment control review requirements established by OMB  (including Circular A-11, part 7) and that it (6) comply with the DHS  enterprise architecture. For example, although DHS determined that ACE  was aligned with the DHS enterprise architecture, the agency’s analysis did  not address key aspects of an architectural alignment, such as alignment  with the architecture’s data reference model.
section_title: Status of GAO Recommendations
paragraphs: DHS has implemented some, but not all, of the recommendations  pertaining to ACE that we have made since 2003. These recommendations,  along with their status, are summarized here.
Eight recommendations have been implemented. For example, DHS has  developed a range of ACE performance measures and targets needed to  support an outcome-based, results-oriented accountability framework,  including user satisfaction with ACE;  aligned ACE program goals, benefits, desired business outcomes, and   addressed those legislative conditions associated with measuring ACE  performance and results and employing effective independent  verification and validation practices; and  ensured that future expenditure plans are based on cost estimates that  are reconciled with independent cost estimates.
The remaining seven recommendations are in the process of being  implemented, as described here.
Recommendation: Define and implement an ACE accountability  framework that fulfills several conditions, to include ensuring the  currency, relevance, and completeness of commitments made to  Congress in expenditure plans and reporting in future expenditure plans  the progress against commitments that were contained in prior  expenditure plans.
In progress. The program has established an accountability framework  that is providing input for both the annual expenditure plan and quarterly  congressional reports. However, as with prior expenditure plans, the fiscal  year 2007 expenditure plan did not reflect the most current program  information. For example, information on milestones, earned value  management, and risks were about 4 months old when the plan was  submitted to the appropriations committees in February 2007, and program  commitments were no longer current. Further, while program officials  continue to use the quarterly ACE status reports to provide the  appropriations committees with more detailed information, these reports  are generally submitted to Congress 3 to 4 months after the end of each  quarter, thus limiting their currency and relevance as well. CBP and DHS  officials told us that the delays were due to the DHS review and approval  process and that they are exploring ways to accelerate the process.
The 2007 expenditure plan did not adequately report on progress against  previous plan commitments. For example, the plan did not (1) report  progress against milestones in the fiscal year 2006 plan or explain why  these milestones were not achieved, (2) report actual obligation or  expenditure of funds relative to the planned uses of these funds in prior  expenditure plans, or (3) address progress against the milestone dates for  each stage of a release that was included in the prior year’s plan. According  to program officials, the quarterly congressional reports provide more  current information on the program’s progress against prior commitments.  However, these reports have also not fully addressed the commitments  made in prior expenditure plans. Program officials stated that they intend  to start providing this information in the last quarterly report of each year.
Recommendation: Define measures and collect and use associated  metrics for determining whether prior and future program management  improvements are successful.
Planned. The program office has made changes that are to improve overall  program management and, according to its December 2006 quarterly report  to Congress, the program office plans to measure the impact of future  management improvements. Moreover, this report stated that the program  anticipates more changes, including creation of a cargo requirement  management board to decide the disposition of all change requests to  production systems; establishment of a new invoice review policy; and  colocation of personnel within a given business area. However, program  officials told us that they have yet to define measures to determine the  impact of such changes and thus are not yet positioned to determine their  success.
Recommendation: Minimize the degree of overlap and concurrency  across ongoing and future ACE releases and capture and mitigate the  associated risks of any residual concurrence.
In progress. Since May 2006, the program office has reduced overlap and  concurrence of ACE releases and has taken actions to reduce potential  contention for limited resources by, for instance, decoupling (i.e., reducing  dependencies among) certain program components; dividing releases into  smaller subreleases to provide more flexibility in scheduling; improving  planning for development, integration, testing, training activities, and  milestones to better schedule use of development and test environments;  and centralizing management of shared software services. Further, the  program office conducts regular integration meetings with the teams  supporting each release to discuss concerns, decisions, and schedules  associated with resource availability and is using a software tool to track  and mitigate release-specific concurrency risks. However, this tool  contains vague or incomplete data relative to mitigating these risks. These  data limitations make it difficult to determine the status or the  effectiveness of the efforts to reduce the risks associated with overlap and  concurrence among releases.
Recommendation: Direct the appropriate departmental officials to fully  address those legislative conditions associated with having an approved  privacy impact assessment (PIA) and ensuring architectural alignment.
In progress. One legislative condition states that the plan should meet  OMB’s capital planning and investment control review requirements, which  include addressing security and privacy issues. The program office has  developed a PIA for ACE release 4 (e-Manifest: Trucks), which is currently  operational. DHS approved this PIA on July 14, 2006. This PIA addressed  the major elements of DHS’s guidance, and program officials stated that  they would update the assessment for each ACE release. However, this PIA  did not cover other recently completed screening releases. Program  officials told us that these screening releases were considered to be part of  the Automated Targeting System and were covered by the Automated  Targeting System’s PIA. However, this PIA does not specifically identify or  address ACE screening releases.
With respect to architectural alignment, DHS determined in May 2007 that  all required ACE products and technologies were aligned with the DHS  technical reference model and that ACE was thus aligned with the DHS  enterprise architecture. However, we have yet to receive sufficient  documentation describing the criteria and methodology used to make these  determinations or verifiable analysis supporting the determinations.  Moreover, the determinations were based on technical alignment and did  not address other relevant aspects of program alignment to an enterprise  architecture, such as data alignment.
Recommendation: Develop and implement key human capital  management practices.
In progress. In June 2006, the Office of Information Technology Strategic  Human Capital Management Plan, which included an ACE-specific  appendix as a road map for effective management of human capital  resources, was approved. However, the plan does not address the basic  tenets of effective human capital management, such as defining the  positions needed (including core competencies) to perform core program  functions, assessing and inventorying current workforce skills and abilities,  assessing any gaps between needed and existing workforce levels and  capabilities, and filling identified gaps. Officials acknowledged these  limitations in the plans and stated that they are developing an  implementation plan to address these shortfalls.
Recommendation: Include in the June 30, 2006, quarterly update report  to the appropriations committees a strategy for managing ACE human  capital needs and the ACE framework for managing performance and  ensuring accountability.
In progress. The June 30, 2006, quarterly report to the House and Senate  Appropriations Committees included the ACE program’s strategy for  meeting its human capital needs and its accountability framework;  however, the human capital strategy does not meet the basic tenets of  strategic human capital management, as previously explained.
Recommendation: Accurately report to the appropriations committees  on progress in implementing our prior recommendations.
In progress. Quarterly reports to the House and Senate Appropriations  Committees have contained information on the status of our open  recommendations since November 2002, but recent reports remain  outdated due to a 2- to 7-month time lapse between when the reports are  produced and when they are provided to the appropriations committees.  DHS and program officials stated that they are exploring ways to accelerate  the review process and thereby improve the timeliness and accuracy of  their reports to Congress.
section_title: Observations on the Expenditure Plan and Management of ACE
paragraphs: We have three observations about ACE requirements, commercial product  selection, and risk management.
Requirements: Redefinition of requirements for several ACE releases is  now under way to address limitations in completeness of originally  defined requirements, and this redefinition is likely to introduce  program schedule delays and cost increases.
In defining ACE requirements, the program office discovered that its  original approach did not adequately engage all key stakeholders, such as  software programmers and subject matter experts, for the legacy system  ACE is intended to replace. To address this, key stakeholders are now  collaborating and decomposing legacy code. However, this effort is  expected to significantly delay some system releases and drops. Program  officials have taken several actions that they say will minimize the impact  of the delays, such as prioritizing shared functionality, dividing  releases/drops into smaller increments, and changing release deployment  strategies. However, these changes have yet to be approved, and the full  extent of the cost and schedule implications is not yet known. Moreover,  neither the fiscal year 2007 expenditure plan nor the ACE quarterly reports  have disclosed the requirements redefinition and its impact, and neither  has addressed any changes to release deployment strategies.
Commercial product selection: Significant changes to ACE  requirements have led to reevaluation and replacement of a key  commercial off-the-shelf (COTS) product previously selected and being  prepared for use.
The program office conducted several analyses to determine which COTS  products would best meet ACE system requirements, including a general  review of various packages in 2002 to select a provider and a more detailed  analysis in 2004 to define and allocate ACE requirements and select a  specific product—the SAP Enterprise Portal. In December 2006, however,  the ACE Chief System Architect determined through additional analysis  that all planned SAP functionality could be provided by Internet  Transaction Server (ITS) technology and recommended that ITS be  adopted. The program office subsequently stopped work on SAP  Enterprise Portal design and configuration efforts and reported that ITS  would be used for release 5/drop A2 instead of the SAP Enterprise Portal.  This decision is expected to have some near-term schedule impacts  because much of the completed work for A2 had been based on the  planned use of SAP Enterprise Portal. Further, use of ITS raises the risk of  inadequate user response time, which would, in turn, negatively impact  user productivity and introduce a high probability of significant cost and  schedule impacts. Program officials reported that actions are under way to  mitigate the risk through performance modeling and test planning.  However, neither the fiscal year 2007 expenditure plan nor the quarterly  reports to Congress disclose this COTS product change, its impact on  release schedules and cost estimates, or the risk to future system  performance.
Risk management: All program risks are not being effectively managed.
The program office has developed a process guide and implemented an  automated tool (database) for managing ACE risks in accordance with  relevant guidance and best practices. Although the database contains fields  to provide a description, level (high, medium, or low), and mitigation  strategy (including start and end dates, exit criteria, and implementation  status) for each risk, the completeness and quality of this information  varies. Because of such database limitations, we could not determine the  status of and mitigation progress on 17 risks. Moreover, these database  limitations were not reflected in the documentation used at key program  events, indicating that the program does not have the risk-related  information that it needs to inform its program decisions and to reduce the  chances of potential problems becoming actual problems. Program  officials stated that they are taking steps to improve risk management,  including establishing a group to ensure the quality and completeness of  the database, holding regular group meetings with contract staff and team  leads to discuss risks and their impacts, and conducting risk management  training. To date, however, program risks have not been communicated to  oversight organizations through the 2007 expenditure plan or recent  quarterly reports to the House and Senate Appropriations Committees.
section_title: Conclusions
paragraphs: Over the past 7 years, CBP and DHS have worked to fulfill legislatively  mandated annual expenditure plan requirements and to implement dozens  of our recommendations related to these plans and management of the  program. Among other things, these requirements and recommendations  have promoted effective program management and accountability for  performance and results. As a result of these years of effort, the ACE  program is better positioned today for delivering promised capabilities and  benefits than it has been in the past.
Nevertheless, key program management practices relating to, for example,  human capital management, requirements management, and risk  management continue to remain a challenge, and other management areas,  such as information security and architecture alignment, continue to  require attention. As a result, avoiding major program schedule delays and  cost overruns remains a challenge as more of each appears to be on the  horizon. To further improve ACE management and minimize its exposure  to risk, it is important for CBP and DHS to remain vigilant in their efforts to  satisfy ACE legislative requirements and to fully implement our prior  recommendations. Moreover, it is important that they keep Congress fully  informed on where the program stands and what changes are planned to  address emerging cost overruns and schedule delays.
section_title: Recommendations for Executive Action
paragraphs: To further strengthen ACE management and promote accountability for  ACE performance and results, we are recommending that the Secretary of  Homeland Security direct the CBP Commissioner to ensure that future  quarterly reports to the House and Senate Appropriations Committees  disclose   the risks and associated mitigation strategies of not having fully  satisfied the expenditure plan legislative conditions and not having  completed implementation of all our prior recommendations;   the status and impacts on the program’s estimated cost and schedule  and lessons learned from ongoing efforts to redefine requirements and  to implement a different COTS product than originally selected; and   the program’s plans and actions for improving ACE risk management  and its current inventory of program risks, including their associated  mitigation strategies and the status of the strategies’ implementation.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report signed by the Director,  Departmental GAO/OIG Liaison, and reprinted in appendix II, DHS agreed  with our findings and stated that it is committed to addressing them.  Further, the department agreed with our recommendations and described  actions that it said are under way or planned to address them. While most  of the department's stated actions are consistent with our  recommendations, in one case they may not be sufficient. Specifically, the  department stated that it would ensure that future expenditure plans meet  all OMB capital planning and investment control review requirements by  attaching the required OMB budget submission for ACE, referred to as an  Exhibit 300, to all future plans. However, we reviewed the fiscal year 2007  ACE Exhibit 300 as part of our review of the fiscal year 2007 ACE  expenditure plan and found that it did not meet the OMB requirements  cited above. Therefore, unless DHS improves the quality of future ACE  Exhibit 300s by addressing the weaknesses we have identified, this action  alone may not fully address our recommendation.
We are sending copies of this report to the Chairmen and Ranking Minority  Members of the other Senate and House committees and subcommittees  that have authorization and oversight responsibilities for homeland  security. We are also sending copies to the DHS Secretary, the CBP  Commissioner and, on their request, to other interested parties. In addition,  the report will be available at no charge on the GAO Web site at  http://www.gao.gov.
Should you or your offices have any questions on matters discussed in this  report, please contact me at (202) 512-3459 or at hiter@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this report. Other contacts and key contributors  to this report are listed in appendix III.
section_title: Briefing to Subcommittees on Homeland Security, House and Senate Committees on Appropriations
subsections: section_title: Furthermore, program officials told us that although the prime contractor was contractually required to have an SEI CMM® Level 34 capability, they are no longer required to do so because funding contractor program management-related activities is now a lower priority relative to other competing demands for ACE funding.
paragraphs: GAO-06-580; GAO-05-267. As with our prior reviews of ACE expenditure plans, the evaluation does not include compliance with federal acquisition regulations or other federal rules and requirements beyond those encompassed by SEI’s Capability Maturity Models. The SA-CMM® is consistent with the acquisition guidelines and systems acquisition management practices of the federal government, and provides a  management framework that defines acquisition practices for such process areas as acquisition planning, solicitation, requirements development and  management, project management, contract tracking and oversight, and evaluation. Level 3 capability is more advanced than level 2 and indicates that acquisition management processes have been defined throughout the organization.
The program office refers to these plans, processes, and procedures as assets. required activities for program management processes and disseminating  them to ACE staff.
Objective 1: Legislative Conditions  Condition 4. The plan satisfies the condition that it include certification by the  DHS CIO that an IV&V agent is currently under contract.
On October 24, 2006, the DHS Deputy CIO certified in writing that an IV&V agent  is under contract for ACE and that the agent met applicable requirements and  standards. (See open recommendations section of this briefing for more  information on IV&V.)
Objective 1: Legislative Conditions  Condition 5. The plan, including related program documentation and program  officials’ statements, satisfies the requirement that it be reviewed and approved by  the DHS IRB, the Secretary of Homeland Security, and OMB.
The DHS IRB reviewed the program and approved the expenditure plan on  December 12, 2006.
The DHS Under Secretary for Management approved the expenditure plan on  behalf of the Secretary of Homeland Security on February 6, 2007.
OMB approved the expenditure plan on January 22, 2007.
Objective 1: Legislative Conditions  Condition 6. The plan satisfies the requirement that it be reviewed by GAO.
Our review was completed on July 26, 2007.
Open Recommendation 1: Define and implement an ACE accountability  framework that fulfills these conditions: a. Covers all program commitment areas, including key expected or estimated  system (1) capabilities, use, and quality; (2) benefits and mission value; (3) costs;  and (4) milestones and schedules.
Effective program management includes defining and measuring progress against  program commitments and being held accountable for results. Such commitments generally cover expected or estimated (1) capabilities and their associated use  and quality, (2) benefits and mission value, (3) costs, and (4) milestones and  schedules.
Since 2003, we have reported that such commitments for ACE have not always  been defined, although improvements have been made. Most recently, we  reported1 that the program office had developed an initial version of an  accountability framework for measuring several program commitments, such as  capabilities and milestones, but that other commitments, such as benefits, had  not been as well defined.
Objective 2: Open Recommendations Accountability Framework During the last year, the program office has continued to improve its coverage of  program commitments, as evidenced by the content of key program documents— ACE accountability framework, ACE Program Plan (August 2006), the CBP/ACE  performance reference model, periodic Program Management Review (PMR)  reports, and the fiscal year 2007 expenditure plan. For example,   The accountability framework now captures and integrates data on ACE functional and performance capabilities, benefits, and mission value,  estimated and actual costs, milestones and schedule, accomplishments, and  earned value management status. Further, the framework provides visibility  into each ACE release at several levels of detail, and it is being used by the  ACE Executive Director and others as the means for monitoring program  progress, issues, and decisions.
The PMR reports have included the accountability framework data.
The fiscal year 2007 ACE expenditure plan included an example of the accountability framework commitments from the September 2006 PMR report.
Objective 2: Open Recommendations  b. Ensures currency, relevance, and completeness of such commitments made to  Congress in expenditure plans.
We have previously reported that commitments made in expenditure plans relative  to program capabilities, benefits, costs, and schedules need to be current,  relevant, and complete. To the extent that they are not, the currency and  relevance of the plan and its utility to Congress as an accountability mechanism  are limited.
ACE expenditure plans have not always included such information. To address  this limitation, we reported last year1 that CBP was relying on quarterly reports to  Congress to provide the appropriations committees with more current, relevant,  and complete information about the program than could be provided in the  expenditure plan. However, we also noted that the quarterly reports were generally  submitted to Congress 3 to 4 months after the end of each quarter, thus limiting  their currency and relevance.
Objective 2: Open Recommendations  The fiscal year 2007 expenditure plan continues to include information about  program commitments that is not current. Specifically,   The latest expenditure plan was provided to the appropriations committees  on February 6, 2007. However, information from the ACE accountability  framework that was included in the plan, such as milestones, EVM values, and risks, were as of October 2006, making it about 4 months old. For  example, the expenditure plan listed the milestone for the Production  Readiness Review (PRR) for a key release, Release 5/Drop A1, as March  1, 2007. However, this milestone had already slipped by more than 4  months--to July 12, 2007--by the December 2006 PMR (held on January 5,  2007).
Objective 2: Open Recommendations  According to program officials, they continue to use the quarterly ACE status  reports to provide the appropriations committees with more current information.1 However, recent quarterly reports have not been current. For example, The December 2006 quarterly report was not provided until February 26,  2007, and it contained the same outdated PRR milestone as the  expenditure plan did for the release previously mentioned.
Other quarterly reports have been submitted to the appropriations  committees as many as 7 months after the end of the quarter.
CBP and DHS officials told us that the delays were due to the DHS review and  approval process and that incorporating the most current program information would delay the reports even longer. According to the ACE and DHS officials,  they are exploring ways to accelerate the review process.
Quarterly reports should be received by Congress approximately 60 days after the end of the quarter.
Objective 2: Open Recommendations  c. Ensures reliable data relevant to measuring progress against commitments.
The quality of the capabilities that a program is to deliver is a relevant program  commitment. One measure of the quality of system capabilities is the trend in the number and severity of unresolved system defects or problems. Reliable data  about these defects are needed so that system maturity can be understood and  informed investment decisions can be made.
We previously reported1 that ACE defect data were not always consistent  because the two tools that the program used to track defects were not integrated  and reconciled. As a result, the true status of ACE defects, and thus an important  measure of system quality, was not known.
Since then, the program office has integrated the two tools using a cross- referencing function and has instituted manual processes for reconciling the data  in each tool. The December 2006 quarterly congressional report stated that these  efforts have improved the ability to record, assess, and report on system quality  and performance.
Objective 2: Open Recommendations  d. Ensures reporting in future expenditure plans progress against commitments  contained in prior expenditure plans.
The value and utility of an expenditure plan for congressional oversight and  agency accountability depends in large part on whether the plan reports on  progress against commitments--system capabilities, benefits, costs, and  schedules--made in prior plans.
Historically, the ACE expenditure plans have not reported adequately on  progress against previous plan commitments. Most recently, we reported1 that the fiscal year 2006 expenditure plan contained several such reporting gaps,  such as whether funding amounts were actually obligated and expended as  planned and whether releases met schedules. In particular, we reported that the  plan did not address whether the design and development for Release 5 was  actually accomplished as planned.
Objective 2: Open Recommendations  The fiscal year 2007 expenditure plan still does not adequately describe the  program’s progress against the commitments that were made in the fiscal year  2006 plan. For example,  The plan does not report progress against milestones in the fiscal year  2006 plan or explain why these milestones were not achieved.
The plan does not report actual obligation or expenditure of funds relative  to the planned uses of these funds in prior expenditure plans.
The plan did not address progress against the milestone dates for each  stage of a release that was included in the prior year’s plan. As a case in  point, the fiscal year 2006 expenditure plan described specific functionality  planned for Release 5/Drop A1 (Master Data and Enhanced Accounts),  such as online registration for trade representatives. However, the fiscal  year 2007 expenditure plan does not include information on whether these  functions were delivered, stating only that Release 5 functionality would be  deployed in two phases.
Objective 2: Open Recommendations  According to program officials, the quarterly congressional reports provide more  current information on the program’s progress against prior commitments.  However, we found that these reports have also not fully addressed the  commitments made in prior expenditure plans. For example, recent quarterly reports describe progress against developmental milestones for each release, but  do not report whether key functionality associated with each release was  delivered. In particular, the report for the first quarter of fiscal year 2007 identified  planned future capabilities and associated milestones for Release 5/Drop A1 but  did not address the status of the key functions associated with this release.
Program officials stated that they intend to start providing this information in the  last quarterly report of each year.
Objective 2: Open Recommendations  e. Ensures use of criteria for exiting key readiness milestones that adequately  consider indicators of system maturity, such as severity of open defects, and  document the milestone decisions in a way that reflects the associated risks and  plans for mitigating them.1 As noted earlier, one measure of the quality of system capabilities being  delivered is the number and severity of unresolved system defects or problems.  As such, information on such defects is an important consideration when  program decisions, such as key milestone decisions, are made.
We previously reported2 that several key milestones were passed that had  severe open defects and that program officials were unable to provide  documentation regarding how the risks associated with these defects were  assessed. We also reported that these risks were not being tracked in the  program’s risk database.
Objective 2: Open Recommendations  Since then, the system life cycle gate review process for Office of Information  and Technology projects, including ACE, has been amended to include  milestone readiness decisions based on an assessment and acceptance of risks  (including the risks related to unresolved defects). Further, the ACE Risk and  Issue Management Process guidance provides for the systematic identification,  analysis, prioritization, planning, execution, evaluation, and documentation of  program risks and issues and requires that any risks associated with going  forward should be identified as part of each life cycle gate review.
This process has been applied in the following two major gate reviews:  Release A1 (ESAR) Critical Design Review (May 2006) and  Release M1 (E-Manifest: Rail and Sea Manifest) Critical Design Review  (August 2006).
Objective 2: Open Recommendations  In the case of Release 5/Drop A1, program officials reported that no severe  defects existed, but that all known risks associated with proceeding past the  milestone had been entered into the program’s risk database and documented in  the certification package submitted to the DHS CIO.
The quarterly reports to Congress similarly state that risks related to gate review  decisions and the associated impacts are entered into a database to ensure  visibility and mitigation and are included in the package submitted to the DHS CIO  for review and certification to pass a given milestone.
Going forward, officials stated that similar assessments will be part of the  following planned gate reviews for Release 5/Drop A1:  Operational Readiness Review (planned for August 23, 2007) and  Live operations (planned for August 25, 2007).
Objective 2: Open Recommendations  Open recommendation 2: Develop the range of realistic ACE performance  measures and targets needed to support an outcome-based, results-oriented  accountability framework, including user satisfaction with ACE.
We have previously reported on both the absence of meaningful performance  measures to understand ACE progress, quality, and results, and have raised  concerns about the practicality and applicability of some measures that have been  defined. Most recently, we reported1 that defined ACE performance targets are not  always realistic and that goals, expected mission benefits, and performance  measures are not fully defined and adequately aligned.
In response, the program office has established an initial set of performance  measures that are tied to program objectives and related performance goals, and it  has established processes for the collecting, analyzing, and integrating  performance data for each measure. Among other things, these performance  measures address user satisfaction, efficiency, and productivity. Moreover, these  measures have been made an integral part of the ACE accountability framework and they were approved by the CBP Commissioner and the DHS CIO on June 30  and July 6, 2006, respectively.
Objective 2: Open Recommendations  According to program officials, these initial measures will continue to be revised  and augmented, and thus will evolve over time based on lessons learned, changes  to existing release capabilities, and refinements of requirements for future releases.  For example, the program office reported in March 2007 that new measures are  still being developed, reviewed, and evaluated for Releases 2, 3, and 4 at the same  time that performance data is being collected and reported for existing measures  for these releases. According to program officials, the new measures will be  combined with existing performance measures.
In addition, CBP’s December 2006 quarterly congressional report stated that  performance measures for future releases, such as Release 5, are being scheduled for identification and development and that changes are being made to  existing measures when they are determined to be inappropriate. For example, the  program office learned that the performance measure “percentage of truck  manifests being filed electronically” did not recognize that empty trucks are not  required to file manifests, and thus it revised the metric used to divide the number  of electronically filed truck manifests by the total number of required truck manifests  instead of the total number of trucks in order to get a more meaningful reflection of  ACE performance.
To assist in managing these performance measures, the program has established  a life cycle process for performance measures as well as a database tool to record  and manage modifications.
Objective 2: Open Recommendations  Performance Measures Alignment  Open recommendation 3: Explicitly align ACE program goals, benefits, desired  business outcomes, and performance measures.
Use advanced passenger and cargo information (NTC, ATS-Air, ATS, Screening and Targeting- ACE) to pre-screen, target, and identify potential terrorists and terrorist shipments and any related activity.
Increased use of targeting.
subsections: section_title: Further, this model links the measures to specific ACE releases. For example, the above performance measure applies to Screening Foundation (S1) and Targeting Foundation (S2).
paragraphs: Objective 2: Open Recommendations Management Improvement Measures Open recommendation 4: Define measures and collect and use associated  metrics for determining whether prior and future program management  improvements are successful.
As we have previously reported,1 investments in program management  improvements should include defined measures of progress and results. To date,  the program office has implemented a number of such improvements; however, it  has not had measures or metrics to determine the success of the improvements.  Moreover, the December 2006 quarterly congressional report stated that the  program anticipates more changes, including   creation of a cargo requirement management board to decide the disposition  of all change requests to production systems;   establishment of a new invoice review policy; and   co-location of personnel within a given business area.
Objective 2: Open Recommendations Management Improvement Measures According to its December 2006 quarterly reports to Congress, the program office  plans to measure the impact of future management improvements. However, program officials told us that they have yet to define them and thus are not yet  using such measures.
Objective 2: Open Recommendations Legislative Conditions Open recommendation 5: Fully address those legislative conditions associated  with measuring ACE performance and results and employing effective IV&V  practices.
Among the legislative conditions that ACE expenditure plans have been required to  meet are satisfaction of OMB guidance, including that associated with measuring  program performance and results and use of effective IV&V practices. These  conditions reflect good program management practices and, if implemented  properly, can reduce program risks.
As previously discussed in this briefing, the program office has developed a range  of ACE performance measures and taken steps to align them with program, CBP,  and DHS strategic goals and outcomes. According to program officials, they plan to  continue to evolve and refine the performance measures and include the measures  in the ACE accountability framework.
Objective 2: Open Recommendations  In January 2006, the DHS CIO certified that an IV&V agent was under contract for  the ACE program, but noted several issues, including these:   mechanisms were needed to ensure that products were complete, of sufficient quality, and met the needs of the user and  a more explicit technical approach, describing when and how certain activities  should (or should not be) performed, was needed.
Moreover, we subsequently reported1 that the scope of the contractor’s activities  did not extend to both IV&V of key system products and development processes.
Objective 2: Open Recommendations  On October 24, 2006, the Deputy CIO again certified that an IV&V agent was under  contract and stated that the previous issues had been addressed. Since then, the  program office has also developed an IV&V Implementation Management Plan that  addresses the concerns raised by the DHS CIO and us. In particular, the plan requires an IV&V program consistent with the industry standard;  provides a set of objectives, guidelines, and expectations for IV&V activities,  including periodic independent reports on status, observations,  recommendations, and activities; and   addresses satisfaction of quality standards for ACE products and user needs.
Program officials told us that IV&V has allowed early identification and correction of  program process and product weakness.
Objective 2: Open Recommendations Reconciliation of Cost Estimates Open recommendation 6: Ensure that future expenditure plans are based on  cost estimates that are reconciled with independent cost estimates.
It is important that expenditure plans be based on reliable estimates of costs, to  include reconciling differences between government and independent cost  estimates. We recently reported1 that the cost estimate in the fiscal year 2006  expenditure plan was based on government and independent cost estimates that  had been compared and found to be consistent.
For the fiscal year 2007 expenditure plan, our analysis showed that the  government and independent cost estimates differed by about 15 percent.  According to program officials, they reconciled these differences in January 2007,  and concluded that the results did not warrant changes to the expenditure plan  because the government estimates used in the expenditure plan were more  accurate.
Objective 2: Open Recommendations Reconciliation of Cost Estimates According to program officials, two primary factors account for the difference in  estimates. Specifically,  The estimates assumed different timelines for completing development of all  releases. The independent estimate assumed development completion by  fiscal year 2010, while the government estimate assumed fiscal year 2011.  According to program office officials, this accounts for about five percent of the  difference.
The government estimate included a number of items that the independent  estimator did not. For example, the independent estimator did not include  training and outreach items, such as the cost of conferences with trade  associations, training materials, and associated travel. As a result, the  government estimated training and outreach costs over the life of the program  to be about $90 million, while the independent estimate put these costs at  about $39 million. The independent estimate has since been amended to  include the missing items, and the reconciliation process is continuing.
Objective 2: Open Recommendations Rigorous Cost Estimation Open recommendation 7: Develop and implement a rigorous and analytically  verifiable cost estimating program that embodies the tenets of effective estimating.1 The reliability of cost estimates is largely a function of the quality of the estimating  process used to derive them. We previously reported2 that the program did not  have a well-defined cost estimating process, but that it has since made progress in  strengthening its cost estimating program. Specifically, the program office has:  defined and documented processes for estimating program costs (including  management reserve costs) and  hired a contractor to develop costs estimates that were independent of the  government estimates.
In September 2006, an ACE support contractor reported that both the government  and independent cost estimation processes demonstrated significant conformance  to effective estimating practices and concluded that the program is using a rigorous  and verifiable cost estimating approach.
See, for example, models developed by the Carnegie Mellon University SEI, Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of Software Organizations, CMU/SEI-95-SR-005 (Pittsburgh, Pa.: Carnegie Mellon University, 1995) and A Manager's Checklist for Validating Software Cost and Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburgh, Pa.: Carnegie Mellon University,1995).
Open recommendation 8: Use EVM in developing all existing and future  releases.
EVM is a program management tool to measure progress by comparing, during a  given period of time, the value of work accomplished with the amount of work  expected to be accomplished. This comparison permits performance to be  evaluated based on calculated variances from the planned (baselined) cost and  schedule. EVM is both an industry accepted practice and an OMB requirement.
We recently reported1 that the program office was not using EVM for all releases  (e.g., Release 5) due to changes to release baselines and the lack of familiarity  on the part of program staff with EVM practices.
Since then, the program office has established the performance baselines needed  to implement EVM for Release 5/Drops A1 and A2, Screening 3, and Release 6/Drop M1.
Objective 2: Open Recommendations  Earned Value Management  For these releases, the program office reports that it has also applied EVM  standards in establishing baselines and included EVM data in the accountability  framework for management and decision-making purposes. Program officials also  reported that they plan to implement EVM on future releases and task orders.
Objective 2: Open Recommendations  Open recommendation 9: Have future ACE expenditure plans specifically  address any proposals or plans, whether tentative or approved, for extending and  using ACE infrastructure to support other homeland security applications,  including any impact on ACE of such proposals and plans.
Together, the fiscal year 2007 expenditure plan and the quarterly reports to  Congress address ACE’s relationships with other trade processing and DHS  applications. For example, the plan discusses efforts to work with participating  government agencies in defining and deploying the International Trade Data  System (ITDS). The stated goal is to deliver ACE/ITDS in an integrated and  coordinated manner. In this regard, the plan discusses workshops to gather  requirements from participating agencies for Release 6/Drop M2 (first held on July  19, 2006), a working group to address these agencies’ HAZMAT issues  (established on July 17, 2006), and efforts to develop data element inputs for  ACE from other government agencies.
Objective 2: Open Recommendations  In addition, the quarterly reports to Congress cite coordination efforts with related  homeland security programs. For example, the report for the first quarter of 2007  states that  Container Security Initiative will be supported by ACE Release 6 and  Customs-Trade Partnership Against Terrorism is coordinating with ACE  Release 5/Drop A1 capabilities to provide both CBP and trade  representatives with the ability to view the status of CBP programs; and  U.S.-Mexico Border Partnership Plan will coordinate with ACE to implement  any cargo screening standards derived from partnership plan agreements.
In addition, CBP reports that ACE and other CBP applications1 share infrastructure (desktops, clients, and local area networks) at the ports of entry.  This infrastructure is purchased and maintained by CBP’s OIT Program  Integration Division. Each application, including ACE, is responsible for complying  with OIT’s infrastructure standards.
CBP applications include IT systems that provide tools and information to help front-line officers ensure the security of our nation. This includes applications for the United States Visitor and Immigrant Status Indicator Technology (US-VISIT), Container Security Initiative and Automated Targeting System.
Objective 2: Open Recommendations  Overlap and Concurrence  Open recommendation 10: Minimize the degree of overlap and concurrency  across ongoing and future ACE releases and capture and mitigate the associated  risks of any residual concurrence.
Significant overlap and concurrency among major program activities, such as  releases, introduce considerable cost and schedule risks as they can create  contention for limited resources among the releases. We have continued to report  on extensive overlap and concurrence in ACE releases and the cost overruns and  schedule delays that have resulted. Most recently, we reported1 that the ACE  schedule continued to provide for such overlap and concurrency and that the risks  associated with doing so were not being effectively addressed.
Since then, the program office has reduced this overlap and concurrence. For example, it has  decoupled (i.e., reduced dependencies among) certain ACE program components by separating Screening 1-3 from Releases 4-6 and   aligned the development and delivery of functionality for different releases  with the availability of required hardware environments.
Recent quarterly congressional reports stated that CBP has taken actions to  reduce potential contention for limited resources. Examples include  dividing releases into smaller subreleases, called drops, to provide more  improving planning for development, integration, testing, and training activities  and milestones to better schedule use of development and test environments;  and  centralizing management of shared software services to address, among  other things, allocation of resources and responsiveness to workload peaks  and use of consistent technical management approaches across releases.
Moreover, the program’s risk database identifies overlap and concurrency-related  risks. However, the mitigation strategies for these risks contain vague or  incomplete data. These data limitations make it difficult to determine the status or  the effectiveness of the efforts to reduce the risks associated with overlap and  concurrence among releases. (See the observations section of this briefing for  additional information on program risk management.)
Objective 2: Open Recommendations  The program office is also conducting regular integration meetings with the teams  supporting each release to discuss concerns, decisions, and schedules associated  with resource availability and is using a software tool to track and mitigate release- specific concurrency risks.
Notwithstanding these steps, the program continues to face challenges in  managing dependencies among releases, which any associated concurrency in the  program’s development exacerbates. For example, the schedule slips expected  with Release 5/Drop A2 will affect Release 6/Drop M1, as resources have been  shifted from M1 to address the A2 delay. In addition, further delays in A1 and A2  will impact the implementation of M1.
Objective 2: Open Recommendations  Open recommendation 11: Direct the appropriate departmental officials to fully  address those legislative conditions associated with having an approved privacy  impact assessment (PIA)1 and ensuring architectural alignment.
The department approved the ACE PIA for Release 4 in July 2006, however, this  assessment does not cover other completed releases. Further, DHS has determined that ACE is aligned with DHS architecture, but we have yet to receive  documentation to adequately understand and verify the determination.
PIA (In progress)
OMB, Guidance for Implementing the Privacy Provisions of the EGovernment Act of 2002, OMB M-03-22 (Sept. 26, 2003).
Objective 2: Open Recommendations  The program office has developed a PIA for Release 4 of the ACE e-Manifest:  Trucks and the International Trade Data System, and DHS approved it on July 14,  2006. As noted earlier, our analysis shows that this PIA addresses the major elements of DHS’s guidance. Program officials stated that they would update the  assessment for each ACE release. This assessment does not cover other  recently completed screening releases (i.e., S1 and S2). However, program  officials told us that S1 and S2 are considered to be part of the Automated  Targeting System (ATS), and are therefore covered by the ATS PIA. However,  our analysis of the ATS PIA showed that while it addresses screening and  targeting functions, it does not specifically identify or address releases S1 and S2.
Architectural Alignment (in progress)
As discussed in the legislative conditions section of this briefing, DHS has  determined that ACE is aligned with the DHS EA. For example,  In December 2006, DHS’s EA Center of Excellence determined that ACE  was conditionally compliant with the DHS architecture, but that the program  needed to take actions to address four conditions.
Objective 2: Open Recommendations  Privacy Assessment and Architectural Alignment  On March 2007, the DHS EA Board reported that ACE had satisfied three of  the conditions and recommended approval of ACE’s request for program  alignment with one remaining condition—alignment with the DHS Technical  Reference Model.
In May 2007, DHS EA officials reported that all required products and  technologies were aligned with the Technical Reference Model and that ACE  was thus aligned with the DHS EA.
However, as was also discussed earlier in this briefing, we have yet to receive  sufficient documentation describing the criteria and methodology used to make  these determinations or verifiable analysis supporting the determinations.  Moreover, the determinations were based on technical alignment and did not  address other relevant aspects of program alignment to an EA, such as data  alignment. Thus, the program is at risk of being designed and implemented in a  way that is not consistent with all relevant aspects of the DHS EA, such as data  structures and standards, and thus does not support optimized DHS-wide  operations, performance, and results.
Open recommendation 12: Develop and implement key human capital  management practices.
As we have previously reported,1 effective strategic management of program  human capital includes, among other things, defining the skill sets needed to  perform program functions, assessing and inventorying the skill sets of the  program’s current workforce and assessing any associated skill gaps in meeting  future needs, and developing strategies to fill any identified gaps. Moreover, it  includes having a well-defined plan that provides for effective implementation of  these processes.
In June 2006, CBP executives approved the OIT Strategic Human Capital  Management Plan. The plan is intended to be a road map for supporting CBP’s workforce vision and ensuring effective management of human capital resources  across OIT to address enterprise-wide priorities. An ACE-specific plan was  included as an appendix to the CBP plan. This appendix was intended to provide  better near- and long-term human capital management practices for the OIT  offices involved in ACE development.
Objective 2: Open Recommendations Human Capital Practices Neither the OIT nor the ACE-specific plan addresses the basic tenets of effective  human capital management. For example, neither provides for:   defining the positions needed (including core competencies) to perform core   assessing and inventorying current workforce skills and abilities;   assessing any gaps between needed and existing workforce levels and  filling identified gaps via such means as hiring new staff, training existing  staff, and augmenting staff with contractor support.
OIT officials acknowledged these limitations in the plans and stated that they are  developing an implementation plan to address these shortfalls. The officials  stated that the implementation plan will include accountability, timeframes, and  metrics to carry out the larger OIT Strategic Human Capital Management Plan.
According to these officials, this implementation plan was to be completed in  January 2007, following presentation to the OIT Deputy Director's Council and  approval by the OIT Office Directors and Assistant Commissioner. However, as of  July 2007, it had not yet been approved and thus was not available for our review.
In the interim, OIT and program officials reported taking various steps to  address ACE human capital needs. These include, for example,   Using various methods to fill staff vacancies. As of May 2007, the program  reports that it has 62 full-time employees and one vacancy and that  additional staff are working on ACE either full- or part-time.
Lobbying OPM for more flexibility in recruiting, to include higher salaries  and direct hiring authority.
Reorganizing OIT into six program offices aligned to major mission areas  so that the number of government personnel responsible for ACE  development activities can be augmented by IT functional program management expertise.
Offering training to improve the skills of staff.
Nevertheless, these steps have not been guided by a well-defined plan and  thus represent activities that are not tied to strategic goals and outcomes and thus cannot be measured against them. Without a plan, it is unlikely that the  program will be able to adequately ensure that it has the right people at the  right time to deliver ACE successfully.
Objective 2: Open Recommendations  Open recommendation 13: Include in the June 30, 2006, quarterly update report  to the appropriations committees a strategy for managing ACE human capital  needs and the ACE framework for managing performance and ensuring  accountability.
The June 30, 2006 quarterly report to the House and Senate Appropriations  Committees included the program’s strategy for meeting its human capital needs and its accountability framework.
Human Capital Strategy (In progress)
The June 30, 2006, quarterly report to Congress included a description of the  previously described ACE-specific appendix to the OIT Strategic Human Capital  Management Plan, including the plan’s five goals and strategies. Moreover, the  report states that this ACE-specific plan is aligned with OPM’s Human Capital  Assessment and Accountability Framework.1 As previously stated, however, this  ACE-specific plan does not meet the basic tenets of strategic human capital  management, as defined in this framework and other relevant guidance.
As revised by OPM in 2005, this framework reflects guidance from the collaboration of OMB, OPM, and GAO.
Objective 2: Open Recommendations  According to program officials, they are developing a new OIT human capital  strategy and implementation plan, but they have yet to provide us with a date when  it will be completed.
Accountability Framework (Complete)
The June 30, 2006, quarterly report included a description of the ACE  accountability framework. As previously discussed, this framework is the means by  which the program measures performance relative to promised ACE capabilities,  costs, schedules, earned value, risks, and mission values and benefits. The report  also included an appendix illustrating the format and content of the accountability  framework tool. At that time, CBP reported that it would continue to work with  stakeholders to further enhance the format, readability, and utility of the framework  as a program management and reporting tool.
Since June 2006, the program office has refined and implemented the  accountability framework. The framework covers all program commitment areas  and key system aspects to support executive decision making and provides  external program stakeholders with information on the program.
Objective 2: Open Recommendations Quarterly Report on Open Recommendations Open recommendation 14: Report quarterly to the House and Senate  Appropriations Committees on efforts to address open GAO recommendations.
CBP has submitted quarterly reports on ACE to both the House and the Senate  Appropriations Subcommittees since November 2002, including reports for each  quarter of fiscal year 2006 and the first and second quarters of fiscal year 2007.  These reports have addressed CBP’s efforts to address open GAO recommendations.
Objective 2: Open Recommendations  Accurately Report on Open Recommendations  Open recommendation 15: Accurately report to the appropriations committees  on CBP's progress in implementing our prior recommendations.
As previously stated, CBP has included information on progress in meeting  GAO’s recommendations in its quarterly reports1 to the appropriations committees  since November 2002. However, some of this information is dated and thus  inaccurate due to the time lapse between when the reports are produced and  when they are provided to the appropriations committees. Recently, this time  lapse has been between about two and seven months. According to program  officials, they are exploring ways to accelerate the review process and thereby  improve the timeliness and accuracy of their reports to Congress. DHS officials  also told us that they have ongoing efforts to improve the review process;  however, the process has not improved consistently or significantly.
The reports are due to Congress approximately 60 days after the completion of a fiscal quarter. ACE quarterly reports are due to DHS approximately 30 days after the end of the quarter, and then DHS has 30 days to review the report and submit it to Congress.
Objective 3: Observations Requirements Limitations Likely to Cause Major Delays Observation 1: Redefinition of requirements for several ACE releases is now  under way to address limitations in completeness of originally defined  requirements, and this redefinition is likely to introduce program schedule delays  and cost increases.
A key aspect of successful system acquisition programs is having well-defined requirements. Among other things, requirements should be complete, and to  accomplish this, best practices advocate engaging all key stakeholders in the  requirements definition and management process.
In defining ACE requirements, the program office discovered that its original  requirements definition approach did not adequately engage all key stakeholders,  and to its credit, has since taken steps to address this. Specifically,  In the spring of 2004, the program office and the ACE support team conducted  more than 300 business process workshops with the ACE user community to  help define ACE requirements for the future releases. This requirements  definition approach was referred to as the Global Business Blueprint (GBB)  effort.
Objective 3: Observations Requirements Limitations Likely to Cause Delays In June and July of 2005, a key group of ACE stakeholders that were not  originally involved in the GBB (application programmers familiar with the  legacy system that ACE is to replace) raised questions about the completeness of the requirements. To address these questions, the program  office examined the GBB’s completeness by first reverse-engineering the  legacy software for a small number of legacy system programs and then comparing the reverse-engineered requirements to the GBB-derived  requirements. Based on this, which is referred to as legacy code decomposition, the program office found that the GBB-derived requirements  were missing about 20 percent of needed ACE functionality.
In August 2005, the program office decided that it needed to decompose all of  the legacy system code in order to completely capture the requirements for all  ACE releases. Work began with decomposition of the legacy code related to  ACE Release 5/Drop A2 (Entry Summary Accounts and Revenue).
Objective 3: Observations Requirements Limitations Likely to Cause Delays In September 2006, the program office determined that the legacy code decomposition approach alone was not sufficient to gain a full understanding  of the requirements given the size and complexity of A2. As a result, the  requirements definition process was expanded to engage another key  stakeholder group (business process subject matter experts). Under the  expanded approach, referred to as legacy code decomposition and  collaboration, legacy system software programmers and subject matter  experts were to examine the code line by line in defining ACE requirements.
In November 2006, the legacy system code decomposition and collaboration  effort for Release 5/Drop A2 fell behind schedule because of lack of personnel  with legacy system expertise and experience. The schedule for A2 was tentatively revised, but at the time of our review, the schedule was still under  review, had not yet been approved, and thus was not yet available for our  review.
Objective 3: Observations Requirements Limitations Likely to Cause Delays The program office has identified the A2 legacy code decomposition and  collaboration process as a high risk item that will significantly impact the A2  schedule. For example, the decomposition and collaboration process for part of the  functionality on A2’s critical path—the Authorized Broker Interface Entry  Summary—is not expected to be completed before early 2008, at the earliest.
According to program officials, delays for other releases/drops, such as Release  5/Drop A1 and Release 6/Drop M1, will not be as significant. They also said that, while they have not yet estimated how long A2 will be delayed and what the  associated cost implications are, they do not expect the cost increases to breach  the current acquisition program baseline of $3.3 billion, which translates into a cost  increase of less than $200 million. Moreover, they said that several actions have  been taken to minimize the impact of the delays. For example,  A2 functionality necessary for M1 has been given priority in order to support  M1 deployment as originally planned and  A2 scope is being divided into increments to allow some functionality to be  delivered sooner and to minimize the impact on other drops.
Objective 3: Observations Requirements Limitations Likely to Cause Delays However, these actions carry consequences, such as missed opportunities to  combine field training and an increase in the number of legacy interfaces, thus  increasing the potential for introducing software problems.
In addition, program officials told us that they are considering changes to the A2  and M1 deployment strategies to address stakeholder concerns, and they said that  these changes could also minimize the magnitude of the A2 and M1 delays. Specifically, they said that they had planned to deploy on a national basis, meaning  that A2 and M1 functionality would be deployed to all ports at the same time and  concurrently adopted by all users nationwide. However, deployment may change to  a filer-by-filer basis, meaning that A2 and M1 functionality would be deployed to all  ports at the same time, but not all filers would begin using it at the same time.
Objective 3: Observations Requirements Limitations Likely to Cause Delays According to program officials, they believe that the change in the deployment  strategy would both address stakeholder concerns and minimize any A2 and M1  schedule delays caused by the redefinition of requirements. However, they added  that these changes have yet to be approved, and the full extent of the cost and  schedule implications are not yet known. Moreover, neither the fiscal year 2007  expenditure plan nor the ACE quarterly reports have disclosed the A2 and M1  requirements redefinition and its impact, and neither has addressed any changes to  their deployment strategies.
Objective 3: Observations Key COTS Product Being Replaced Observation 2: Significant changes to ACE requirements have, in turn, led to  reevaluation and replacement of a key commercial off-the-shelf product (COTS)  previously selected and being prepared for use.
When acquiring commercial component-based systems, like ACE, best practices  advocate basing decisions on whether to employ a given COTS product on  thorough, rigorous, and continuous analysis of a number of factors, including how  well competing products satisfy defined system requirements.
To the program office’s credit, it reports having followed the CBP system life cycle  methodology to determine which COTS product would best meet the requirements  for Release 5/Drop A2. These analyses include: In 2002, the program office reviewed, in general terms, various COTS  packages and determined that a solution using SAP (formerly Systems  Application and Products), a COTS provider, combined with other commercial  solutions and customized development, provided the best combination of  capability, performance, and cost for ACE.
Objective 3: Observations Key COTS Product Being Replaced In 2004, a more detailed analysis was conducted as part of the previously  mentioned GBB process, which was intended to define and allocate ACE requirements to future ACE releases and provide the basis for, among other  things, selecting a specific SAP product. At that time, the SAP Enterprise  Portal product was selected for Release 5/Drop A2.
In December 2006, the ACE Chief System Architect recommended that the  Internet Transaction Server (ITS) technology already used by CBP should be  adopted instead of the SAP Enterprise Portal, based on the determination that  all currently planned SAP functionality could be presented using the ITS  technology. This decision was based on improved understanding of the requirements and previous analyses of ITS.
Objective 3: Observations Key COTS Product Being Replaced On the basis of these analyses and the Chief System Architect’s recommendation,  the program office subsequently stopped work on SAP Enterprise Portal design  and configuration efforts and, in March 2007, the program reported that the SAP  Internet Transaction Server would be used for Release 5/Drop A2 instead of the  SAP Enterprise Portal. While this decision was expected to have some near-term  schedule impact because much of the completed work for A2 had been based on  the planned use of SAP Enterprise Portal, the program office reports that these  impacts are offset by the cost advantages of other ACE releases already using the  Internet Transaction Server technology.
However, in March 2007, program officials identified a risk of inadequate response  time for the Internet Transaction Server—thus negatively impacting user  productivity—and that there was high probability of significant cost and schedule  impacts. Actions are underway to mitigate the risk through performance modeling  and test planning.
Neither the fiscal year 2007 expenditure plan nor the quarterly reports to Congress  disclose this COTS product change, its impact on release schedules and cost  estimates, or risk to future system performance.
Observation 3: All program risks are not being effectively managed.
Risk management is a continuous, forward-looking process that is intended to  either prevent program cost, schedule, and performance problems from occurring  or to minimize the impact if they occur. According to relevant guidance and best  practices, effective risk management involves proactively identifying, assessing,  and disclosing risks; defining and implementing cost-effective strategies for  mitigating these risks; and measuring and disclosing progress in doing so.
To its credit, the program office has developed a process guide and implemented  an automated tool (database) for managing ACE risks in accordance with relevant  guidance and best practices. Among other things, the database contains the  description, level (high, medium, or low), and mitigation strategy (including start  and end dates, exit criteria, and implementation status) for each risk.
Objective 3: Observations Risks Not Being Effectively Managed However, the completeness and quality of the information on each of the risks in  the risk database1 vary. For example,   many risks were missing information on the status of efforts to implement the   many risks (18) were missing criteria for completing mitigation steps, clear  descriptions of what the risk entailed, and start and end dates for planned  mitigation activities.
Because of such database limitations, we could not determine the status of and  mitigation progress on 17 risks. Moreover, these database limitations were  reflected in the documentation used at key program events, such as PMRs. This  means that the program does not have the risk-related information that it needs to  inform its program decisions and to reduce the chances of potential problems  becoming actual problems.
As of May 23, 2007 the risk database contained 46 risks.
Objective 3: Observations Risks Not Being Effectively Managed Program officials, including the official responsible for risk management, stated that  risk management is immature and needs to be strengthened. The reasons they  gave for these risk management weaknesses are due to   all staff not being trained on how to use the tool (last training was provided in  2003, and the since then a number of people have joined the program); the tool is unique to the ACE program and thus no CBP guidance exists on its  use); and  each ACE group addresses risk differently in its weekly meetings.
To improve ACE risk management, program officials told us that they are:  establishing a group to ensure the quality and completeness of the database;  holding regular group meetings with contract staff and team leads to discuss  risks and their impacts; and   conducting risk management training.
Objective 3: Observations Risks Not Being Effectively Managed If implemented effectively, such steps should result in more meaningful information  about program risks that can be useful to DHS in managing the program and to  Congress in overseeing it. To date, however, ACE program risks have not been  communicated to oversight organizations through the 2007 expenditure plan or  recent quarterly reports to the House and Senate Appropriations Committees.
section_title: Nevertheless, key program management practices relating to, for example, human capital management, requirements management, and risk management continue to remain a challenge, and other management areas, such as information security and architecture alignment, continue to require attention. As a result, avoiding major program schedule delays and cost overruns remains a challenge as more of each appear to be on the horizon.
section_title: To further improve ACE management and minimize its exposure to risk, it is important for CBP and DHS to remain vigilant in their efforts to satisfy ACE legislative requirements and to fully implement our prior recommendations. Moreover, it is important that they keep the Congress fully informed on where the program stands and what changes are planned to address emerging cost overruns and schedule delays.
section_title: To further strengthen ACE management and promote accountability for ACE performance and results, we are making the following recommendation to the Secretary of Homeland Security to direct the CBP Commissioner to ensure that future quarterly reports to the House and Senate Appropriations Committees disclose
paragraphs: (1)the risks and associated mitigation strategies of not having fully satisfied the  expenditure plan legislative conditions and not having completed implementation of all our prior recommendations;  (2)the status and impacts on the program’s estimated cost and schedule and  lessons learned from ongoing efforts to redefine requirements and to  implement a different COTS product than originally selected; and (3)the program’s plans and actions for improving ACE risk management and its  current inventory of program risks, including their associated mitigation  strategies and the status of the strategies’ implementation.
section_title: In oral comments on a draft of this briefing, DHS and CBP officials agreed with our conclusions and recommendations, and provided clarifying information and technical comments that we incorporated in the briefing, as appropriate.
section_title: Scope and Methodology
subsections: section_title: To accomplish our objectives, we analyzed the ACE fiscal year 2007 expenditure plan and supporting documentation, and compared them to relevant federal requirements and guidance, applicable best practices, and our prior recommendations. We also interviewed DHS and CBP officials and ACE program contractors. In particular, we reviewed
paragraphs: SEI’s institutional and project-specific estimating guidelines are defined in Robert E. Park, Checklists and Criteria for  Evaluating the Cost and Schedule Estimating Capabilities of Software Organizations, CMU/SEI-95-SR-005 (Pittsburgh, Pa.:  Carnegie Mellon University Software Engineering Institute, 1995) and A Manager’s Checklist for Validating Software Cost  and Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburg, Pa: 1995), respectively.
Attachment 1 Scope and Methodology independent verification and validation (IV&V) activities using the Institute of  Electrical and Electronics Engineers standard for software verification and  validation;2  CBP actions to coordinate ACE with related programs;  CBP’s reorganization documentation, including the organizational charts and  roles and responsibilities matrix;  ACE’s accountability framework; and  cost and schedule data and program commitments from program  management documentation.
Attachment 1 Scope and Methodology For DHS-, CBP-, and contractor-provided data that we did not substantiate, we  have made appropriate attribution indicating the data's source.
We conducted our work at CBP headquarters and contractor facilities in the  Washington, D.C., metropolitan area from December 2006 through July 2007 in  accordance with generally accepted government auditing standards.
Information Technology: Customs Has Made Progress on Automated  Commercial Environment System, but IT Faces Long-Standing Management  Challenges and New Risks. GAO-06-580. Washington, D.C.: May 31, 2006.
Information Technology: Customs Automated Commercial Environment Program Progressing, but Need for Management Improvements Continues. GAO-05-267. Washington, D.C.: March 14, 2005.
Information Technology: Early Releases of Customs Trade System Operating,  but Pattern of Cost and Schedule Problems Needs to Be Addressed. GAO-04- 719. Washington, D.C.: May 14, 2004.
Customs Service Modernization: Automated Commercial Environment  Progressing, but Further Acquisition Management Improvements Needed. GAO- 03-406. Washington, D.C.: February 28, 2003.
Customs Service Modernization: Third Expenditure Plan Meets Legislative  Conditions, but Cost Estimating Improvements Needed. GAO-02-908. Washington, D.C.: August 9, 2002.
Customs Service Modernization: Management Improvements Needed on High- Risk Automated Commercial Environment Project. GAO-02-545. Washington,  D.C.: May 13, 2002.
Customs Service Modernization: Results of Review of First Automated  Commercial Environment Expenditure Plan. GAO-01-696. Washington, D.C.:  June 5, 2001.
Information Security: Homeland Security Needs to Enhance Effectiveness of Its  Program. GAO-07-1003T. Washington, D.C.: June 20, 2007.
Intellectual Property: Better Data Analysis and Integration Could Help U.S.  Customs and Border Protection Improve Border Enforcement Efforts. GAO-07- 735. Washington, D.C.: April 26, 2007.
Cargo Container Inspections: Preliminary Observations on the Status of Efforts  to Improve the Automated Targeting System. GAO-06-591T. Washington, D.C.:  March 30, 2006.
section_title: Comments from the Department of Homeland Security
section_title: Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the person named above, Daniel Castro, Dawn Day, Neil  Doherty, Nancy Glover, Paula Moore, Jamelyn Payan, Nik Rapelje, and  Karen Talley made key contributions to this report.