DOD Business Systems Modernization: Important Management Controls Being Implemented on Major Navy Program, but Improvements Needed in Key Areas
Released: 2008-09-08T00:00:00

Letter

Background

DON’s primary mission is to organize, train, maintain, and equip combat- ready naval forces capable of winning the global war on terrorism and any other armed conflict, deterring aggression by would-be foes, preserving freedom of the seas, and promoting peace and security. To support this mission, DON performs a variety of interrelated and interdependent business functions (e.g., acquisition and financial management), relying heavily on IT systems. In fiscal year 2008, DON’s budget for business systems and associated infrastructure was about $2.7 billion, of which $2.2 billion was allocated to operations and maintenance of existing systems and the remaining $500 million to systems in development and modernization. Of the approximately 3,000 business systems that DOD reports in its current inventory, DON accounts for 904, or about 30 percent, of the total. Navy ERP is one such system investment.

Navy ERP: A Brief Description

In July 2003, the Assistant Secretary of the Navy for Research, Development, and Acquisition established Navy ERP to “converge” four separate pilot programs that were under way at four separate Navy commands. This program is to leverage a commercial, off-the-shelf software known as an enterprise resource planning product. Such products consist of multiple, integrated functional modules that perform a variety of business-related tasks, such as acquisition and financial management. Table 1 provides a brief description and status of each of the pilots.

According to DOD, Navy ERP is to address the Navy’s long-standing problems related to financial transparency and asset visibility. Specifically, the program is intended to standardize the Navy’s acquisition, financial, program management, maintenance, plant and wholesale supply, and workforce management business processes across its dispersed organizational components. When the program is fully implemented, it is to support over 86,000 users.

Navy ERP is being developed in a series of increments using the Systems Applications and Products (SAP) commercial software package, augmented as needed by customized software. SAP consists of multiple, integrated functional modules that perform a variety of business related tasks, such as finance and acquisition. The first increment, called Template 1, is currently the only funded portion of the program and consists of three releases: 1.0 Financial and Acquisition, 1.1 Wholesale and Retail Supply, and 1.2 Intermediate-Level Maintenance. Release 1.0 is the largest of the three releases in terms of the functional requirements being addressed. Specifically, it is to provide about 56 percent of Template 1 requirements. See table 2 for a description of these releases.

DON estimates the life cycle cost for the program’s first increment to be about $2.4 billion, including about $1 billion for acquisition, and $1.4 billion for operations and maintenance. The life cycle cost of the entire program has not yet been determined because future increments have not been defined. The program office reported that approximately $400 million was spent from fiscal year 2004 through fiscal year 2007 on the first increment. For fiscal year 2008, about $200 million is planned to be spent.

Program Oversight and Management Roles and Responsibilities

To manage the acquisition and deployment of Navy ERP, DON established a program management office within the Program Executive Office for Executive Information Systems. The program office manages the program’s scope and funding and is responsible for ensuring that the program meets its objectives. To accomplish this, the program office is responsible for key program management areas, such as architectural alignment, economic justification, earned value management, requirements management, and risk management. In addition, various DOD and DON organizations share program oversight and review activities. A listing of key entities and their roles and responsibilities is in table 3.

Overview of Navy ERP’s Status

The first increment of Navy ERP is currently in the production and deployment phase of the defense acquisition system. The system consists of five key program life cycle phases and three related milestone decision points. These five phases and related milestones, along with a summary of key program activities completed during or planned for each phase, are as follows: 1. Concept Refinement: The purpose of this phase is to refine the initial system solution (concept) and create a strategy for acquiring the solution. This phase began in July 2003, at which time DON began to converge the four pilot programs into Navy ERP and developed its first cost estimate in September 2003. This phase of the program was combined with the next phase, thus creating a combined Milestone A/B decision point. 2. Technology Development: The purpose of this phase is to determine the appropriate set of technologies to be integrated into the investment solution by iteratively assessing the viability of the various technologies while simultaneously refining user requirements. During the combined Concept Refinement and Technology Development phase, the program office prepared a concept of operations and operational requirements document; performed an analysis of alternatives, business case analysis, and economic analysis; and established its first Acquisition Program Baseline. It also selected SAP as the commercial off-the-shelf ERP software. The combined phase was completed in August 2004, when the MDA approved Milestone A/B to allow the program to move to the next phase. 3. System Development and Demonstration: The purpose of this phase is to develop a system and demonstrate through developer testing that the system can function in its target environment. This phase was completed in September 2007, when Release 1.0 passed development testing and its deployment to NAVAIR began. This was 17 months later than the program’s original schedule set in August 2004 but on time according to the revised schedule set in December 2006.

In September 2004, the program office awarded a $176 million system integration contract to BearingPoint for full system design, development, and delivery using SAP’s off-the-shelf product and related customized software. In January 2006, the program office (1) reduced the contractor’s scope of work from development and integration of the first increment to only development of the first release and (2) assumed responsibility and accountability for overall system integration. According to the program office, reasons for this change included the need to change the development plan to reflect improvements in the latest SAP product released and the lack of authority by the contractor to adjudicate and reconcile differences among the various Navy user organizations (i.e., Navy commands).

In December 2006, the program office revised its Acquisition Program Baseline to reflect an increase of about $461 million in the life cycle cost estimate due, in part, to restructuring the program (e.g., changing the order of the releases, changing the role of system integrator from contractor to the program office) and resolving problems related to, among other things, converting data from legacy systems to run on Navy ERP and establishing interfaces between legacy systems and Navy ERP. In addition, the program office awarded a $151 million contract for Release 1.1 and 1.2 configuration and development to IBM in June 2007. In September 2007, prior to entering the next phase, the program revised its Acquisition Program Baseline again to reflect a $9 million decrease in the life cycle cost estimate and a 5-month increase in its program schedule. Soon after, the MDA approved Milestone C to move to the next phase. 4. Production and Deployment: The purpose of this phase is to achieve an operational capability that satisfies the mission needs, as verified through independent operational test and evaluation, and to implement the system at all applicable locations. This phase began in September 2007, focusing first on achieving initial operational capability (IOC) of Release 1.0 at NAVAIR by May 2008. This date is 22 months later than the baseline established for Milestone A/B in August 2004, and 4 months later than the new baseline established in September 2007. According to program documentation, these delays were due, in part, to challenges experienced at NAVAIR in converting data from legacy systems to run on the new system and implementing new business procedures associated with the system.

In light of the delays at NAVAIR in achieving IOC, the deployment schedules for the other commands were also revised. Specifically, Release 1.0 is still to be deployed at NAVSUP on October 2008, but Release 1.0 deployment at SPAWAR is now scheduled 18 months later than planned (October 2009), and deployment at NAVSEA general fund and Navy Working Capital Fund is now scheduled to be 12 months later than planned (October 2010 and 2011, respectively). Because of the Release 1.0 delays, Release 1.1 is now planned for deployment at NAVSUP 7 months later than planned (February 2010). Release 1.2 is still scheduled to be released at Regional Maintenance Centers in October 2010.

The program office is currently in the process of again re-baselining the program, and DON plans to address any cost overruns through reprogramming of fiscal year 2008 DON funds. It estimates that this phase will be completed with full operational capability (FOC) by August 2013 (26 months later than the baseline established in 2004, and 5 months later than the re-baseline established in September 2007). 5. Operations and Support: The purpose of this phase is to operationally sustain the system in the most cost-effective manner over its life cycle. In this phase, the program plans to provide centralized support to its users across all system commands. Each deployment site is expected to perform complementary support functions, such as data maintenance.

Overall, Increment 1 was originally planned to reach FOC in fiscal year 2011, and its estimated life cycle cost was about $1.87 billion. The estimate was later baselined in August 2004 at about $2.0 billion. In December 2006 and again in September 2007, the program was re- baselined. FOC is now planned for fiscal year 2013, and the estimated life cycle cost is about $2.4 billion (31 percent increase over the original estimate). Key activities for each phase are depicted in figure 1, changes in the deployment schedule are depicted in figure 2, and cost estimates are depicted in figure 3.

Use of IT Acquisition Management Controls Maximizes Chances for Success

IT acquisition management controls are tried and proven methods, processes, techniques, and activities that organizations define and use to minimize program risks and maximize the chances of a program’s success. Using these controls can result in better outcomes, including cost savings, improved service and product quality, and a better return on investment. For example, two software engineering analyses of nearly 200 systems acquisition projects indicate that teams using systems acquisition controls that reflected best practices produced cost savings of at least 11 percent over similar projects conducted by teams that did not employ the kind of rigor and discipline embedded in these practices. In addition, our research shows that these controls are a significant factor in successful acquisition outcomes, including increasing the likelihood that programs and projects will be executed within cost and schedule estimates.

We and others have identified and promoted the use of a number of IT acquisition management controls associated with acquiring IT systems. See table 4 for a description of several of these activities.

Prior GAO Reviews Have Identified IT Acquisition Management Control Weaknesses on DOD Business System Investments

We have previously reported that DOD has not effectively managed a number of business system investments. Among other things, our reviews of individual system investments have identified weaknesses in such things as architectural alignment and informed investment decision making, which are also the focus areas of the Fiscal Year 2005 Defense Authorization Act business system provisions. Our reviews have also identified weaknesses in other system acquisition and investment management areas—such as EVM, economic justification, requirements management, and risk management.

In July 2007, we reported that the Army’s approach for investing about $5 billion over the next several years in its General Fund Enterprise Business System, Global Combat Support System-Army Field/Tactical, and Logistics Modernization Program did not include alignment with the Army enterprise architecture or use of a portfolio-based business system investment review process. Moreover, we reported that the Army did not have reliable analyses, such as economic analyses, to support its management of these programs. We concluded that, until the Army adopts a business system investment management approach that provides for reviewing groups of systems and making enterprise decisions on how these groups will collectively interoperate to provide a desired capability, it runs the risk of investing significant resources in business systems that do not provide the desired functionality and efficiency. Accordingly, we made recommendations aimed at improving the department’s efforts to achieve total asset visibility and enhancing its efforts to improve its control and accountability over business system investments. The department agreed with our recommendations.

We also reported that DON had not, among other things, economically justified its ongoing and planned investment in the Naval Tactical Command Support System (NTCSS) and had not invested in NTCSS within the context of a well-defined DOD or DON enterprise architecture. In addition, we reported that DON had not effectively performed key measurement, reporting, budgeting, and oversight activities and had not adequately conducted requirements management and testing activities. We concluded that, without this information, DON could not determine whether NTCSS, as defined, and as being developed, is the right solution to meet its strategic business and technological needs. Accordingly, we recommended that the department develop the analytical basis to determine if continued investment in NTCSS represents prudent use of limited resources and to strengthen management of the program, conditional upon a decision to proceed with further investment in the program. The department largely agreed with our recommendations.

In addition, we reported that the Army had not defined and developed its Transportation Coordinators’ Automated Information for Movements System II (TC-AIMS II)—a joint services system with the goal of helping to manage the movement of forces and equipment within the United States and abroad— in the context of a DOD enterprise architecture. In addition, we reported that the Army had not economically justified the program on the basis of reliable estimates of life cycle costs and benefits and had not effectively implemented risk management. As a result, we concluded that the Army did not know if its investment in TC-AIMS II, as planned, was warranted or represented a prudent use of limited DOD resources. Accordingly, we recommended that the department, among other things, develop the analytical basis needed to determine if continued investment in TC-AIMS II represents prudent use of limited defense resources. In response, the department agreed with our recommendations and has since reduced the program’s scope by canceling future investments.

Furthermore, in 2005, we reported that DON had invested approximately $1 billion in the four previously cited ERP pilots without marked improvement in its day-to-day operations. More specifically, we reported that the program office had not implemented an EVM system. We also identified significant challenges and risks as the project moved forward, such as developing and implementing system interfaces, converting data from legacy systems into the ERP system, meeting its estimated completion date of 2011 at an estimated cost of $800 million, and achieving alignment with DOD’s BEA. To address these areas, we made recommendations that DOD improve oversight of Navy ERP, including developing quantitative metrics to evaluate the program. DOD generally agreed with our recommendations.

Implementation of Key DOD and Related IT Acquisition Management Controls on Navy ERP Varies

DOD IT-related acquisition policies and guidance, along with other relevant guidance, provide an acquisition management control framework within which to manage business system programs like Navy ERP. Effective implementation of this framework can minimize program risks and better ensure that system investments are defined in a way to optimally support mission operations and performance, as well as deliver promised system capabilities and benefits on time and within budget. To varying degrees of effectiveness, Navy ERP has been managed in accordance with aspects of this framework. However, implementation of key management controls has not been effective. Specifically, compliance with DOD’s federated BEA has not been sufficiently investment in the program has been economically justified on the basis of expected life cycle benefits that will likely exceed estimated life cycle costs, although some estimating limitations nevertheless exist; earned value management has not been effectively implemented; an important requirements management activity has been effectively a risk management process has been defined, but not effectively implemented for all risks.

The reasons that program management and oversight officials cited for why these key practices have not been sufficiently executed range from limitations in the applicable DOD guidance and tools to the complexity and challenges of managing and implementing a program of this size. Each of these reasons is described in the applicable sections of this report. By not effectively implementing all the above key IT acquisition management functions, the program is at increased risk of (1) not being defined in a way that best meets corporate mission needs and enhances performance and (2) adding to the more than 2 years in program schedule delays and about $570 million in program cost increases experienced to date.

Navy ERP Compliance with DOD’s Federated BEA Has Not Been Sufficiently Demonstrated

DOD and other guidance, recognize the importance of investing in business systems within the context of an enterprise architecture. Moreover, the Fiscal Year 2005 Defense Authorization Act requires that defense business systems be compliant with DOD’s federated BEA. Our research and experience in reviewing federal agencies show that not making investments within the context of a well-defined enterprise architecture often results in systems that are duplicative, are not well integrated, are unnecessarily costly to interface and maintain, and do not optimally support mission outcomes.

To its credit, the program office has followed DOD’s BEA compliance guidance. However, this guidance does not adequately provide for addressing all relevant aspects of BEA compliance. Moreover, DON’s enterprise architecture, which is a major component of DOD’s federated BEA, as well as key aspects of DOD’s corporate BEA, have yet to be sufficiently defined to permit thorough compliance determinations. In addition, current policies and guidance do not require DON investments to comply with its enterprise architecture. This means that the department does not have a sufficient basis for knowing if Navy ERP has been defined to minimize overlap with and duplication of other programs’ functionality and maximize interoperability among related programs. Each of these architecture alignment limitations is discussed here: The program’s compliance assessments did not include all relevant architecture products. In particular, the program did not assess compliance with the BEA’s technical standards profile, which outlines, for example, the standards governing how systems physically communicate with other systems and how they secure data from unauthorized access. This is particularly important because systems like Navy ERP need to share information with other systems and, for these systems to accomplish this effectively and efficiently, they need to employ common standards. A case in point is the relationship between Navy ERP and the Global Combat Support System—Marine Corps (GCSS-MC) program. Specifically, Navy ERP has identified 25 technical standards that are not in the BEA technical standards profile, and GCSS-MC has identified 13 technical standards that are not in the profile. Among these non-BEA standards are program-unique information sharing protocols, which could limit information sharing between Navy ERP and GCSS-MC, and with other systems.

In addition, the program office did not assess compliance with the BEA products that describe system-level characteristics. This is important because doing so would create a body of information about programs that could be used to identify common system components and services that could potentially be shared by the programs, thus avoiding wasteful duplication. For example, our analysis of Navy ERP program documentation shows that it contains system functions related to receiving goods, taking physical inventories, and returning goods, which are also system functions cited by the GCSS-MC program. However, because compliance with the BEA system products was not assessed, the extent to which these functions are potentially duplicative was not considered.

Furthermore, the program office did not assess compliance with BEA system products that describe data exchanges among systems. As we previously reported, establishing and using standard system interfaces is a critical enabler to sharing data. For example, Navy ERP program documentation indicates that it is to exchange inventory order and status data with other systems. System interfaces are important for understanding how information is to be exchanged between systems. However, since the program was not assessed for compliance with these products, it does not have the basis for understanding how its approach to exchanging information differs from that of other systems that it is to interface with. Compliance against each of these BEA products was not assessed because DOD’s compliance guidance does not provide for doing so and, according to BTA officials, because some BEA system products are not sufficiently defined. According to these officials, BTA plans to continue to define these products as the BEA evolves.

The compliance assessment was not used to identify potential areas of duplication across programs, which DOD has stated is an explicit goal of its federated BEA and associated investment review and decision-making processes. More specifically, even though the compliance guidance provides for assessing programs’ compliance with the BEA product that defines DOD operational activities, and Navy ERP was assessed for compliance with this product, the results were not used to identify programs that support the same operational activities and related business processes. Given that the federated BEA is intended to identify and avoid not only duplications within DOD components, but also between components, it is important that such commonality be addressed. For example, BEA compliance assessments for Navy ERP and GCSS-MC, as well as two Air Force programs (Defense Enterprise Accounting and Management System—Air Force and the Air Force Expeditionary Combat Support System) show that each program supports at least six of the same BEA operational activities (e.g., conducting physical inventory, delivering property and services) and three of these four programs support at least 18 additional operational activities (e.g., performing budgeting, managing receipt and acceptance). However, since the potential overlap among these and other programs was not assessed, these programs may be investing in duplicative functionality. Reasons for this were that the compliance guidance does not provide for such analyses to be conducted and programs have not been granted access rights to use this functionality in the compliance tool.

The program’s compliance assessment did not address compliance against DON’s enterprise architecture, which is one of the biggest members of the federated BEA. This is particularly important given that DOD’s approach to fully satisfying the architecture requirements of the Fiscal Year 2005 Defense Authorization Act is to develop and use a federated architecture in which component architectures are to provide the additional details needed to supplement the thin layer of corporate policies, rules, and standards included in the corporate BEA. As we recently reported, DON’s enterprise architecture is not mature because, among other things, it is missing a sufficient description of its current and future environments in terms of business and information/data. However, certain aspects of an architecture nevertheless exist and, according to DON CIO officials, these aspects will be leveraged in its efforts to develop a complete enterprise architecture. For example, the FORCEnet architecture is intended to document Navy’s technical infrastructure. Therefore, opportunities exist for DON to assess its programs in relation to these architecture products, and to understand where its programs are exposed to risks because products do not exist, are not mature, or at odds with other Navy programs. According to DOD officials, compliance with the DON architecture was not assessed because DOD compliance policy is limited to compliance with the corporate BEA, and a number of aspects of the DON enterprise architecture have yet to be sufficiently developed.

The program’s compliance assessment was not validated by DON or DOD investment oversight and decision-making authorities. More specifically, neither the DOD IRBs nor the DBSMC, nor the BTA in supporting both of these investment oversight and decision-making authorities, reviewed the program’s assessments. According to BTA officials, under DOD’s tiered approach to investment accountability, these entities are not responsible for validating programs’ compliance assessments. Rather, this is a component responsibility, and thus they rely on the military departments and defense agencies to validate the assessments.

However, DON Office of the CIO, which is responsible for precertifying investments as compliant before they are reviewed by the IRB, did not validate any of the program’s compliance assessments. According to Office of the CIO officials, they rely on Functional Area Managers to validate a program’s compliance assessments. However, no DON policy or guidance exists that describes how the Functional Area Managers should conduct such validations. CIO officials stated that this is because these authorities do not have the resources that they need to validate the assessments, and because a number of aspects of the DON architecture are not yet sufficiently developed.

Validation of program assessments is further complicated by the absence of information captured in the assessment tool about what program documentation or other source materials were used by the program office in making its compliance determinations. Specifically, the tool is only configured, and thus was only used, to capture the results of a program’s comparison of program architecture products to BEA products. Thus, it was not used to capture the system products used in making these determinations.

The limitations in existing BEA compliance-related policy and guidance, the supporting compliance assessment tool, and the federated BEA, put programs like Navy ERP at increased risk of being defined and implemented in a way that does not sufficiently ensure interoperability and avoid duplication and overlap. We recently completed a review examining multiple programs’ compliance with the federated BEA, including Navy ERP, for the Senate Armed Services Committee, Subcommittee on Readiness and Management Support. We addressed the architectural compliance guidance, tool, and validation limitations as part of this review.

Investment in Navy ERP Has Been Economically Justified, but Important Estimating Practices Were Not Implemented

The investment in Navy ERP has been economically justified on the basis of expected life cycle benefits that far exceed estimated life cycle costs. According to the program’s benefit/cost analysis, Navy ERP will produce about $8.6 billion in estimated benefits for an estimated cost of about $2.4 billion over its 20-year life cycle. While these benefit estimates were not subject to any analysis of how uncertainty in assumptions and data could impact the estimates, as called for by relevant guidance, our examination of key uncertainty variables, such as the timing of legacy systems’ retirement, showed that the savings impact would be relatively minor. However, the reliability of the cost estimate is limited because it was derived using several, but not all, key estimating practices. For example, the estimate was not grounded in a historical record of comparable data from similar programs and was not based on a reliable schedule baseline, which are both necessary to having a cost estimate that can be considered credible and accurate. These practices were not employed for various reasons, including DOD’s lack of historical data from similar programs and the lack of an integrated master schedule for the program that includes all releases.

Notwithstanding the fact that these limitations could materially increase the $2.4 billion cost estimate, it is nevertheless unlikely that these factors would increase the estimate to a level approaching the program’s benefit expectations. Therefore, we have no reason to believe that Navy ERP will not produce a positive return on investment.

Benefit Estimates Are Sufficiently Reliable Despite Absence of Uncertainty Analysis

Forecasting expected benefits over the life of a program is a key aspect of economically justifying an investment. The Office of Management and Budget (OMB) guidance advocates economically justifying investments on the basis of expected benefits, costs, and risks. Since estimates of benefits can be uncertain because of the imprecision in both the underlying data and modeling assumptions used, the guidance also provides for analyzing and reporting the effects of this uncertainty. By doing this, informed investment decision making can occur through the life of the program, and a baseline can be established against which to compare the accrual of actual benefits from deployed system capabilities.

The most recent economic analysis, dated August 2007, includes monetized benefit estimates for fiscal years 2004–2023, in three key areas—about $2.7 billion in legacy system cost savings, $3.3 billion in cost savings from inventory reductions, and $2.7 billion in cost savings from labor productivity improvements. Collectively, these benefits total about $8.6 billion.

The program office calculated expected benefits in terms of cost savings, which is consistent with established practices and guidance. For example, the program is to result in the retirement of 138 legacy systems (including the 4 pilot systems) between fiscal years 2005 and 2015, and the yearly maintenance costs for a single system are expected to be as high as about $39 million. According to relevant guidance, cost saving estimates should also be analyzed in terms of how uncertainty in assumptions and data could impact them. However, the program office did not perform such uncertainty analysis. According to program officials, uncertainty analysis is not warranted because they have taken and continue to take steps to validate the assumptions and the data, such as using the latest budget data associated with the legacy systems, and monitoring changes to the systems’ retirement dates. While these steps are positive, they do not eliminate the need for uncertainty analysis. Accordingly, we assessed key uncertainty variables, such as the timing of the legacy systems’ retirement, and found that the retirement dates of some of these systems have changed since the estimate was prepared, due to, among other things, schedule delays in the program. While the inherent uncertainty in these dates would reduce expected savings (e.g., only $11 million based on the 134 legacy systems that we examined), the reduction would be small relative to a total benefit estimate of $8.6 billion.

The Cost Estimate Was Not Reliably Derived

A reliable cost estimate is a key variable in calculating return on investment, and it provides the basis for informed investment decision making, realistic budget formulation and program resourcing, meaningful progress measurement, proactive course correction, and accountability for results. According to OMB, programs must maintain current and well- documented cost estimates, and these estimates must encompass the full life cycle of the program. OMB states that generating reliable cost estimates is a critical function necessary to support OMB’s capital programming process. Without reliable estimates, programs are at increased risk of experiencing cost overruns, missed deadlines, and performance shortfalls.

Our research has identified a number of practices that are the basis of effective program cost estimating. We have issued guidance that associates these practices with four characteristics of a reliable cost estimate. Specifically, these four characteristics are as follows: Comprehensive: The cost estimates should include both government and contractor costs over the program’s full life cycle, from the inception of the program through design, development, deployment, and operation and maintenance to retirement. They should also provide an appropriate level of detail to ensure that cost elements are neither omitted nor double counted and include documentation of all cost-influencing ground rules and assumptions.

Well-documented: The cost estimates should have clearly defined purposes and be supported by documented descriptions of key program or system characteristics (e.g., relationships with other systems, performance parameters). Additionally, they should capture in writing such things as the source data used and their significance, the calculations performed and their results, and the rationale for choosing a particular estimating method or reference. Moreover, this information should be captured in such a way that the data used to derive the estimate can be traced back to, and verified against, their sources. The final cost estimate should be reviewed and accepted by management on the basis of confidence in the estimating process and the estimate produced by the process.

Accurate: The cost estimates should provide for results that are unbiased and should not be overly conservative or optimistic (i.e., they should represent most likely costs). In addition, the estimates should be updated regularly to reflect material changes in the program, and steps should be taken to minimize mathematical mistakes and their significance. Among other things, the estimate should be grounded in a historical record of cost estimating and actual experiences on comparable programs.

Credible: The cost estimates should discuss any limitations in the analysis performed due to uncertainty or biases surrounding data or assumptions. Further, the estimates’ derivation should provide for varying any major assumptions and recalculating outcomes based on sensitivity analyses, and their associated risks and inherent uncertainty should be disclosed. Also, the estimates should be verified based on cross-checks using other estimating methods and by comparing the results with independent cost estimates.

The $2.4 billion life cycle cost estimate for Navy ERP reflects many of the practices associated with a reliable cost estimate, including all practices associated with being comprehensive and well-documented, and several related to being accurate and credible (see table 5). However, several important practices related to accuracy and credibility were not performed. To be reliable, a cost estimate should be comprehensive, well- documented, accurate, and credible.

The cost estimate is comprehensive because it includes both the government and contractor costs specific to development, acquisition (nondevelopment), implementation, and operations and support over the program’s 20-year life cycle. Moreover, the estimate clearly describes how the various subelements are aggregated to produce amounts for each cost category, thereby ensuring that all pertinent costs are included, and no costs are double counted. Finally, cost-influencing ground rules and assumptions, such as the program’s schedule, labor rates, and inflation rates are documented.

The cost estimate is also well-documented in that the purpose of the cost estimate is clearly defined, and the technical baseline includes, among other things, the hardware components and planned performance parameters. Furthermore, the calculations and results used to derive the estimate are documented, including descriptions of the methodologies used and evidence of traceability back to source data (e.g., vendor quotes, salary tables). Also, the cost estimate was reviewed by the Naval Center for Cost Analysis and the Office of the Secretary of Defense, Director for Program Analysis and Evaluation, which adds a level of confidence in the estimating process and the estimate produced.

However, the estimate lacks accuracy because not all important practices related to this characteristic were performed. Specifically, while the estimate is grounded in documented assumptions (e.g., hardware refreshment every 5 years) and periodically updated to reflect changes to the program, it is not adequately grounded in historical experience with comparable programs. While the program office did leverage historical cost data from the Navy ERP pilot programs, program officials told us that the level of cost accounting on these programs did not provide sufficient data. As stated in our guide, estimates should be based on historical records of cost and schedule estimates from comparable programs, and such historical data should be maintained and used for evaluation purposes and future estimates on comparable programs. The importance of doing so is evident by the fact that Navy ERP’s cost estimate has increased by about $570 million since fiscal year 2003, which program officials attributed to, among other things, site implementation costs (e.g., training and converting legacy system data) not included in the original cost estimate, schedule delays, and the lack of historical data from similar ERP programs.

This lack of cost data for large-scale ERP programs is, in part, due to DOD not having a standardized cost element structure for these programs that can be used for capturing actual cost data, which is a prerequisite to capturing and maintaining the kind of historical data that can inform cost estimates on similar programs. This means that programs like Navy ERP will not be able to ground their cost estimates in actual costs from comparable programs. According to officials with the Defense Cost and Resource Center, such cost element structures are needed, along with a requirement for programs to report on their costs, but approval and resources have yet to be gained for either these structures or the reporting of their costs. We recently completed work that addressed standardization of DOD’s ERP cost element structure and maintenance of a database for historical ERP cost data for use on ERP programs.

Compounding the estimate’s limited accuracy are limitations in its credibility. Specifically, while the estimate satisfies some of the key practices for a credible cost estimate (e.g., confirming key cost drivers, performing sensitivity analyses, and having an independent cost estimate prepared by the Naval Center for Cost Analysis that was within 11 percent of the program’s estimate), the program lacks a reliable schedule baseline, which is a key component of a reliable cost estimate because it serves as the basis for future work to be performed. Other factors that limit confidence in the cost estimate’s accuracy are (1) past increases in the program’s cost estimate (as discussed earlier) and (2) trends in EVM data (as discussed later). Taken together, the program’s cost estimate is not sufficiently credible and accurate and thus not reliable.

While important cost estimating practices were not implemented, it is nevertheless unlikely that these limitations would materially increase the $2.4 billion cost estimate to a level approaching the program’s $8.6 billion benefit expectations.

Earned Value Management Has Not Been Effectively Implemented

Measuring and reporting progress against cost and schedule commitments (i.e., baselines) is a vital element of effective program management. EVM provides a proven means for measuring such progress and thereby identifying potential cost overruns and schedule delays early, when their impact can be minimized.

To its credit, the program has elected to implement program-level EVM, which is a best practice that has rarely been implemented in the federal government. In doing so, however, basic EVM activities have not been executed. In particular, an integrated baseline review, which is to verify that the program’s cost and schedule are reasonable given the program’s scope of work and associated risks, has not been performed. Moreover, other accepted industry standards have not been sufficiently implemented, and surveillance of EVM implementation by an entity independent of the program office has not occurred. Not performing these important practices has contributed to the cost overruns and lengthy schedule delays already experienced on Release 1.0, and they will likely result in more. In fact, our analysis of the latest estimate to complete just the budgeted development work for all three releases, which is about $844 million, shows that this estimate will most likely be exceeded by about $152 million.

The Program Has Elected to Implement Program-Level EVM

As we previously reported, EVM offers many benefits when done properly. In particular, it allows performance to be measured, and it serves as an early warning system for deviations from plans. It, therefore, enables a program office to mitigate the risks of cost and schedule overruns. OMB policy recognizes the use of EVM as an important part of program management and decision making.

Implementing EVM at the program level rather than just the contract level is considered a best practice, and OMB recently began requiring it to measure how well a program’s approved cost, schedule, and performance goals are being met. According to OMB, integrating government and contractor cost, schedule, and performance status should result in better program execution through more effective management. In addition, integrated EVM data can be used to better justify budget requests.

To minimize the risk associated with its decision to transition responsibility for Navy ERP system integration from the contractor to the government and to improve cost and schedule performance, the program office elected in October 2006 to perform EVM at the program level. We support the use of program-level EVM. However, if not implemented effectively, this program-level approach will be of little value.

An Integrated Baseline Review Has Not Been Performed

A fundamental aspect of effective EVM is the development of a performance measurement baseline (PMB), which represents the cumulative value of planned work and serves as the baseline against which variances are calculated. According to relevant best practice guidance, a PMB consists of a complete work breakdown structure, a complete integrated master schedule, and accurate budgets for all planned work.

To validate the PMB, an integrated baseline review is performed to obtain stakeholder agreement on the baseline. According to DOD guidance and best practices, such a review should be held within 6 months of a contract award and conducted on an as needed basis throughout the life of a program to ensure that the baseline reflects (1) all tasks in the statement of work, (2) adequate resources (staff and materials) to complete the tasks, and (3) integration of the tasks into a well-defined schedule. Further, the contract performance reports that are to be used to monitor performance against the PMB should be validated during the integrated baseline review.

The program office has satisfied some of the prerequisites for having a reliable PMB, such as developing a work breakdown structure and specifying the contract performance reports that are to be used to monitor performance. However, it has not conducted an integrated baseline review. Specifically, a review was not conducted for Release 1.0, even though the contract was finalized about 30 months ago (January 2006). Also, while the review for Release 1.1 was recently scheduled for August 2008, this is 8 months later than when such a review should be held, according to DOD guidance and best practices. This means that the reasonableness of the program’s scope and schedule relative to the program risks has not been assured, and has likely been, and will likely continue to be a primary contributor to future cost increases and schedule delays.

According to program officials, a review was not performed on the first release because development of this release was largely complete by the time the program office established the underlying capabilities needed to perform program-level EVM. In addition, program officials stated that an integrated baseline review has yet to be performed on the other two releases because their priority has been on deploying and stabilizing the first release. In our view, not assuring the validity of the PMB precludes effective implementation of EVM. Until a review is conducted, DOD will not have reasonable assurance that the program’s scope and schedule are achievable, and thus, additional cost and schedule overruns are likely.

Industry EVM Standards Have Not Been Fully Implemented and Independently Surveilled

In 1996, DOD adopted industry EVM guidance that identifies 32 essential practices organized into five categories: (1) organization; (2) planning, scheduling and budgeting; (3) accounting; (4) analysis and management reports; and (5) revisions and data maintenance. DOD requires that all programs’ implementation of EVM undergo a compliance audit against the 32 industry practices. In addition, DOD policy and guidance state that independent surveillance of EVM should occur over the life of the program to guarantee the validity of the performance data and ensure that EVM is being used effectively to manage cost, schedule, and technical performance.

On Navy ERP, compliance with the 32 accepted industry practices has not been verified, and surveillance of EVM by an independent entity has not occurred. Therefore, the program does not have the required basis for ensuring that EVM is being effectively implemented on Navy ERP. According to program officials, surveillance was performed by NAVAIR for Release 1.0. However, NAVAIR officials said that they did not perform such surveillance because they did not receive the Release 1.0 cost performance data needed to do so. Program officials also stated that DON’s Center for Earned Value Management has conducted an initial assessment of their EVM management system, and that they intend to have the Center perform surveillance. However, they did not have a plan for accomplishing this. Until compliance with the standards is verified and continuous surveillance occurs, and deviations are addressed, the program will likely continue to experience cost overruns and schedule delays.

Estimated Schedule Baseline Was Not Derived in Accordance with All Key Schedule Estimating Practices

The success of any program depends in part on having a reliable schedule of when the program’s work activities will occur, how long they will take, and how they are related to one another. As such, the schedule not only provides a road map for the systematic execution of a program but also provides the means by which to estimate costs, gauge progress, identify and address potential problems, and promote accountability. Our research has identified nine practices associated with effective schedule estimating. These practices are (1) capturing key activities, (2) sequencing key activities, (3) establishing the duration of key activities, (4) assigning resources to key activities, (5) integrating key activities horizontally and vertically, (6) establishing the critical path for key activities, (7) identifying “float time” between key activities, (8) distributing reserves to high-risk activities, and (9) performing a schedule risk analysis.

The program’s estimated schedule was developed using some of these practices, but several key practices were not fully employed that are fundamental to having a schedule that provides a sufficiently reliable basis for estimating costs, measuring progress and forecasting slippages. On the positive side, the schedule for the first two releases captures key activities and their durations and is integrated horizontally and vertically, meaning that multiple teams executing different aspects of the program can effectively work to the same master schedule. Moreover, for these two releases, the program has established float time between key activities and distributed schedule reserve to high-risk activities. However, the program has not adequately sequenced and assigned resources to key program activities. Moreover, the estimated schedule for the first increment is not grounded in an integrated master schedule of all the releases, and thus the schedule for this increment does not reflect the program’s critical path of work that must be performed to achieve the target completion date. Also, it does not reflect the results of a schedule-risk analysis across all three releases with schedule reserve allocated to high-risk activities because such risks were not examined. See table 6 for the results of our analyses relative to each of the nine practices.

According to program documentation, they have plans to address the logical sequencing of activities (to ensure that it reflects how work is to be performed), but program officials stated that they do not plan to combine all three releases into a single integrated master schedule for the entire first increment of the program because doing so would produce an overly complex and nonexecutable schedule involving as many as 15,000 activities. However, our research of and experience in evaluating major programs’ use of EVM and integrated master schedules show that while large, complex programs necessitate schedules involving thousands of activities, successful programs ensure that their schedules integrate these activities. In our view, not adequately performing these practices does not allow the program to effectively assign resources, identify the critical path, and perform a schedule risk analysis that would allow it to understand, disclose, and compensate for its schedule risks. This means that the program is not well-positioned to understand progress and forecast its impact. To illustrate, the program recently experienced delays in deploying its first release at NAVAIR, which according to a recent operational test and evaluation report has significantly affected the schedule’s critical path. These schedule impacts are because resources supporting the deployment at NAVAIR began to shift to the next scheduled deployment site and thus are no longer available to resolve critical issues at NAVAIR. Since the schedule baseline is not integrated across all releases, the impact of this delay on other releases, and thus the program as a whole, cannot be readily determined.

Trends in EVM Data Show Pattern of Cost Overruns and Schedule Slippages

Program data show a pattern of actual cost overruns and schedule delays between January 2007 and May 2008. Moreover, our analysis of the data supports a most likely program cost growth of about $152 million to complete all three releases.

Differences from the PMB are measured in both cost and schedule variances. Positive variances indicate that activities are costing less or are completed ahead of schedule. Negative variances indicate that activities are costing more or are falling behind schedule. These cost and schedule variances can then be used in forecasting the cost and time needed to complete the program.

Based on program-provided data for the first increment over a 17-month period ending May 2008, the program has experienced negative cost variances. Specifically, while these cost variances have fluctuated during this period, they have consistently been negative. (See fig. 4.) Moreover, our analysis of the cost to complete just the budgeted development work (also known as the PMB) for all three releases, which is about $844 million, will be exceeded by between about $102 million and $316 million, with a most likely overrun of about $152 million. In contrast, the program office reports that the overrun at completion will be $55 million but has yet to provide us with documentation supporting this calculation. Moreover, our calculation does not reflect the recent problems discovered during the operational test and evaluation at NAVAIR and thus the overrun is likely to be higher.

During this same 17-month period, the program has experienced negative schedule variances and, since January 2008, they have almost doubled each month. Further, as of May 2008, the program had not completed about $24 million in scheduled work. (See fig. 5.) An inability to meet schedule performance is a frequent indication of future cost increases, as more spending is often necessary to resolve schedule delays.

Because the program office has not performed important reliability checks, such as EVM validation and integrated baseline reviews, as discussed above, we cannot be certain that the PMB is reliable (i.e., reflects all the work to be done and has identified all the risks). As a result, the overrun that we are forecasting could be higher.

By not executing basic EVM practices, the program has and will likely continue to experience cost and schedule shortfalls. Until the program office implements these important EVM practices, it will likely not be able to track actual program costs and schedules close to estimates.

Important Requirements Management Activity Has Been Effectively Implemented

Well-defined and managed requirements are recognized by DOD guidance as essential, and they can be viewed as a cornerstone of effective system acquisition. One aspect of effective requirements management is requirements traceability. By tracing requirements both backward from system requirements to higher level business or operational requirements and forward to system design specifications and test plans, the chances of the deployed product satisfying requirements is increased, and the ability to understand the impact of any requirement changes and thus make informed decisions about such changes, is enhanced.

The program office is effectively implementing requirements traceability for its 1,733 Release 1.0 system requirements. To verify this traceability, we randomly selected and analyzed 60 of the 1,733 system requirements and confirmed that 58 of the 60 were traceable both backward to higher level requirements and forward to design specifications and test results. The remaining 2 had been allocated to the other releases, and thus we also confirmed the program’s ability to maintain traceability between product releases. In doing so, the program utilized a tool called DOORS, which if implemented properly, allows each requirement to be linked from its most conceptual definition to its most detailed definition, as well as to design specifications and test cases. In effect, the tool maintains the linkages among requirement documents, design documents, and test cases even if requirements change.

If DON continues to effectively implement requirements traceability, it will increase the chances that system requirements will be met by the deployed system.

A Risk Management Process Has Been Defined, but All Risks Have Not Been Effectively Mitigated

Proactively managing program risks is a key acquisition management control and, if defined and implemented properly, it can increase the chances of programs delivering promised capabilities and benefits on time and within budget. To the program office’s credit, it has defined a risk management process that meets relevant guidance. However, it has not effectively implemented the process for all identified risks. As a result, these risks have not been proactively mitigated and either have contributed to cost and schedule shortfalls, or could potentially contribute to such shortfalls.

DOD acquisition management guidance, as well as other relevant guidance advocates identifying facts and circumstances that can increase the probability of an acquisition’s failing to meet cost, schedule, and performance commitments and then taking steps to reduce the probability of their occurrence and impact. In brief, effective risk management consists of: (1) establishing a written plan for managing risks; (2) designating responsibility for risk management activities; (3) encouraging program-wide participation in the identification and mitigation of risks; (4) defining and implementing a process that provides for the identification, analysis, and mitigation of risks; and (5) examining the status of identified risks in program milestone reviews.

The program office has developed a written plan for managing risks and established a process that together provide for the above-cited risk management practices. Moreover, it has largely followed its plan and process as per the following examples: The program manager has been assigned overall responsibility for managing risks and serves as the chair of the risk management board. Also, a functional team lead (i.e., subject matter expert) is assigned responsibility for analyzing and mitigating each identified risk.

Program-wide participation in the identification, analysis, and mitigation of risks is encouraged. Specifically, a manager for each release is responsible for providing risk management guidance to the staff, which includes staff identification and analysis of risks. Also, according to the program office’s risk management plan, all program personnel can submit a risk for approval. In addition, stakeholders participate in risk management activities during acquisition milestone reviews.

The program office has identified and categorized individual risks. As of June 2008, the risk database contained 15 active risks—3 high, 8 medium, and 4 low.

Program risks are considered during program milestone reviews. For example, during the program’s critical design review, which is a key event of the system development and demonstration phase, key risks regarding implementing new business processes and legacy system changes were discussed. Furthermore, the program manager receives a monthly risk report that describes the status of program risks.

However, the program office has not consistently followed other aspects of its process. In particular, it has not effectively implemented steps for mitigating the risks associated with (1) converting data from NAVAIR’s legacy systems to run on Navy ERP and (2) positioning NAVAIR for adopting the new business processes embedded in Navy ERP. As we have previously reported, it is important for organizations that are to operate and use commercial off-the-shelf software products, such as Navy ERP, to proactively manage and position themselves for the organizational impact of introducing functionality embedded in the commercial products. If they do not, the organization’s performance will suffer.

To the program office’s credit, it identified numerous risks associated with data conversion and organizational change management and developed and implemented strategies that were intended to mitigate these risks. However, it closed these risks even though they were never effectively mitigated, as evidenced by the results of recently completed DON operational test and evaluation. According to the June 2008 operational test and evaluation report for NAVAIR, significant problems relating to both legacy system data conversion and adoption of new business processes were experienced. The report states that these problems have contributed to increases in the costs to operate the system, including unexpected manual effort. It further states that these problems have rendered the deployed version not operationally effective and that deployment of the system to other sites should not occur until the change management process has been analyzed and improved. It also attributed the realization of the problems to the program office and NAVAIR not having adequately engaged and communicated early with each other to coordinate and resolve differences in organizational perspectives and priorities and provide intensive pre-deployment preparation and training. Program officials acknowledged these shortcomings and attributed them to their limited authority over the commands. In this regard, they have previously surfaced these risks with department oversight and approval authorities, but actions were not taken by these authorities that ensured that the risks were being effectively mitigated.

Beyond not effectively mitigating these risks, the program office has not ensured that all risks are captured in the risk inventory. For example, the inventory does not include the risks described in this report that are associated with not having adequately demonstrated the program’s alignment to the federated BEA and not having implemented program- level EVM in a manner that reflects industry practices. This means that these risks are not being disclosed or mitigated.

By not effectively addressing all risks associated with the program, these risks can and have become problems that contribute to cost and schedule shortfalls. Until all significant risks are proactively addressed, to include ensuring that all associated mitigation steps are implemented and that they accomplished their intended purpose, the program will likely experience further problems at subsequent deployment sites.

Conclusions

DOD’s success in delivering large-scale business systems, such as Navy ERP, is in large part determined by the extent to which it employs the kind of rigorous and disciplined IT management controls that are reflected in department policies and related guidance. While implementing these controls does not guarantee a successful program, it does minimize a program’s exposure to risk and thus the likelihood that it will fall short of expectations. In the case of Navy ERP, living up to expectations is important because the program is large, complex, and critical to addressing the department’s long-standing problems related to financial transparency and asset visibility.

The effectiveness to which key IT management controls have been implemented in Navy ERP varies, with one control and several aspects of others being effectively implemented, and others less so. Moreover, those controls that have not been effectively implemented have, in part, contributed to the sizable cost and schedule shortfalls experienced to date on the program. Unless this changes, more shortfalls can be expected.

While the program office is primarily responsible for ensuring that effective IT management controls are implemented, other oversight and stakeholder organizations share responsibility. For example, even though the program has not demonstrated its alignment with the federated BEA, it nevertheless followed established DOD architecture compliance guidance and used the related compliance assessment tool in assessing and asserting its compliance. The root cause for not demonstrating compliance thus is not traceable to the program office but rather is due to, among other things, the limitations of the compliance guidance and tool, and the program’s oversight entities not validating the compliance assessment and assertion. Also, the reason that the program’s cost estimate was not informed by the cost experiences of other programs of the same size and scope is because DOD does not have a standard ERP cost element structure and has not maintained a historical database of costs for like programs to use. In contrast, effective implementation of other management controls, such as implementing EVM, requirements traceability, and risk management is the responsibility of the program office.

All told, addressing the management control weaknesses requires the combined efforts of the various organizations that share responsibility for managing and overseeing the program. By doing so, the department can better assure itself that Navy ERP will optimally support its performance goals and will deliver promised capabilities and benefits on time and within budget.

Because we recently completed work that more broadly addresses the above cited architectural alignment and comparable program cost data limitations, we are not making recommendations in this report for addressing them.

Recommendations for Executive Action

To strengthen Navy ERP management control and better provide for the program’s success, we are making the following recommendations: To improve the reliability of Navy ERP benefit estimates and cost estimates, we recommend that the Secretary of Defense direct the Secretary of the Navy, through the appropriate chain of command, to ensure that future Navy ERP estimates include uncertainty analyses of estimated benefits, reflect the risks associated with not having cost data for comparable ERP programs, and are otherwise derived in full accordance with the other key estimating practices, and economic analysis practices discussed in this report.

To enhance Navy ERP’s use of EVM, we recommend that the Secretary of Defense direct the Secretary of the Navy, through the appropriate chain of command, to ensure that (1) an integrated baseline review on the last two releases of the first increment is conducted, (2) compliance against the 32 accepted industry EVM practices is verified, and (3) a plan to have an independent organization perform surveillance of the program’s EVM system is developed and implemented.

To increase the quality of the program’s integrated master schedule, we recommend that the Secretary of Defense direct the Secretary of the Navy, through the appropriate chain of command, to ensure that the schedule (1) includes the logical sequencing of all activities, (2) reflects whether all required resources will be available when needed, (3) defines a critical path that integrates all three releases, (4) allocates reserve for the high- risk activities on the entire program’s critical path, and (5) incorporates the results of a schedule risk analysis for all three releases and recalculates program cost and schedule variances to more accurately determine a most likely cost and schedule overrun.

To improve Navy ERP’s management of program risks, we recommend that the Secretary of Defense direct the Secretary of the Navy, through the appropriate chain of command, to ensure that (1) the plans for mitigating the risks associated with converting data from legacy systems to Navy ERP and positioning the commands for adopting the new business processes embedded in the Navy ERP are re-evaluated in light of the recent experience with NAVAIR and adjusted accordingly, (2) the status and results of these and other mitigation plans’ implementation are periodically reported to program oversight and approval authorities, (3) these authorities ensure that those entities responsible for implementing these strategies are held accountable for doing so, and (4) each of the risks discussed in this report are included in the program’s inventory of active risks and managed accordingly.

Agency Comments and Our Evaluation

In written comments on a draft of this report, signed by the Deputy Under Secretary of Defense (Business Transformation) and reprinted in appendix II, DOD stated that it concurred with two of our four recommendations and partially concurred with the remaining two. Further, it stated that it has taken steps to address some of our recommendations, adding that it is committed to implementing recommendations that contribute to the program’s success. The department’s comments relative to both of the recommendations that it partially concurred with, as well as additional comments, are discussed below.

For our recommendation associated with improving the program’s benefit and cost estimates, DOD concurred with two of the recommendation’s three parts, but it did not concur with one part—ensuring that future cost estimates reflect the risk of not having cost data for comparable programs. While acknowledging that the program had limited cost data from comparable programs on which to base its cost estimate, DOD stated that an uncertainty analysis had been applied to the estimate to account for the risk associated with not having such data. The department further stated that actual experience on the program will continue to be used to refine the program’s cost estimating methodology. While we support DOD’s stated commitment to using actual program cost experience in deriving future estimates, we do not agree that the latest estimate accounted for the risk of not having cost data from comparable programs. We examined the uncertainty analysis as part of our review, and found that it did not recognize this risk. Moreover, DOD’s comments offered no new evidence to the contrary.

For our recommendation associated with improving the program’s schedule estimating, DOD concurred with four of the recommendation’s five parts, and partially concurred with one part—ensuring that the schedule defines a critical path that integrates all releases. In taking this position, the department stated that a critical path has been established for each release rather than across all three releases, and it attributes this to the size and complexity of the program. We do not take issue with either of these statements, as they are already recognized in our report. However, DOD offers no new information in its comments. Further, our report also recognizes that to be successful, large and complex programs that involve thousands of activities need to ensure that their schedules integrate these activities. In this regard, we support the department’s commitment to explore the feasibility of implementing this part of our recommendation.

In addition, while stating that it concurred with all parts of our recommendation associated with improving the program’s use of EVM, DOD nevertheless provided additional comments as justification for having not conducted an integrated baseline review on Release 1.0. Specifically, it stated that when it rebaselined this release in December 2006, the release’s development activities were essentially complete and the release was in the latter stages of testing. Further, it stated that the risks associated with the Release 1.0 schedule were assessed 3 months after this rebaselining, and these risks were successfully mitigated. To support this statement, it said that Release 1.0 achieved its “Go-Live” as scheduled at NAVAIR. We do not agree with these comments for several reasons. First, at the time of the rebaselining, about 9 months of scheduled Release 1.0 development remained, and thus the release was far from complete. Moreover, the significance of the amount of work that remained, and still remains today on Release 1.0 is acknowledged in DOD’s own comment that the scheduled integrated baseline review for Release 1.1 will also include remaining Release 1.0 work. Second, the Release 1.0 contract was awarded in January 2006, and DOD’s own guidance requires that an integrated baseline review be conducted within 6 months of a contract’s award. Third, although DOD states that the program achieved “Go-Live” as scheduled on October 1, 2007, the program achieved initial operational capability 7 months later than established in the December 2006 baseline.

In addition to these comments, the department also described actions under way or planned to address our recommendations. We support the actions described, as they are consistent with the intent of our recommendations. If fully and properly implemented, these actions will go a long way in addressing the management control weaknesses that our recommendations are aimed at correcting.

We are sending copies of this report to interested congressional committees; the Director, Office of Management and Budget; the Congressional Budget Office; the Secretary of Defense; and the Department of Defense Office of the Inspector General. We also will make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3439 or hiter@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Objective, Scope, and Methodology

Our objective was to determine whether the Department of the Navy is effectively implementing information technology management controls on the Navy Enterprise Resource Planning (Navy ERP) program. To accomplish this, we focused on the first increment of Navy ERP and the following management areas (1) architectural alignment, (2) economic justification, (3) earned value management (EVM), (4) requirements management, and (5) risk management.

To determine whether Navy ERP was aligned with the Department of Defense’s (DOD) federated business enterprise architecture (BEA), we reviewed the program’s BEA compliance assessments and system architecture products, as well as Versions 4.0, 4.1, and 5.0 of the BEA, and compared them with the BEA compliance requirements described in the Fiscal Year 2005 Defense Authorization Act and DOD’s BEA compliance guidance, and we evaluated the extent to which the compliance assessments addressed all relevant BEA products. We also determined the extent to which the program-level architecture documentation supported the BEA compliance assessments. We obtained documentation, such as the BEA compliance assessments from the Navy ERP and Global Combat Support System—Marine Corps programs, as well as the Air Force’s Defense Enterprise Accounting and Management System and Air Force Expeditionary Combat Support System programs. We then compared these assessments to identify potential redundancies or opportunities for reuse and determined if the compliance assessments examined duplication across programs, and if the tool that supports these assessments is being used to identify such duplication. In doing so, we interviewed program officials and officials from the Department of the Navy, Office of the Chief Information Officer and reviewed recent GAO reports to determine the extent to which the programs were assessed for compliance against the Department of the Navy enterprise architecture. We also interviewed program officials and officials from the Business Transformation Agency and the Department of the Navy, including the logistics Functional Area Manager, and obtained guidance documentation from these officials to determine the extent to which the compliance assessments were subject to oversight or validation.

To determine whether the program had economically justified its investment in Navy ERP, we reviewed the latest economic analysis to determine the basis for the cost and benefit estimates. This included evaluating the analysis against Office of Management and Budget guidance and GAO’s Cost Assessment Guide. In doing so, we interviewed cognizant program officials, including the program manager and cost analysis team, regarding their respective roles, responsibilities, and actual efforts in developing and/or reviewing the economic analysis. We also interviewed officials at the Office of Program Analysis and Evaluation and Naval Center for Cost Analysis as to their respective roles, responsibilities, and actual efforts in developing and/or reviewing the economic analysis. We did not verify the validity of the source data used to calculate estimated benefits, such as those data used to determine the yearly costs associated with legacy systems planned for retirement.

To determine the extent to which the program had effectively implemented EVM, we reviewed relevant documentation, such as contract performance reports, acquisition program baselines, performance measurement baseline, and schedule estimates and compared them with DOD policies and guidance. To identify trends that could affect the program baseline in the future, we assessed cost and schedule performance and, in doing so, we applied earned value analysis techniques to data from contract performance reports. We compared the cost of work completed with the budgeted costs for scheduled work over a 17-month period, from January 2007 to May 2008, to show trends in cost and schedule performance. We also used data from the reports to estimate the likely costs at completion of the program through established earned value formulas. This resulted in three different values, with the middle value being the most likely. We checked EVM data to see if there were any mathematical errors or inconsistencies that would lead to the data being unreliable. We interviewed cognizant officials from the Naval Air Systems Command and program officials to determine whether the program had conducted an integrated baseline review, whether the EVM system had been validated against industry guidelines, and to better understand the anomalies in the EVM data and determine what outside surveillance was being done to ensure that the industry standards are being met. We also reviewed the program’s schedule estimates and compared them with relevant best practices to determine the extent to which they reflect key estimating practices that are fundamental to having a reliable schedule. In doing so, we interviewed cognizant program officials to discuss their use of best practices in creating the program’s current schedule.

To determine the extent to which the program has effectively implemented requirements management, we reviewed relevant program documentation, such as the program management plan and baseline list of requirements. To determine the extent to which the program has maintained traceability backward to high-level business operation requirements and system requirements, and forward to system design specifications, and test plans, we randomly selected 60 program requirements and traced them both backward and forward. This sample was designed with a 5 percent tolerable error rate at the 95 percent level of confidence so that, if we found 0 problems in our sample, we could conclude statistically that the error rate was less than 5 percent. In addition, we interviewed program officials involved in the requirements management process to discuss their roles and responsibilities for managing requirements.

To determine the extent to which the program implemented risk management, we reviewed relevant risk management documentation, such as the program’s risk management plan and risk database reports demonstrating the status of the program’s major risks and compared the program office’s activities with DOD acquisition management guidance and related industry practices. We also reviewed the program’s mitigation process with respect to key risks to determine the extent to which these risks were effectively managed. In doing so, we interviewed cognizant program officials, such as the program manager and risk manager, to discuss their roles and responsibilities and obtain clarification on the program’s approach to managing risks associated with acquiring and implementing Navy ERP.

We conducted this performance audit at DOD offices in the Washington, D.C., metropolitan area and Annapolis, Md., from June 2007 to September 2008, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, key contributors to this report were Neelaxi Lakhmani, Assistant Director; Monica Anatalio; Harold Brumm; Neil Doherty; Cheryl Dottermusch; Nancy Glover; Mustafa Hassan; Michael Holland; Ethan Iczkovitz; Anh Le; Josh Leiling; Emily Longcore; Lee McCracken; Madhav Panwar; Karen Richey; Melissa Schermerhorn; Karl Seifert; Sushmita Srikanth; Jonathan Ticehurst; and Adam Vodraska.