DOD Business Systems Modernization: Important Management Controls Being Implemented on Major Navy Program, but Improvements Needed in Key Areas
Published: 2008-09-08T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) has long been challenged in implementing key information technology (IT) management controls on its thousands of business system investments. For this and other reasons, GAO has designated DOD's business systems modernization efforts as high-risk. One of the larger business system investments is the Department of the Navy's Enterprise Resource Planning (Navy ERP) program. Initiated in 2003, the program is to standardize the Navy's business processes, such as acquisition and financial management. It is being delivered in increments, the first of which is to cost about $2.4 billion over its useful life and be fully deployed in fiscal year 2013. GAO was asked to determine whether key IT management controls are being implemented on the program. To do this, GAO analyzed, for example, requirements management, economic justification, earned value management, and risk management.
section_title: What GAO Found
paragraphs: DOD has implemented key IT management controls on its Navy ERP program to varying degrees of effectiveness. To its credit, the control associated with managing system requirements is being effectively implemented. In addition, important aspects of other controls have at least been partially implemented, including those associated with economically justifying investment in the program and proactively managing program risks. Nevertheless, other aspects of these controls, as well as the bulk of what is needed to effectively implement earned value management, which is a recognized means for measuring program progress, have not been effectively implemented. Among other things, these control weaknesses have contributed to the more than 2-year schedule delay and the almost $600 million cost overruns already experienced on the program since it began, and they will likely contribute to future delays and overruns if they are not corrected. Examples of the weaknesses are: (1) Investment in the program has been economically justified on the basis of expected benefits that far exceed estimated costs ($8.6 billion versus $2.4 billion over a 20-year life cycle). However, important estimating practices, such as using historical cost data from comparable programs and basing the cost estimate on a reliable schedule baseline were not employed. While these weaknesses are important because they limit the reliability of the estimates, GAO's analysis shows that they would not have altered the estimates to the point of not producing a positive return on investment. (2) Earned value management has not been effectively implemented. To its credit, the program office has elected to implement program-level earned value management. In doing so, however, basic prerequisites for effectively managing earned value have not been executed. In particular, the integrated master schedule was not derived in accordance with key estimating practices, and an integrated baseline review has not been performed on any of the first increment's releases. (3) A defined process for proactively avoiding problems, referred to as risk management, has been established, but risk mitigation strategies have not been effectively implemented for all significant risks, such as those associated with data conversion and organizational change management, as well the risks associated with the above-cited weaknesses. The reasons that program management and oversight officials cited for these practices not being executed range from the complexity and challenges of managing and implementing a program of this size to limitations in the program office's scope and authority. Notwithstanding the effectiveness with which important aspects of several controls have been implemented, the above-cited weaknesses put DOD at risk of investing in a system solution that does not optimally support corporate mission needs and mission performance, and meet schedule and cost commitments.

section_title: Letter
paragraphs: For decades, the Department of Defense (DOD) has been challenged in  modernizing its timeworn business systems. In 1995, we designated DOD’s  business systems modernization program as high risk, and continue to do  so today. Our reasons include the modernization’s large size, complexity,  and its critical role in addressing other long-standing transformation and  financial management challenges. Other reasons are that DOD has yet to  institutionalize key system modernization management controls, and it has  not demonstrated the ability to consistently deliver promised system  capabilities and benefits on time and within budget.
Nevertheless, DOD continues to invest billions of dollars in thousands of  business systems, including about a hundred that the department has  labeled as business transformational programs, 12 of which account for  about 50 percent of these programs’ costs. The Navy Enterprise Resource  Planning (Navy ERP) program is one such program. Initiated in 2003, Navy  ERP is to standardize the Navy’s acquisition, financial, program  management, maintenance, plant and wholesale supply, and workforce  management business processes across its dispersed organizational  environment. As envisioned, the program consists of a series of major  increments, the first of which includes three releases and is expected to  cost approximately $2.4 billion over its 20-year life cycle and to be fully  operational in fiscal year 2013.
As agreed, our objective was to determine whether the Department of the  Navy (DON) is effectively implementing information technology (IT)  management controls on Navy ERP. To accomplish this, we focused on  the program’s first increment by analyzing a range of program  documentation and interviewing cognizant officials relative to the  following management areas: architectural alignment, economic  justification, earned value management (EVM), requirements management,  and risk management. We conducted this performance audit from June  2007 to September 2008, in accordance with generally accepted  government auditing standards. Those standards require that we plan and  perform the audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objective. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objective.  Additional details on our objective, scope, and methodology are in  appendix I.
subsections: section_title: Background
paragraphs: DON’s primary mission is to organize, train, maintain, and equip combat- ready naval forces capable of winning the global war on terrorism and any  other armed conflict, deterring aggression by would-be foes, preserving  freedom of the seas, and promoting peace and security. To support this  mission, DON performs a variety of interrelated and interdependent  business functions (e.g., acquisition and financial management), relying  heavily on IT systems. In fiscal year 2008, DON’s budget for business  systems and associated infrastructure was about $2.7 billion, of which $2.2  billion was allocated to operations and maintenance of existing systems  and the remaining $500 million to systems in development and  modernization. Of the approximately 3,000 business systems that DOD  reports in its current inventory, DON accounts for 904, or about 30  percent, of the total. Navy ERP is one such system investment.
subsections: section_title: Navy ERP: A Brief Description
paragraphs: In July 2003, the Assistant Secretary of the Navy for Research,  Development, and Acquisition established Navy ERP to “converge” four  separate pilot programs that were under way at four separate Navy  commands. This program is to leverage a commercial, off-the-shelf  software known as an enterprise resource planning product. Such  products consist of multiple, integrated functional modules that perform a  variety of business-related tasks, such as acquisition and financial  management. Table 1 provides a brief description and status of each of the  pilots.
According to DOD, Navy ERP is to address the Navy’s long-standing  problems related to financial transparency and asset visibility. Specifically,  the program is intended to standardize the Navy’s acquisition, financial,  program management, maintenance, plant and wholesale supply, and  workforce management business processes across its dispersed  organizational components. When the program is fully implemented, it is  to support over 86,000 users.
Navy ERP is being developed in a series of increments using the Systems  Applications and Products (SAP) commercial software package,  augmented as needed by customized software. SAP consists of multiple,  integrated functional modules that perform a variety of business related  tasks, such as finance and acquisition. The first increment, called  Template 1, is currently the only funded portion of the program and  consists of three releases: 1.0 Financial and Acquisition, 1.1 Wholesale and  Retail Supply, and 1.2 Intermediate-Level Maintenance. Release 1.0 is the  largest of the three releases in terms of the functional requirements being  addressed. Specifically, it is to provide about 56 percent of Template 1  requirements. See table 2 for a description of these releases.
DON estimates the life cycle cost for the program’s first increment to be  about $2.4 billion, including about $1 billion for acquisition, and $1.4  billion for operations and maintenance. The life cycle cost of the entire  program has not yet been determined because future increments have not  been defined. The program office reported that approximately $400 million  was spent from fiscal year 2004 through fiscal year 2007 on the first  increment. For fiscal year 2008, about $200 million is planned to be spent.
section_title: Program Oversight and Management Roles and Responsibilities
paragraphs: To manage the acquisition and deployment of Navy ERP, DON established  a program management office within the Program Executive Office for  Executive Information Systems. The program office manages the  program’s scope and funding and is responsible for ensuring that the  program meets its objectives. To accomplish this, the program office is  responsible for key program management areas, such as architectural  alignment, economic justification, earned value management,  requirements management, and risk management. In addition, various  DOD and DON organizations share program oversight and review  activities. A listing of key entities and their roles and responsibilities is in  table 3.
section_title: Overview of Navy ERP’s Status
paragraphs: The first increment of Navy ERP is currently in the production and  deployment phase of the defense acquisition system. The system consists  of five key program life cycle phases and three related milestone decision  points. These five phases and related milestones, along with a summary of  key program activities completed during or planned for each phase, are as  follows:  1.  Concept Refinement: The purpose of this phase is to refine the initial  system solution (concept) and create a strategy for acquiring the  solution. This phase began in July 2003, at which time DON began to  converge the four pilot programs into Navy ERP and developed its first  cost estimate in September 2003. This phase of the program was  combined with the next phase, thus creating a combined Milestone A/B  decision point.  2.  Technology Development: The purpose of this phase is to determine  the appropriate set of technologies to be integrated into the investment  solution by iteratively assessing the viability of the various  technologies while simultaneously refining user requirements. During  the combined Concept Refinement and Technology Development  phase, the program office prepared a concept of operations and  operational requirements document; performed an analysis of  alternatives, business case analysis, and economic analysis; and  established its first Acquisition Program Baseline. It also selected SAP  as the commercial off-the-shelf ERP software. The combined phase  was completed in August 2004, when the MDA approved Milestone A/B  to allow the program to move to the next phase.  3.  System Development and Demonstration: The purpose of this phase is  to develop a system and demonstrate through developer testing that  the system can function in its target environment. This phase was  completed in September 2007, when Release 1.0 passed development  testing and its deployment to NAVAIR began. This was 17 months later  than the program’s original schedule set in August 2004 but on time  according to the revised schedule set in December 2006.
In September 2004, the program office awarded a $176 million system  integration contract to BearingPoint for full system design,  development, and delivery using SAP’s off-the-shelf product and  related customized software. In January 2006, the program office (1)  reduced the contractor’s scope of work from development and  integration of the first increment to only development of the first  release and (2) assumed responsibility and accountability for overall  system integration. According to the program office, reasons for this  change included the need to change the development plan to reflect  improvements in the latest SAP product released and the lack of  authority by the contractor to adjudicate and reconcile differences  among the various Navy user organizations (i.e., Navy commands).
In December 2006, the program office revised its Acquisition Program  Baseline to reflect an increase of about $461 million in the life cycle  cost estimate due, in part, to restructuring the program (e.g., changing  the order of the releases, changing the role of system integrator from  contractor to the program office) and resolving problems related to,  among other things, converting data from legacy systems to run on  Navy ERP and establishing interfaces between legacy systems and  Navy ERP. In addition, the program office awarded a $151 million  contract for Release 1.1 and 1.2 configuration and development to IBM  in June 2007. In September 2007, prior to entering the next phase, the  program revised its Acquisition Program Baseline again to reflect a $9  million decrease in the life cycle cost estimate and a 5-month increase  in its program schedule. Soon after, the MDA approved Milestone C to  move to the next phase.  4.  Production and Deployment: The purpose of this phase is to achieve  an operational capability that satisfies the mission needs, as verified  through independent operational test and evaluation, and to  implement the system at all applicable locations. This phase began in  September 2007, focusing first on achieving initial operational  capability (IOC) of Release 1.0 at NAVAIR by May 2008. This date is 22  months later than the baseline established for Milestone A/B in August  2004, and 4 months later than the new baseline established in  September 2007. According to program documentation, these delays  were due, in part, to challenges experienced at NAVAIR in converting  data from legacy systems to run on the new system and implementing  new business procedures associated with the system.
In light of the delays at NAVAIR in achieving IOC, the deployment  schedules for the other commands were also revised. Specifically,  Release 1.0 is still to be deployed at NAVSUP on October 2008, but  Release 1.0 deployment at SPAWAR is now scheduled 18 months later  than planned (October 2009), and deployment at NAVSEA general fund  and Navy Working Capital Fund is now scheduled to be 12 months  later than planned (October 2010 and 2011, respectively). Because of  the Release 1.0 delays, Release 1.1 is now planned for deployment at  NAVSUP 7 months later than planned (February 2010). Release 1.2 is  still scheduled to be released at Regional Maintenance Centers in  October 2010.
The program office is currently in the process of again re-baselining  the program, and DON plans to address any cost overruns through  reprogramming of fiscal year 2008 DON funds. It estimates that this  phase will be completed with full operational capability (FOC) by  August 2013 (26 months later than the baseline established in 2004, and  5 months later than the re-baseline established in September 2007).  5.  Operations and Support: The purpose of this phase is to operationally  sustain the system in the most cost-effective manner over its life cycle.  In this phase, the program plans to provide centralized support to its  users across all system commands. Each deployment site is expected  to perform complementary support functions, such as data  maintenance.
Overall, Increment 1 was originally planned to reach FOC in fiscal year  2011, and its estimated life cycle cost was about $1.87 billion. The  estimate was later baselined in August 2004 at about $2.0 billion. In  December 2006 and again in September 2007, the program was re- baselined. FOC is now planned for fiscal year 2013, and the estimated life  cycle cost is about $2.4 billion (31 percent increase over the original  estimate). Key activities for each phase are depicted in figure 1, changes  in the deployment schedule are depicted in figure 2, and cost estimates are  depicted in figure 3.
section_title: Use of IT Acquisition Management Controls Maximizes Chances for Success
paragraphs: IT acquisition management controls are tried and proven methods,  processes, techniques, and activities that organizations define and use to  minimize program risks and maximize the chances of a program’s success.  Using these controls can result in better outcomes, including cost savings,  improved service and product quality, and a better return on investment.  For example, two software engineering analyses of nearly 200 systems  acquisition projects indicate that teams using systems acquisition controls  that reflected best practices produced cost savings of at least 11 percent  over similar projects conducted by teams that did not employ the kind of  rigor and discipline embedded in these practices. In addition, our  research shows that these controls are a significant factor in successful  acquisition outcomes, including increasing the likelihood that programs  and projects will be executed within cost and schedule estimates.
We and others have identified and promoted the use of a number of IT  acquisition management controls associated with acquiring IT systems.  See table 4 for a description of several of these activities.
section_title: Prior GAO Reviews Have Identified IT Acquisition Management Control Weaknesses on DOD Business System Investments
paragraphs: We have previously reported that DOD has not effectively managed a  number of business system investments. Among other things, our reviews  of individual system investments have identified weaknesses in such  things as architectural alignment and informed investment decision  making, which are also the focus areas of the Fiscal Year 2005 Defense  Authorization Act business system provisions. Our reviews have also  identified weaknesses in other system acquisition and investment  management areas—such as EVM, economic justification, requirements  management, and risk management.
In July 2007, we reported that the Army’s approach for investing about $5  billion over the next several years in its General Fund Enterprise Business  System, Global Combat Support System-Army Field/Tactical, and Logistics  Modernization Program did not include alignment with the Army enterprise  architecture or use of a portfolio-based business system investment review  process. Moreover, we reported that the Army did not have reliable analyses,  such as economic analyses, to support its management of these programs. We  concluded that, until the Army adopts a business system investment  management approach that provides for reviewing groups of systems and  making enterprise decisions on how these groups will collectively interoperate  to provide a desired capability, it runs the risk of investing significant resources  in business systems that do not provide the desired functionality and efficiency.  Accordingly, we made recommendations aimed at improving the department’s  efforts to achieve total asset visibility and enhancing its efforts to improve its  control and accountability over business system investments. The department  agreed with our recommendations.
We also reported that DON had not, among other things, economically  justified its ongoing and planned investment in the Naval Tactical  Command Support System (NTCSS) and had not invested in NTCSS  within the context of a well-defined DOD or DON enterprise architecture.  In addition, we reported that DON had not effectively performed key  measurement, reporting, budgeting, and oversight activities and had not  adequately conducted requirements management and testing activities. We  concluded that, without this information, DON could not determine  whether NTCSS, as defined, and as being developed, is the right solution  to meet its strategic business and technological needs. Accordingly, we  recommended that the department develop the analytical basis to  determine if continued investment in NTCSS represents prudent use of  limited resources and to strengthen management of the program,  conditional upon a decision to proceed with further investment in the  program. The department largely agreed with our recommendations.
In addition, we reported that the Army had not defined and developed its  Transportation Coordinators’ Automated Information for Movements System  II (TC-AIMS II)—a joint services system with the goal of helping to manage  the movement of forces and equipment within the United States and abroad— in the context of a DOD enterprise architecture. In addition, we reported  that the Army had not economically justified the program on the basis of  reliable estimates of life cycle costs and benefits and had not effectively  implemented risk management. As a result, we concluded that the Army did  not know if its investment in TC-AIMS II, as planned, was warranted or  represented a prudent use of limited DOD resources. Accordingly, we  recommended that the department, among other things, develop the  analytical basis needed to determine if continued investment in TC-AIMS II  represents prudent use of limited defense resources. In response, the  department agreed with our recommendations and has since reduced the  program’s scope by canceling future investments.
Furthermore, in 2005, we reported that DON had invested approximately  $1 billion in the four previously cited ERP pilots without marked  improvement in its day-to-day operations. More specifically, we reported  that the program office had not implemented an EVM system. We also  identified significant challenges and risks as the project moved forward,  such as developing and implementing system interfaces, converting data  from legacy systems into the ERP system, meeting its estimated  completion date of 2011 at an estimated cost of $800 million, and achieving  alignment with DOD’s BEA. To address these areas, we made  recommendations that DOD improve oversight of Navy ERP, including  developing quantitative metrics to evaluate the program. DOD generally  agreed with our recommendations.
section_title: Implementation of Key DOD and Related IT Acquisition Management Controls on Navy ERP Varies
paragraphs: DOD IT-related acquisition policies and guidance, along with other  relevant guidance, provide an acquisition management control framework  within which to manage business system programs like Navy ERP.  Effective implementation of this framework can minimize program risks  and better ensure that system investments are defined in a way to  optimally support mission operations and performance, as well as deliver  promised system capabilities and benefits on time and within budget. To  varying degrees of effectiveness, Navy ERP has been managed in  accordance with aspects of this framework. However, implementation of  key management controls has not been effective. Specifically,    compliance with DOD’s federated BEA has not been sufficiently  investment in the program has been economically justified on the basis of  expected life cycle benefits that will likely exceed estimated life cycle  costs, although some estimating limitations nevertheless exist;   earned value management has not been effectively implemented;    an important requirements management activity has been effectively    a risk management process has been defined, but not effectively  implemented for all risks.
The reasons that program management and oversight officials cited for  why these key practices have not been sufficiently executed range from  limitations in the applicable DOD guidance and tools to the complexity  and challenges of managing and implementing a program of this size. Each  of these reasons is described in the applicable sections of this report. By  not effectively implementing all the above key IT acquisition management  functions, the program is at increased risk of (1) not being defined in a  way that best meets corporate mission needs and enhances performance  and (2) adding to the more than 2 years in program schedule delays and  about $570 million in program cost increases experienced to date.
subsections: section_title: Navy ERP Compliance with DOD’s Federated BEA Has Not Been Sufficiently Demonstrated
paragraphs: DOD and other guidance, recognize the importance of investing in  business systems within the context of an enterprise architecture.  Moreover, the Fiscal Year 2005 Defense Authorization Act requires that  defense business systems be compliant with DOD’s federated BEA. Our  research and experience in reviewing federal agencies show that not  making investments within the context of a well-defined enterprise  architecture often results in systems that are duplicative, are not well  integrated, are unnecessarily costly to interface and maintain, and do not  optimally support mission outcomes.
To its credit, the program office has followed DOD’s BEA compliance  guidance. However, this guidance does not adequately provide for  addressing all relevant aspects of BEA compliance. Moreover, DON’s  enterprise architecture, which is a major component of DOD’s federated  BEA, as well as key aspects of DOD’s corporate BEA, have yet to be  sufficiently defined to permit thorough compliance determinations. In  addition, current policies and guidance do not require DON investments to  comply with its enterprise architecture. This means that the department  does not have a sufficient basis for knowing if Navy ERP has been defined  to minimize overlap with and duplication of other programs’ functionality  and maximize interoperability among related programs. Each of these  architecture alignment limitations is discussed here:    The program’s compliance assessments did not include all relevant  architecture products. In particular, the program did not assess  compliance with the BEA’s technical standards profile, which outlines, for  example, the standards governing how systems physically communicate  with other systems and how they secure data from unauthorized access.  This is particularly important because systems like Navy ERP need to  share information with other systems and, for these systems to accomplish  this effectively and efficiently, they need to employ common standards. A  case in point is the relationship between Navy ERP and the Global Combat  Support System—Marine Corps (GCSS-MC) program. Specifically, Navy  ERP has identified 25 technical standards that are not in the BEA technical  standards profile, and GCSS-MC has identified 13 technical standards that  are not in the profile. Among these non-BEA standards are program-unique  information sharing protocols, which could limit information sharing  between Navy ERP and GCSS-MC, and with other systems.
In addition, the program office did not assess compliance with the BEA  products that describe system-level characteristics. This is important  because doing so would create a body of information about programs that  could be used to identify common system components and services that  could potentially be shared by the programs, thus avoiding wasteful  duplication. For example, our analysis of Navy ERP program  documentation shows that it contains system functions related to  receiving goods, taking physical inventories, and returning goods, which  are also system functions cited by the GCSS-MC program. However,  because compliance with the BEA system products was not assessed, the  extent to which these functions are potentially duplicative was not  considered.
Furthermore, the program office did not assess compliance with BEA  system products that describe data exchanges among systems. As we  previously reported, establishing and using standard system interfaces is a  critical enabler to sharing data. For example, Navy ERP program  documentation indicates that it is to exchange inventory order and status  data with other systems. System interfaces are important for  understanding how information is to be exchanged between systems.  However, since the program was not assessed for compliance with these  products, it does not have the basis for understanding how its approach to  exchanging information differs from that of other systems that it is to  interface with. Compliance against each of these BEA products was not  assessed because DOD’s compliance guidance does not provide for doing  so and, according to BTA officials, because some BEA system products  are not sufficiently defined. According to these officials, BTA plans to  continue to define these products as the BEA evolves.
The compliance assessment was not used to identify potential areas of  duplication across programs, which DOD has stated is an explicit goal of  its federated BEA and associated investment review and decision-making  processes. More specifically, even though the compliance guidance  provides for assessing programs’ compliance with the BEA product that  defines DOD operational activities, and Navy ERP was assessed for  compliance with this product, the results were not used to identify  programs that support the same operational activities and related business  processes. Given that the federated BEA is intended to identify and avoid  not only duplications within DOD components, but also between  components, it is important that such commonality be addressed. For  example, BEA compliance assessments for Navy ERP and GCSS-MC, as  well as two Air Force programs (Defense Enterprise Accounting and  Management System—Air Force and the Air Force Expeditionary Combat  Support System) show that each program supports at least six of the same  BEA operational activities (e.g., conducting physical inventory, delivering  property and services) and three of these four programs support at least   18 additional operational activities (e.g., performing budgeting, managing  receipt and acceptance). However, since the potential overlap among  these and other programs was not assessed, these programs may be  investing in duplicative functionality. Reasons for this were that the  compliance guidance does not provide for such analyses to be conducted  and programs have not been granted access rights to use this functionality  in the compliance tool.
The program’s compliance assessment did not address compliance against  DON’s enterprise architecture, which is one of the biggest members of the  federated BEA. This is particularly important given that DOD’s approach  to fully satisfying the architecture requirements of the Fiscal Year 2005  Defense Authorization Act is to develop and use a federated architecture  in which component architectures are to provide the additional details  needed to supplement the thin layer of corporate policies, rules, and   standards included in the corporate BEA. As we recently reported,  DON’s enterprise architecture is not mature because, among other things,  it is missing a sufficient description of its current and future environments  in terms of business and information/data. However, certain aspects of an  architecture nevertheless exist and, according to DON CIO officials, these  aspects will be leveraged in its efforts to develop a complete enterprise  architecture. For example, the FORCEnet architecture is intended to  document Navy’s technical infrastructure. Therefore, opportunities exist  for DON to assess its programs in relation to these architecture products,  and to understand where its programs are exposed to risks because  products do not exist, are not mature, or at odds with other Navy  programs. According to DOD officials, compliance with the DON  architecture was not assessed because DOD compliance policy is limited  to compliance with the corporate BEA, and a number of aspects of the  DON enterprise architecture have yet to be sufficiently developed.
The program’s compliance assessment was not validated by DON or DOD  investment oversight and decision-making authorities. More specifically,  neither the DOD IRBs nor the DBSMC, nor the BTA in supporting both of  these investment oversight and decision-making authorities, reviewed the  program’s assessments. According to BTA officials, under DOD’s tiered  approach to investment accountability, these entities are not responsible  for validating programs’ compliance assessments. Rather, this is a  component responsibility, and thus they rely on the military departments  and defense agencies to validate the assessments.
However, DON Office of the CIO, which is responsible for precertifying  investments as compliant before they are reviewed by the IRB, did not  validate any of the program’s compliance assessments. According to Office  of the CIO officials, they rely on Functional Area Managers to validate a  program’s compliance assessments. However, no DON policy or guidance  exists that describes how the Functional Area Managers should conduct  such validations. CIO officials stated that this is because these authorities  do not have the resources that they need to validate the assessments, and  because a number of aspects of the DON architecture are not yet  sufficiently developed.
Validation of program assessments is further complicated by the absence  of information captured in the assessment tool about what program  documentation or other source materials were used by the program office  in making its compliance determinations. Specifically, the tool is only  configured, and thus was only used, to capture the results of a program’s  comparison of program architecture products to BEA products. Thus, it  was not used to capture the system products used in making these  determinations.
The limitations in existing BEA compliance-related policy and guidance,  the supporting compliance assessment tool, and the federated BEA, put  programs like Navy ERP at increased risk of being defined and  implemented in a way that does not sufficiently ensure interoperability  and avoid duplication and overlap. We recently completed a review  examining multiple programs’ compliance with the federated BEA,  including Navy ERP, for the Senate Armed Services Committee,  Subcommittee on Readiness and Management Support. We addressed the  architectural compliance guidance, tool, and validation limitations as part  of this review.
section_title: Investment in Navy ERP Has Been Economically Justified, but Important Estimating Practices Were Not Implemented
paragraphs: The investment in Navy ERP has been economically justified on the basis  of expected life cycle benefits that far exceed estimated life cycle costs.  According to the program’s benefit/cost analysis, Navy ERP will produce  about $8.6 billion in estimated benefits for an estimated cost of about $2.4  billion over its 20-year life cycle. While these benefit estimates were not  subject to any analysis of how uncertainty in assumptions and data could  impact the estimates, as called for by relevant guidance, our examination  of key uncertainty variables, such as the timing of legacy systems’  retirement, showed that the savings impact would be relatively minor.  However, the reliability of the cost estimate is limited because it was  derived using several, but not all, key estimating practices. For example,  the estimate was not grounded in a historical record of comparable data  from similar programs and was not based on a reliable schedule baseline,  which are both necessary to having a cost estimate that can be considered  credible and accurate. These practices were not employed for various  reasons, including DOD’s lack of historical data from similar programs and  the lack of an integrated master schedule for the program that includes all  releases.
Notwithstanding the fact that these limitations could materially increase  the $2.4 billion cost estimate, it is nevertheless unlikely that these factors  would increase the estimate to a level approaching the program’s benefit  expectations. Therefore, we have no reason to believe that Navy ERP will  not produce a positive return on investment.
subsections: section_title: Benefit Estimates Are Sufficiently Reliable Despite Absence of Uncertainty Analysis
paragraphs: Forecasting expected benefits over the life of a program is a key aspect of  economically justifying an investment. The Office of Management and  Budget (OMB) guidance advocates economically justifying investments  on the basis of expected benefits, costs, and risks. Since estimates of  benefits can be uncertain because of the imprecision in both the  underlying data and modeling assumptions used, the guidance also  provides for analyzing and reporting the effects of this uncertainty. By  doing this, informed investment decision making can occur through the  life of the program, and a baseline can be established against which to  compare the accrual of actual benefits from deployed system capabilities.
The most recent economic analysis, dated August 2007, includes  monetized benefit estimates for fiscal years 2004–2023, in three key  areas—about $2.7 billion in legacy system cost savings, $3.3 billion in cost  savings from inventory reductions, and $2.7 billion in cost savings from  labor productivity improvements. Collectively, these benefits total about  $8.6 billion.
The program office calculated expected benefits in terms of cost savings,  which is consistent with established practices and guidance. For example,  the program is to result in the retirement of 138 legacy systems (including  the 4 pilot systems) between fiscal years 2005 and 2015, and the yearly  maintenance costs for a single system are expected to be as high as about  $39 million. According to relevant guidance, cost saving estimates should  also be analyzed in terms of how uncertainty in assumptions and data  could impact them. However, the program office did not perform such  uncertainty analysis. According to program officials, uncertainty analysis  is not warranted because they have taken and continue to take steps to  validate the assumptions and the data, such as using the latest budget data  associated with the legacy systems, and monitoring changes to the  systems’ retirement dates. While these steps are positive, they do not  eliminate the need for uncertainty analysis. Accordingly, we assessed key  uncertainty variables, such as the timing of the legacy systems’ retirement,  and found that the retirement dates of some of these systems have  changed since the estimate was prepared, due to, among other things,  schedule delays in the program. While the inherent uncertainty in these  dates would reduce expected savings (e.g., only $11 million based on the  134 legacy systems that we examined), the reduction would be small  relative to a total benefit estimate of $8.6 billion.
section_title: The Cost Estimate Was Not Reliably Derived
paragraphs: A reliable cost estimate is a key variable in calculating return on  investment, and it provides the basis for informed investment decision  making, realistic budget formulation and program resourcing, meaningful  progress measurement, proactive course correction, and accountability for  results. According to OMB, programs must maintain current and well- documented cost estimates, and these estimates must encompass the full  life cycle of the program. OMB states that generating reliable cost  estimates is a critical function necessary to support OMB’s capital  programming process. Without reliable estimates, programs are at  increased risk of experiencing cost overruns, missed deadlines, and  performance shortfalls.
Our research has identified a number of practices that are the basis of  effective program cost estimating. We have issued guidance that  associates these practices with four characteristics of a reliable cost  estimate. Specifically, these four characteristics are as follows:    Comprehensive: The cost estimates should include both government and  contractor costs over the program’s full life cycle, from the inception of  the program through design, development, deployment, and operation and  maintenance to retirement. They should also provide an appropriate level  of detail to ensure that cost elements are neither omitted nor double  counted and include documentation of all cost-influencing ground rules  and assumptions.
Well-documented: The cost estimates should have clearly defined  purposes and be supported by documented descriptions of key program or  system characteristics (e.g., relationships with other systems, performance  parameters). Additionally, they should capture in writing such things as  the source data used and their significance, the calculations performed  and their results, and the rationale for choosing a particular estimating  method or reference. Moreover, this information should be captured in  such a way that the data used to derive the estimate can be traced back to,  and verified against, their sources. The final cost estimate should be  reviewed and accepted by management on the basis of confidence in the  estimating process and the estimate produced by the process.
Accurate: The cost estimates should provide for results that are unbiased  and should not be overly conservative or optimistic (i.e., they should  represent most likely costs). In addition, the estimates should be updated  regularly to reflect material changes in the program, and steps should be  taken to minimize mathematical mistakes and their significance. Among  other things, the estimate should be grounded in a historical record of cost  estimating and actual experiences on comparable programs.
Credible: The cost estimates should discuss any limitations in the analysis  performed due to uncertainty or biases surrounding data or assumptions.  Further, the estimates’ derivation should provide for varying any major  assumptions and recalculating outcomes based on sensitivity analyses,  and their associated risks and inherent uncertainty should be disclosed.  Also, the estimates should be verified based on cross-checks using other  estimating methods and by comparing the results with independent cost  estimates.
The $2.4 billion life cycle cost estimate for Navy ERP reflects many of the  practices associated with a reliable cost estimate, including all practices  associated with being comprehensive and well-documented, and several  related to being accurate and credible (see table 5). However, several  important practices related to accuracy and credibility were not  performed. To be reliable, a cost estimate should be comprehensive, well- documented, accurate, and credible.
The cost estimate is comprehensive because it includes both the  government and contractor costs specific to development, acquisition  (nondevelopment), implementation, and operations and support over the  program’s 20-year life cycle. Moreover, the estimate clearly describes how  the various subelements are aggregated to produce amounts for each cost  category, thereby ensuring that all pertinent costs are included, and no  costs are double counted. Finally, cost-influencing ground rules and  assumptions, such as the program’s schedule, labor rates, and inflation  rates are documented.
The cost estimate is also well-documented in that the purpose of the cost  estimate is clearly defined, and the technical baseline includes, among  other things, the hardware components and planned performance  parameters. Furthermore, the calculations and results used to derive the  estimate are documented, including descriptions of the methodologies  used and evidence of traceability back to source data (e.g., vendor quotes,  salary tables). Also, the cost estimate was reviewed by the Naval Center  for Cost Analysis and the Office of the Secretary of Defense, Director for  Program Analysis and Evaluation, which adds a level of confidence in the  estimating process and the estimate produced.
However, the estimate lacks accuracy because not all important practices  related to this characteristic were performed. Specifically, while the  estimate is grounded in documented assumptions (e.g., hardware  refreshment every 5 years) and periodically updated to reflect changes to  the program, it is not adequately grounded in historical experience with  comparable programs. While the program office did leverage historical  cost data from the Navy ERP pilot programs, program officials told us that  the level of cost accounting on these programs did not provide sufficient  data. As stated in our guide, estimates should be based on historical  records of cost and schedule estimates from comparable programs, and  such historical data should be maintained and used for evaluation  purposes and future estimates on comparable programs. The importance  of doing so is evident by the fact that Navy ERP’s cost estimate has  increased by about $570 million since fiscal year 2003, which program  officials attributed to, among other things, site implementation costs (e.g.,  training and converting legacy system data) not included in the original  cost estimate, schedule delays, and the lack of historical data from similar  ERP programs.
This lack of cost data for large-scale ERP programs is, in part, due to DOD  not having a standardized cost element structure for these programs that  can be used for capturing actual cost data, which is a prerequisite to  capturing and maintaining the kind of historical data that can inform cost  estimates on similar programs. This means that programs like Navy ERP  will not be able to ground their cost estimates in actual costs from  comparable programs. According to officials with the Defense Cost and  Resource Center, such cost element structures are needed, along with a  requirement for programs to report on their costs, but approval and  resources have yet to be gained for either these structures or the reporting  of their costs. We recently completed work that addressed standardization  of DOD’s ERP cost element structure and maintenance of a database for  historical ERP cost data for use on ERP programs.
Compounding the estimate’s limited accuracy are limitations in its  credibility. Specifically, while the estimate satisfies some of the key  practices for a credible cost estimate (e.g., confirming key cost drivers,  performing sensitivity analyses, and having an independent cost estimate  prepared by the Naval Center for Cost Analysis that was within 11 percent  of the program’s estimate), the program lacks a reliable schedule baseline,  which is a key component of a reliable cost estimate because it serves as  the basis for future work to be performed. Other factors that limit  confidence in the cost estimate’s accuracy are (1) past increases in the  program’s cost estimate (as discussed earlier) and (2) trends in EVM data  (as discussed later). Taken together, the program’s cost estimate is not  sufficiently credible and accurate and thus not reliable.
While important cost estimating practices were not implemented, it is  nevertheless unlikely that these limitations would materially increase the  $2.4 billion cost estimate to a level approaching the program’s $8.6 billion  benefit expectations.
section_title: Earned Value Management Has Not Been Effectively Implemented
paragraphs: Measuring and reporting progress against cost and schedule commitments  (i.e., baselines) is a vital element of effective program management. EVM  provides a proven means for measuring such progress and thereby  identifying potential cost overruns and schedule delays early, when their  impact can be minimized.
To its credit, the program has elected to implement program-level EVM,  which is a best practice that has rarely been implemented in the federal  government. In doing so, however, basic EVM activities have not been  executed. In particular, an integrated baseline review, which is to verify  that the program’s cost and schedule are reasonable given the program’s  scope of work and associated risks, has not been performed. Moreover,  other accepted industry standards have not been sufficiently implemented,  and surveillance of EVM implementation by an entity independent of the  program office has not occurred. Not performing these important  practices has contributed to the cost overruns and lengthy schedule delays  already experienced on Release 1.0, and they will likely result in more. In  fact, our analysis of the latest estimate to complete just the budgeted  development work for all three releases, which is about $844 million,  shows that this estimate will most likely be exceeded by about $152  million.
subsections: section_title: The Program Has Elected to Implement Program-Level EVM
paragraphs: As we previously reported, EVM offers many benefits when done  properly. In particular, it allows performance to be measured, and it serves  as an early warning system for deviations from plans. It, therefore, enables  a program office to mitigate the risks of cost and schedule overruns. OMB  policy recognizes the use of EVM as an important part of program  management and decision making.
Implementing EVM at the program level rather than just the contract level  is considered a best practice, and OMB recently began requiring it to  measure how well a program’s approved cost, schedule, and performance  goals are being met. According to OMB, integrating government and  contractor cost, schedule, and performance status should result in better  program execution through more effective management. In addition,  integrated EVM data can be used to better justify budget requests.
To minimize the risk associated with its decision to transition  responsibility for Navy ERP system integration from the contractor to the  government and to improve cost and schedule performance, the program  office elected in October 2006 to perform EVM at the program level. We  support the use of program-level EVM. However, if not implemented  effectively, this program-level approach will be of little value.
section_title: An Integrated Baseline Review Has Not Been Performed
paragraphs: A fundamental aspect of effective EVM is the development of a  performance measurement baseline (PMB), which represents the  cumulative value of planned work and serves as the baseline against which  variances are calculated. According to relevant best practice guidance, a  PMB consists of    a complete work breakdown structure,    a complete integrated master schedule, and    accurate budgets for all planned work.
To validate the PMB, an integrated baseline review is performed to obtain  stakeholder agreement on the baseline. According to DOD guidance and  best practices, such a review should be held within 6 months of a contract  award and conducted on an as needed basis throughout the life of a  program to ensure that the baseline reflects (1) all tasks in the statement  of work, (2) adequate resources (staff and materials) to complete the  tasks, and (3) integration of the tasks into a well-defined schedule.  Further, the contract performance reports that are to be used to monitor  performance against the PMB should be validated during the integrated  baseline review.
The program office has satisfied some of the prerequisites for having a  reliable PMB, such as developing a work breakdown structure and  specifying the contract performance reports that are to be used to monitor  performance. However, it has not conducted an integrated baseline  review. Specifically, a review was not conducted for Release 1.0, even  though the contract was finalized about 30 months ago (January 2006).  Also, while the review for Release 1.1 was recently scheduled for August  2008, this is 8 months later than when such a review should be held,  according to DOD guidance and best practices. This means that the  reasonableness of the program’s scope and schedule relative to the  program risks has not been assured, and has likely been, and will likely  continue to be a primary contributor to future cost increases and schedule  delays.
According to program officials, a review was not performed on the first  release because development of this release was largely complete by the  time the program office established the underlying capabilities needed to  perform program-level EVM. In addition, program officials stated that an  integrated baseline review has yet to be performed on the other two  releases because their priority has been on deploying and stabilizing the  first release. In our view, not assuring the validity of the PMB precludes  effective implementation of EVM. Until a review is conducted, DOD will  not have reasonable assurance that the program’s scope and schedule are  achievable, and thus, additional cost and schedule overruns are likely.
section_title: Industry EVM Standards Have Not Been Fully Implemented and Independently Surveilled
paragraphs: In 1996, DOD adopted industry EVM guidance that identifies 32 essential  practices organized into five categories: (1) organization; (2) planning,  scheduling and budgeting; (3) accounting; (4) analysis and management  reports; and (5) revisions and data maintenance. DOD requires that all  programs’ implementation of EVM undergo a compliance audit against the  32 industry practices. In addition, DOD policy and guidance state that  independent surveillance of EVM should occur over the life of the program  to guarantee the validity of the performance data and ensure that EVM is  being used effectively to manage cost, schedule, and technical  performance.
On Navy ERP, compliance with the 32 accepted industry practices has not  been verified, and surveillance of EVM by an independent entity has not  occurred. Therefore, the program does not have the required basis for  ensuring that EVM is being effectively implemented on Navy ERP.  According to program officials, surveillance was performed by NAVAIR  for Release 1.0. However, NAVAIR officials said that they did not perform  such surveillance because they did not receive the Release 1.0 cost  performance data needed to do so. Program officials also stated that  DON’s Center for Earned Value Management has conducted an initial  assessment of their EVM management system, and that they intend to have  the Center perform surveillance. However, they did not have a plan for  accomplishing this. Until compliance with the standards is verified and  continuous surveillance occurs, and deviations are addressed, the program  will likely continue to experience cost overruns and schedule delays.
section_title: Estimated Schedule Baseline Was Not Derived in Accordance with All Key Schedule Estimating Practices
paragraphs: The success of any program depends in part on having a reliable schedule  of when the program’s work activities will occur, how long they will take,  and how they are related to one another. As such, the schedule not only  provides a road map for the systematic execution of a program but also  provides the means by which to estimate costs, gauge progress, identify  and address potential problems, and promote accountability. Our research  has identified nine practices associated with effective schedule  estimating. These practices are (1) capturing key activities, (2)  sequencing key activities, (3) establishing the duration of key activities, (4)  assigning resources to key activities, (5) integrating key activities  horizontally and vertically, (6) establishing the critical path for key  activities, (7) identifying “float time” between key activities, (8)  distributing reserves to high-risk activities, and (9) performing a schedule  risk analysis.
The program’s estimated schedule was developed using some of these  practices, but several key practices were not fully employed that are  fundamental to having a schedule that provides a sufficiently reliable basis  for estimating costs, measuring progress and forecasting slippages. On the  positive side, the schedule for the first two releases captures key activities  and their durations and is integrated horizontally and vertically, meaning  that multiple teams executing different aspects of the program can  effectively work to the same master schedule. Moreover, for these two  releases, the program has established float time between key activities and  distributed schedule reserve to high-risk activities. However, the program  has not adequately sequenced and assigned resources to key program  activities. Moreover, the estimated schedule for the first increment is not  grounded in an integrated master schedule of all the releases, and thus the  schedule for this increment does not reflect the program’s critical path of  work that must be performed to achieve the target completion date. Also,  it does not reflect the results of a schedule-risk analysis across all three  releases with schedule reserve allocated to high-risk activities because  such risks were not examined. See table 6 for the results of our analyses  relative to each of the nine practices.
According to program documentation, they have plans to address the  logical sequencing of activities (to ensure that it reflects how work is to be  performed), but program officials stated that they do not plan to combine  all three releases into a single integrated master schedule for the entire  first increment of the program because doing so would produce an overly  complex and nonexecutable schedule involving as many as 15,000  activities. However, our research of and experience in evaluating major  programs’ use of EVM and integrated master schedules show that while  large, complex programs necessitate schedules involving thousands of  activities, successful programs ensure that their schedules integrate these  activities. In our view, not adequately performing these practices does not  allow the program to effectively assign resources, identify the critical path,  and perform a schedule risk analysis that would allow it to understand,  disclose, and compensate for its schedule risks. This means that the  program is not well-positioned to understand progress and forecast its  impact. To illustrate, the program recently experienced delays in  deploying its first release at NAVAIR, which according to a recent  operational test and evaluation report has significantly affected the  schedule’s critical path. These schedule impacts are because resources  supporting the deployment at NAVAIR began to shift to the next scheduled  deployment site and thus are no longer available to resolve critical issues  at NAVAIR. Since the schedule baseline is not integrated across all  releases, the impact of this delay on other releases, and thus the program  as a whole, cannot be readily determined.
section_title: Trends in EVM Data Show Pattern of Cost Overruns and Schedule Slippages
paragraphs: Program data show a pattern of actual cost overruns and schedule delays  between January 2007 and May 2008. Moreover, our analysis of the data  supports a most likely program cost growth of about $152 million to  complete all three releases.
Differences from the PMB are measured in both cost and schedule  variances. Positive variances indicate that activities are costing less or  are completed ahead of schedule. Negative variances indicate that  activities are costing more or are falling behind schedule. These cost and  schedule variances can then be used in forecasting the cost and time  needed to complete the program.
Based on program-provided data for the first increment over a 17-month  period ending May 2008, the program has experienced negative cost  variances. Specifically, while these cost variances have fluctuated during  this period, they have consistently been negative. (See fig. 4.) Moreover,  our analysis of the cost to complete just the budgeted development work  (also known as the PMB) for all three releases, which is about $844  million, will be exceeded by between about $102 million and $316 million,  with a most likely overrun of about $152 million. In contrast, the program  office reports that the overrun at completion will be $55 million but has  yet to provide us with documentation supporting this calculation.  Moreover, our calculation does not reflect the recent problems discovered  during the operational test and evaluation at NAVAIR and thus the overrun  is likely to be higher.
During this same 17-month period, the program has experienced negative  schedule variances and, since January 2008, they have almost doubled  each month. Further, as of May 2008, the program had not completed  about $24 million in scheduled work. (See fig. 5.) An inability to meet  schedule performance is a frequent indication of future cost increases, as  more spending is often necessary to resolve schedule delays.
Because the program office has not performed important reliability  checks, such as EVM validation and integrated baseline reviews, as  discussed above, we cannot be certain that the PMB is reliable (i.e.,  reflects all the work to be done and has identified all the risks). As a result,  the overrun that we are forecasting could be higher.
By not executing basic EVM practices, the program has and will likely  continue to experience cost and schedule shortfalls. Until the program  office implements these important EVM practices, it will likely not be able  to track actual program costs and schedules close to estimates.
section_title: Important Requirements Management Activity Has Been Effectively Implemented
paragraphs: Well-defined and managed requirements are recognized by DOD guidance  as essential, and they can be viewed as a cornerstone of effective system  acquisition. One aspect of effective requirements management is  requirements traceability. By tracing requirements both backward from  system requirements to higher level business or operational requirements  and forward to system design specifications and test plans, the chances of  the deployed product satisfying requirements is increased, and the ability  to understand the impact of any requirement changes and thus make  informed decisions about such changes, is enhanced.
The program office is effectively implementing requirements traceability  for its 1,733 Release 1.0 system requirements. To verify this traceability,  we randomly selected and analyzed 60 of the 1,733 system requirements  and confirmed that 58 of the 60 were traceable both backward to higher  level requirements and forward to design specifications and test results.  The remaining 2 had been allocated to the other releases, and thus we also  confirmed the program’s ability to maintain traceability between product  releases. In doing so, the program utilized a tool called DOORS, which if  implemented properly, allows each requirement to be linked from its most  conceptual definition to its most detailed definition, as well as to design  specifications and test cases. In effect, the tool maintains the linkages  among requirement documents, design documents, and test cases even if  requirements change.
If DON continues to effectively implement requirements traceability, it will  increase the chances that system requirements will be met by the deployed  system.
section_title: A Risk Management Process Has Been Defined, but All Risks Have Not Been Effectively Mitigated
paragraphs: Proactively managing program risks is a key acquisition management  control and, if defined and implemented properly, it can increase the  chances of programs delivering promised capabilities and benefits on time  and within budget. To the program office’s credit, it has defined a risk  management process that meets relevant guidance. However, it has not  effectively implemented the process for all identified risks. As a result,  these risks have not been proactively mitigated and either have  contributed to cost and schedule shortfalls, or could potentially contribute  to such shortfalls.
DOD acquisition management guidance, as well as other relevant guidance  advocates identifying facts and circumstances that can increase the  probability of an acquisition’s failing to meet cost, schedule, and  performance commitments and then taking steps to reduce the probability  of their occurrence and impact. In brief, effective risk management  consists of: (1) establishing a written plan for managing risks; (2)  designating responsibility for risk management activities; (3) encouraging  program-wide participation in the identification and mitigation of risks; (4)  defining and implementing a process that provides for the identification,  analysis, and mitigation of risks; and (5) examining the status of identified  risks in program milestone reviews.
The program office has developed a written plan for managing risks and  established a process that together provide for the above-cited risk  management practices. Moreover, it has largely followed its plan and  process as per the following examples:    The program manager has been assigned overall responsibility for  managing risks and serves as the chair of the risk management board.  Also, a functional team lead (i.e., subject matter expert) is assigned  responsibility for analyzing and mitigating each identified risk.
Program-wide participation in the identification, analysis, and mitigation  of risks is encouraged. Specifically, a manager for each release is  responsible for providing risk management guidance to the staff, which  includes staff identification and analysis of risks. Also, according to the  program office’s risk management plan, all program personnel can submit  a risk for approval. In addition, stakeholders participate in risk  management activities during acquisition milestone reviews.
The program office has identified and categorized individual risks. As of  June 2008, the risk database contained 15 active risks—3 high, 8 medium,  and 4 low.
Program risks are considered during program milestone reviews. For  example, during the program’s critical design review, which is a key event  of the system development and demonstration phase, key risks regarding  implementing new business processes and legacy system changes were  discussed. Furthermore, the program manager receives a monthly risk  report that describes the status of program risks.
However, the program office has not consistently followed other aspects of its  process. In particular, it has not effectively implemented steps for mitigating the  risks associated with (1) converting data from NAVAIR’s legacy systems to run  on Navy ERP and (2) positioning NAVAIR for adopting the new business  processes embedded in Navy ERP. As we have previously reported, it is  important for organizations that are to operate and use commercial off-the-shelf  software products, such as Navy ERP, to proactively manage and position  themselves for the organizational impact of introducing functionality  embedded in the commercial products. If they do not, the organization’s  performance will suffer.
To the program office’s credit, it identified numerous risks associated with  data conversion and organizational change management and developed  and implemented strategies that were intended to mitigate these risks.  However, it closed these risks even though they were never effectively  mitigated, as evidenced by the results of recently completed DON  operational test and evaluation. According to the June 2008 operational  test and evaluation report for NAVAIR, significant problems relating to  both legacy system data conversion and adoption of new business  processes were experienced. The report states that these problems have  contributed to increases in the costs to operate the system, including  unexpected manual effort. It further states that these problems have  rendered the deployed version not operationally effective and that  deployment of the system to other sites should not occur until the change  management process has been analyzed and improved. It also attributed  the realization of the problems to the program office and NAVAIR not  having adequately engaged and communicated early with each other to  coordinate and resolve differences in organizational perspectives and  priorities and provide intensive pre-deployment preparation and training.  Program officials acknowledged these shortcomings and attributed them  to their limited authority over the commands. In this regard, they have  previously surfaced these risks with department oversight and approval  authorities, but actions were not taken by these authorities that ensured  that the risks were being effectively mitigated.
Beyond not effectively mitigating these risks, the program office has not  ensured that all risks are captured in the risk inventory. For example, the  inventory does not include the risks described in this report that are  associated with not having adequately demonstrated the program’s  alignment to the federated BEA and not having implemented program- level EVM in a manner that reflects industry practices. This means that  these risks are not being disclosed or mitigated.
By not effectively addressing all risks associated with the program, these  risks can and have become problems that contribute to cost and schedule  shortfalls. Until all significant risks are proactively addressed, to include  ensuring that all associated mitigation steps are implemented and that they  accomplished their intended purpose, the program will likely experience  further problems at subsequent deployment sites.
section_title: Conclusions
paragraphs: DOD’s success in delivering large-scale business systems, such as Navy  ERP, is in large part determined by the extent to which it employs the kind  of rigorous and disciplined IT management controls that are reflected in  department policies and related guidance. While implementing these  controls does not guarantee a successful program, it does minimize a  program’s exposure to risk and thus the likelihood that it will fall short of  expectations. In the case of Navy ERP, living up to expectations is  important because the program is large, complex, and critical to  addressing the department’s long-standing problems related to financial  transparency and asset visibility.
The effectiveness to which key IT management controls have been  implemented in Navy ERP varies, with one control and several aspects of  others being effectively implemented, and others less so. Moreover, those  controls that have not been effectively implemented have, in part,  contributed to the sizable cost and schedule shortfalls experienced to date  on the program. Unless this changes, more shortfalls can be expected.
While the program office is primarily responsible for ensuring that effective IT  management controls are implemented, other oversight and stakeholder  organizations share responsibility. For example, even though the program has  not demonstrated its alignment with the federated BEA, it nevertheless  followed established DOD architecture compliance guidance and used the  related compliance assessment tool in assessing and asserting its compliance.  The root cause for not demonstrating compliance thus is not traceable to the  program office but rather is due to, among other things, the limitations of the  compliance guidance and tool, and the program’s oversight entities not  validating the compliance assessment and assertion. Also, the reason that the  program’s cost estimate was not informed by the cost experiences of other  programs of the same size and scope is because DOD does not have a standard  ERP cost element structure and has not maintained a historical database of  costs for like programs to use. In contrast, effective implementation of other  management controls, such as implementing EVM, requirements traceability,  and risk management is the responsibility of the program office.
All told, addressing the management control weaknesses requires the  combined efforts of the various organizations that share responsibility for  managing and overseeing the program. By doing so, the department can  better assure itself that Navy ERP will optimally support its performance  goals and will deliver promised capabilities and benefits on time and  within budget.
Because we recently completed work that more broadly addresses the  above cited architectural alignment and comparable program cost data  limitations, we are not making recommendations in this report for  addressing them.
section_title: Recommendations for Executive Action
paragraphs: To strengthen Navy ERP management control and better provide for the  program’s success, we are making the following recommendations:    To improve the reliability of Navy ERP benefit estimates and cost  estimates, we recommend that the Secretary of Defense direct the  Secretary of the Navy, through the appropriate chain of command, to  ensure that future Navy ERP estimates include uncertainty analyses of  estimated benefits, reflect the risks associated with not having cost data  for comparable ERP programs, and are otherwise derived in full  accordance with the other key estimating practices, and economic analysis  practices discussed in this report.
To enhance Navy ERP’s use of EVM, we recommend that the Secretary of  Defense direct the Secretary of the Navy, through the appropriate chain of  command, to ensure that (1) an integrated baseline review on the last two  releases of the first increment is conducted, (2) compliance against the 32  accepted industry EVM practices is verified, and (3) a plan to have an  independent organization perform surveillance of the program’s EVM  system is developed and implemented.
To increase the quality of the program’s integrated master schedule, we  recommend that the Secretary of Defense direct the Secretary of the Navy,  through the appropriate chain of command, to ensure that the schedule (1)  includes the logical sequencing of all activities, (2) reflects whether all  required resources will be available when needed, (3) defines a critical  path that integrates all three releases, (4) allocates reserve for the high- risk activities on the entire program’s critical path, and (5) incorporates  the results of a schedule risk analysis for all three releases and  recalculates program cost and schedule variances to more accurately  determine a most likely cost and schedule overrun.
To improve Navy ERP’s management of program risks, we recommend  that the Secretary of Defense direct the Secretary of the Navy, through the  appropriate chain of command, to ensure that (1) the plans for mitigating  the risks associated with converting data from legacy systems to Navy  ERP and positioning the commands for adopting the new business  processes embedded in the Navy ERP are re-evaluated in light of the  recent experience with NAVAIR and adjusted accordingly, (2) the status  and results of these and other mitigation plans’ implementation are  periodically reported to program oversight and approval authorities, (3)  these authorities ensure that those entities responsible for implementing  these strategies are held accountable for doing so, and (4) each of the risks  discussed in this report are included in the program’s inventory of active  risks and managed accordingly.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, signed by the Deputy Under  Secretary of Defense (Business Transformation) and reprinted in appendix  II, DOD stated that it concurred with two of our four recommendations  and partially concurred with the remaining two. Further, it stated that it  has taken steps to address some of our recommendations, adding that it is  committed to implementing recommendations that contribute to the  program’s success. The department’s comments relative to both of the  recommendations that it partially concurred with, as well as additional  comments, are discussed below.
For our recommendation associated with improving the program’s benefit  and cost estimates, DOD concurred with two of the recommendation’s three  parts, but it did not concur with one part—ensuring that future cost estimates  reflect the risk of not having cost data for comparable programs. While  acknowledging that the program had limited cost data from comparable  programs on which to base its cost estimate, DOD stated that an uncertainty  analysis had been applied to the estimate to account for the risk associated  with not having such data. The department further stated that actual  experience on the program will continue to be used to refine the program’s  cost estimating methodology. While we support DOD’s stated commitment to  using actual program cost experience in deriving future estimates, we do not  agree that the latest estimate accounted for the risk of not having cost data  from comparable programs. We examined the uncertainty analysis as part of  our review, and found that it did not recognize this risk. Moreover, DOD’s  comments offered no new evidence to the contrary.
For our recommendation associated with improving the program’s  schedule estimating, DOD concurred with four of the recommendation’s  five parts, and partially concurred with one part—ensuring that the  schedule defines a critical path that integrates all releases. In taking this  position, the department stated that a critical path has been established for  each release rather than across all three releases, and it attributes this to  the size and complexity of the program. We do not take issue with either  of these statements, as they are already recognized in our report. However,  DOD offers no new information in its comments. Further, our report also  recognizes that to be successful, large and complex programs that involve  thousands of activities need to ensure that their schedules integrate these  activities. In this regard, we support the department’s commitment to  explore the feasibility of implementing this part of our recommendation.
In addition, while stating that it concurred with all parts of our  recommendation associated with improving the program’s use of EVM,  DOD nevertheless provided additional comments as justification for  having not conducted an integrated baseline review on Release 1.0.  Specifically, it stated that when it rebaselined this release in December  2006, the release’s development activities were essentially complete and  the release was in the latter stages of testing. Further, it stated that the  risks associated with the Release 1.0 schedule were assessed 3 months  after this rebaselining, and these risks were successfully mitigated. To  support this statement, it said that Release 1.0 achieved its “Go-Live” as  scheduled at NAVAIR. We do not agree with these comments for several  reasons. First, at the time of the rebaselining, about 9 months of scheduled  Release 1.0 development remained, and thus the release was far from  complete. Moreover, the significance of the amount of work that  remained, and still remains today on Release 1.0 is acknowledged in  DOD’s own comment that the scheduled integrated baseline review for  Release 1.1 will also include remaining Release 1.0 work. Second, the  Release 1.0 contract was awarded in January 2006, and DOD’s own  guidance requires that an integrated baseline review be conducted within  6 months of a contract’s award. Third, although DOD states that the  program achieved “Go-Live” as scheduled on October 1, 2007, the program  achieved initial operational capability 7 months later than established in  the December 2006 baseline.
In addition to these comments, the department also described actions  under way or planned to address our recommendations. We support the  actions described, as they are consistent with the intent of our  recommendations. If fully and properly implemented, these actions will go  a long way in addressing the management control weaknesses that our  recommendations are aimed at correcting.
We are sending copies of this report to interested congressional  committees; the Director, Office of Management and Budget; the  Congressional Budget Office; the Secretary of Defense; and the  Department of Defense Office of the Inspector General. We also will make  copies available to others upon request. In addition, the report will be  available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-3439 or hiter@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix III.
section_title: Appendix I: Objective, Scope, and Methodology
paragraphs: Our objective was to determine whether the Department of the Navy is  effectively implementing information technology management controls on  the Navy Enterprise Resource Planning (Navy ERP) program. To  accomplish this, we focused on the first increment of Navy ERP and the  following management areas (1) architectural alignment, (2) economic  justification, (3) earned value management (EVM), (4) requirements  management, and (5) risk management.
To determine whether Navy ERP was aligned with the Department of  Defense’s (DOD) federated business enterprise architecture (BEA), we  reviewed the program’s BEA compliance assessments and system  architecture products, as well as Versions 4.0, 4.1, and 5.0 of the BEA, and  compared them with the BEA compliance requirements described in the  Fiscal Year 2005 Defense Authorization Act and DOD’s BEA compliance  guidance, and we evaluated the extent to which the compliance  assessments addressed all relevant BEA products. We also determined the  extent to which the program-level architecture documentation supported  the BEA compliance assessments. We obtained documentation, such as  the BEA compliance assessments from the Navy ERP and Global Combat  Support System—Marine Corps programs, as well as the Air Force’s  Defense Enterprise Accounting and Management System and Air Force  Expeditionary Combat Support System programs. We then compared  these assessments to identify potential redundancies or opportunities for  reuse and determined if the compliance assessments examined duplication  across programs, and if the tool that supports these assessments is being  used to identify such duplication. In doing so, we interviewed program  officials and officials from the Department of the Navy, Office of the Chief  Information Officer and reviewed recent GAO reports to determine the  extent to which the programs were assessed for compliance against the  Department of the Navy enterprise architecture. We also interviewed  program officials and officials from the Business Transformation Agency  and the Department of the Navy, including the logistics Functional Area  Manager, and obtained guidance documentation from these officials to  determine the extent to which the compliance assessments were subject  to oversight or validation.
To determine whether the program had economically justified its  investment in Navy ERP, we reviewed the latest economic analysis to  determine the basis for the cost and benefit estimates. This included  evaluating the analysis against Office of Management and Budget guidance  and GAO’s Cost Assessment Guide. In doing so, we interviewed cognizant  program officials, including the program manager and cost analysis team,  regarding their respective roles, responsibilities, and actual efforts in  developing and/or reviewing the economic analysis. We also interviewed  officials at the Office of Program Analysis and Evaluation and Naval  Center for Cost Analysis as to their respective roles, responsibilities, and  actual efforts in developing and/or reviewing the economic analysis. We  did not verify the validity of the source data used to calculate estimated  benefits, such as those data used to determine the yearly costs associated  with legacy systems planned for retirement.
To determine the extent to which the program had effectively  implemented EVM, we reviewed relevant documentation, such as contract  performance reports, acquisition program baselines, performance  measurement baseline, and schedule estimates and compared them with  DOD policies and guidance. To identify trends that could affect the  program baseline in the future, we assessed cost and schedule  performance and, in doing so, we applied earned value analysis  techniques to data from contract performance reports. We compared the  cost of work completed with the budgeted costs for scheduled work over  a 17-month period, from January 2007 to May 2008, to show trends in cost  and schedule performance. We also used data from the reports to estimate  the likely costs at completion of the program through established earned  value formulas. This resulted in three different values, with the middle  value being the most likely. We checked EVM data to see if there were any  mathematical errors or inconsistencies that would lead to the data being  unreliable. We interviewed cognizant officials from the Naval Air Systems  Command and program officials to determine whether the program had  conducted an integrated baseline review, whether the EVM system had  been validated against industry guidelines, and to better understand the  anomalies in the EVM data and determine what outside surveillance was  being done to ensure that the industry standards are being met. We also  reviewed the program’s schedule estimates and compared them with  relevant best practices to determine the extent to which they reflect key  estimating practices that are fundamental to having a reliable schedule. In  doing so, we interviewed cognizant program officials to discuss their use  of best practices in creating the program’s current schedule.
To determine the extent to which the program has effectively implemented  requirements management, we reviewed relevant program documentation,  such as the program management plan and baseline list of requirements.  To determine the extent to which the program has maintained traceability  backward to high-level business operation requirements and system  requirements, and forward to system design specifications, and test plans,  we randomly selected 60 program requirements and traced them both  backward and forward. This sample was designed with a 5 percent  tolerable error rate at the 95 percent level of confidence so that, if we  found 0 problems in our sample, we could conclude statistically that the  error rate was less than 5 percent. In addition, we interviewed program  officials involved in the requirements management process to discuss their  roles and responsibilities for managing requirements.
To determine the extent to which the program implemented risk  management, we reviewed relevant risk management documentation, such  as the program’s risk management plan and risk database reports  demonstrating the status of the program’s major risks and compared the  program office’s activities with DOD acquisition management guidance  and related industry practices. We also reviewed the program’s mitigation  process with respect to key risks to determine the extent to which these  risks were effectively managed. In doing so, we interviewed cognizant  program officials, such as the program manager and risk manager, to  discuss their roles and responsibilities and obtain clarification on the  program’s approach to managing risks associated with acquiring and  implementing Navy ERP.
We conducted this performance audit at DOD offices in the Washington,  D.C., metropolitan area and Annapolis, Md., from June 2007 to September  2008, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objective. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objective.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, key contributors to this report  were Neelaxi Lakhmani, Assistant Director; Monica Anatalio; Harold  Brumm; Neil Doherty; Cheryl Dottermusch; Nancy Glover; Mustafa  Hassan; Michael Holland; Ethan Iczkovitz; Anh Le; Josh Leiling; Emily  Longcore; Lee McCracken; Madhav Panwar; Karen Richey; Melissa  Schermerhorn; Karl Seifert; Sushmita Srikanth; Jonathan Ticehurst; and  Adam Vodraska.