DOD Business Systems Modernization: Key Marine Corps System Acquisition Needs to Be Better Justified, Defined, and Managed
Published: 2008-07-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO has designated the Department of Defense's (DOD) business systems modernization as a high-risk program because, among other things, it has been challenged in implementing key information technology (IT) management controls on its thousands of business systems. The Global Combat Support System-Marine Corps program is one such system. Initiated in 2003, the program is to modernize the Marine Corps logistics systems. The first increment is to cost about $442 million and be deployed in fiscal year 2010. GAO was asked to determine whether the Department of the Navy is effectively implementing IT management controls on this program. To accomplish this, GAO analyzed the program's implementation of several key IT management disciplines, including economic justification, earned value management, risk management, and system quality measurement.
section_title: What GAO Found
paragraphs: DOD has not effectively implemented key IT management controls provided for in DOD and related acquisition guidance on this program. If implemented effectively, these and other IT management disciplines increase the likelihood that a given system investment will produce the right solution to fill a mission need and that this system solution will be acquired and deployed in a manner that maximizes the chances of delivering promised system capabilities and benefits on time and within budget. Neither of these outcomes is being fully realized on this program, as evidenced by the fact that its first increment has already slipped more than 3 years and is expected to cost about $193 million more than envisioned. These slippages and cost overruns can be attributed in part to the management control weaknesses discussed in this report and summarized below. Moreover, additional slippages and overruns are likely if these and other IT management weaknesses are not addressed. Investment in the system has not been economically justified on the basis of reliable estimates of both benefits and costs. Specifically, while projected benefits were risk-adjusted to compensate for limited data and questionable assumptions, the cost side of the benefit/cost equation is not sufficiently reliable because it was not derived in accordance with key cost estimating practices. In particular, it was not based on historical data from similar programs and it did not account for schedule risks, both of which are needed for the estimate to be considered accurate and credible. Earned value management that the program uses to measure progress has not been adequately implemented. Specifically, the schedule baseline against which the program gauges progress is not based on key estimating practices provided for in federal guidance, such as assessing schedule risks and allocating schedule reserves to address these risks. As a result, program progress cannot be adequately measured, and likely program completion dates cannot be projected based on actual work performed. Some significant program risks have not been adequately managed. While a well-defined risk management plan and supporting process have been put in place, the process has not always been followed. Specifically, mitigation steps for significant risks either have not been implemented or proved ineffective, allowing the risks to become actual problems. The data needed to produce key indicators of system quality, such as trends in the volume of significant and unresolved problems and change requests, are not being collected. Without such data, it is unclear whether the system is becoming more or less mature and stable. The reasons for these weaknesses range from limitations of DOD guidance and tools, to not collecting relevant data. Until they are addressed, DOD is at risk of delivering a solution that does not cost-effectively support mission operations and falls short of cost, schedule, and capability expectations.

section_title: Letter
paragraphs: For decades, the Department of Defense (DOD) has been challenged in  modernizing its timeworn business systems. In 1995, we designated DOD’s  business systems modernization program as high risk and continue to do  so today. Our reasons include the modernization’s large size, complexity,  and critical role in addressing other long-standing transformation and  financial management challenges. Other reasons are that DOD has yet to  institutionalize key system modernization management controls, and it has  not demonstrated the ability to consistently deliver promised system  capabilities and benefits on time and within budget.
Nevertheless, DOD continues to invest billions of dollars in thousands of  business systems, including about a hundred that the department has  labeled as business transformational programs, 12 of which account for  about 50 percent of these programs’ costs. The Global Combat Support  System-Marine Corps (GCSS-MC) is one such program. Initiated in 2003,  GCSS-MC is to modernize the Marine Corps logistics systems and thereby  provide decision makers with timely and complete logistics information to  support the warfighter. As envisioned, the program consists of a series of  major increments, the first of which is expected to cost approximately  $442 million and be fully deployed in fiscal year 2010.
As agreed, our objective was to determine whether the Department of the  Navy is effectively implementing information technology (IT) management  controls on GCSS-MC. To accomplish this, we focused on the first  increment of GCSS-MC by analyzing a range of program documentation  and interviewing cognizant officials relative to the following management  areas: architectural alignment, economic justification, earned value  management, requirements management, risk management, and system  quality measurement. We conducted our performance audit from June  2007 to July 2008, in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objective. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objective. Additional details  on our objective, scope, and methodology are in appendix I.
subsections: section_title: Background
paragraphs: The Department of the Navy (DON) is a major component of DOD,  consisting of two uniformed services: the Navy and the Marine Corps. The  Marine Corps’ primary mission is to serve as a “total force in readiness” by  responding quickly in a wide spectrum of responsibilities, such as attacks  from sea to land in support of naval operations, air combat, and security of  naval bases. As the only service that operates in three dimensions—in the  air, on land, and at sea, the Marine Corps must be equipped to provide  rapid and precise logistics support to operating forces in any  environment.
The Marine Corps’ many and dispersed organization components rely  heavily on IT to perform their respective mission-critical operations and  related business functions, such as logistics and financial management.  For fiscal year 2008, the Marine Corps budget for IT business systems is  about $1.3 billion, of which $746 million (57 percent) is for operations and  maintenance of existing systems and $553 million (43 percent) is for  systems development and modernization. Of the approximately 904  systems in DON’s current inventory, the Marine Corps accounts for 81, or  about 9 percent, of the total. The GCSS-MC is one such system investment.  According to DOD, it is intended to address the Marine Corps’ long- standing problem of stove-piped logistics systems that collectively provide  limited data visibility and access, are unable to present a common,  integrated logistics picture in support of the warfighter, and do not  provide important decision support tools.
subsections: section_title: GCSS-MC: A Brief Description
paragraphs: In September 2003, the Marine Corps initiated GCSS-MC to (1) deliver  integrated functionality across the logistics areas (e.g., supply and  maintenance), (2) provide timely and complete logistics information to  authorized users for decision making, and (3) provide access to logistics  information and applications regardless of location. The system is  intended to function in three operational environments—deployed  operations (i.e., in theater of war or exercise environment on land or at  sea), in-transit, and in garrison. When GCSS-MC is fully implemented, it is  to support about 33,000 users located around the world.
GCSS-MC is being developed in a series of large and complex increments  using commercially available enterprise resource planning (ERP)  software and hardware components. The first increment is currently the  only funded portion of the program and is to provide a range of asset  management capabilities, including    planning inventory requirements to support current and future  requesting and tracking the status of products (e.g., supplies and  personnel) and services (e.g., maintenance and engineering);    allocating resources (e.g., inventory, warehouse capacity, and  personnel) to support unit demands for specific products; and    scheduling maintenance resources (e.g., manpower, equipment, and  supplies) for specific assets, such as vehicles.
Additionally, the first increment is to replace four legacy systems  scheduled for retirement in 2010. Table 1 describes these four systems.
Future increments are to provide additional functionality (e.g.,  transportation and wholesale inventory management), enhance existing  functionality, and potentially replace up to 44 additional legacy systems.
The program office estimates the total life cycle cost for the first  increment to be about $442 million, including $169 million for acquisition  and $273 million for operations and maintenance. The total life cycle cost  of the entire program has not yet been determined because future  increments are currently in the planning stages and have not been defined.  As of April 2008, the program office reported that approximately $125  million has been spent on the first increment.
section_title: Program Oversight and Management Roles and Responsibilities
paragraphs: To manage the acquisition and deployment of GCSS-MC, the Marine Corps  established a program management office within the Program Executive  Office for Executive Information Systems. The program office is led by the  Program Manager who is responsible for managing the program’s scope  and funding and ensuring that the program meets its objectives. To  accomplish this, the program office is responsible for key acquisition  management controls, such as architectural alignment, economic  justification, EVM, requirements management, risk management, and  system quality measurement. In addition, various DOD and DON  organizations share program oversight and review activities relative to  these and other acquisition management controls. A listing of key entities  and their roles and responsibilities is in table 2.
section_title: Overview of GCSS-MC’s Status
paragraphs: The program reports that the first increment of GCSS-MC is currently in  the system development and demonstration phase of the defense  acquisition system (DAS). The DAS consists of five key program life cycle  phases and three related milestone decision points. These five phases and  related milestones are described along with a summary of key program  activities completed during, or planned, for each phase as follows:  1.  Concept refinement: The purpose of this phase is to refine the initial  system solution (concept) and create a strategy for acquiring the  investment solution. During this phase, the program office defined the  acquisition strategy and analyzed alternative solutions. The first  increment completed this phase on July 23, 2004, which was 1 month  later than planned, and the MDA approved a Milestone A decision to  move to the next phase.  2.  Technology development: The purpose of this phase is to determine  the appropriate set of technologies to be integrated into the investment  solution by iteratively assessing the viability of various technologies  while simultaneously refining user requirements. During this phase,  the program office selected Oracle’s E-Business Suite as the  commercial off-the-shelf ERP software. In addition, the program office  awarded Accenture the system integration contract to, among other  things, configure the software, establish system interfaces, and  implement the new system. This system integration contract was  divided into two phases—Part 1 for the planning, analysis, and  conceptual design of the solution and Part 2 for detailed design, build,  test, and deployment of the solution. The program office did not  exercise the option for Part 2 of the contract to Accenture and shortly  thereafter established a new program baseline in June 2006. In  November 2006, it awarded a time-and-materials system integration  contract valued at $28.4 million for solution design to Oracle. The first  increment completed this phase on June 8, 2007, which was 25 months  later than planned due in part to contractual performance shortfalls,  and the MDA approved a Milestone B decision to move to the next  phase.  3.  System development and demonstration: The purpose of this phase is  to develop the system and demonstrate through developer testing that  the system can function in its target environment. During this phase,  the program office extended the solution design contract and  increased funding to $67.5 million due, in part, to delays in completing  the detailed design activities. As a result, the program office has not  yet awarded the next contract (which includes both firm-fixed-price  and time-and-materials task orders) for build and testing activities,  originally planned for July 2007. Instead, it entered what it termed a  “transition period” to complete detailed design activities. According to  the program’s baseline, the MDA is expected to approve a Milestone C  decision to move to the next phase in October 2008. However, program  officials stated that Milestone C is now scheduled for April 2009, which  is 35 months later than originally planned.  4.  Production and deployment: The purpose of this phase is to achieve  an operational capability that satisfies the mission needs, as verified  through independent operational test and evaluation, and implement  the system at all applicable locations. The program office plans to  award a separate firm-fixed-price plus award fee contract for these  activities with estimated costs yet to be determined.  5.  Operations and support: The purpose of this phase is to operationally  sustain the system in the most cost-effective manner over its life cycle.  The details of this phase have not yet been defined.
Overall, GCSS-MC was originally planned to reach full operational  capability (FOC) in fiscal year 2007 at an estimated cost of about $126  million over a 7-year life cycle. This cost estimate was later revised in  2005 to about $249 million over a 13-year life cycle. However, the  program now expects to reach FOC in fiscal year 2010 at a cost of about  $442 million over a 12-year life cycle. Figures 1 and 2 show the program’s  current status against original milestones and original, revised, and current  cost estimates.
section_title: Use of IT Acquisition Management Best Practices Maximizes Chances for Success
paragraphs: Acquisition best practices are tried and proven methods, processes,  techniques, and activities that organizations define and use to minimize  program risks and maximize the chances of a program’s success. Using  best practices can result in better outcomes, including cost savings,  improved service and product quality, and a better return on investment.  For example, two software engineering analyses of nearly 200 systems  acquisition projects indicate that teams using systems acquisition best  practices produced cost savings of at least 11 percent over similar projects  conducted by teams that did not employ the kind of rigor and discipline  embedded in these practices. In addition, our research shows that best  practices are a significant factor in successful acquisition outcomes and  increase the likelihood that programs and projects will be executed within  cost and schedule estimates.
We and others have identified and promoted the use of a number of best  practices associated with acquiring IT systems. See table 3 for a  description of several of these activities.
section_title: Prior GAO Reviews Have Identified IT Acquisition Management Weaknesses in DOD’s Business System Investments
paragraphs: We have previously reported that DOD has not effectively managed a  number of business system investments. Among other things, our reviews  of individual system investments have identified weaknesses in such areas  as architectural alignment and informed investment decision making,  which are also the focus areas of the Fiscal Year 2005 National Defense  Authorization Act business system provisions. Our reviews have also  identified weaknesses in other system acquisition and investment  management areas—such as EVM, economic justification, requirements  management, risk management, and test management.
Most recently, for example, we reported that the Army’s approach to  investing about $5 billion over the next several years in its General Fund  Enterprise Business System, Global Combat Support System-Army  Field/Tactical, and Logistics Modernization Program did not include  alignment with Army enterprise architecture or use a portfolio-based  business system investment review process. Moreover, we reported that  the Army did not have reliable analyses, such as economic analyses, to  support its management of these programs. We concluded that until the  Army adopts a business system investment management approach that  provides for reviewing groups of systems and making enterprise decisions  on how these groups will collectively interoperate to provide a desired  capability, it runs the risk of investing significant resources in business  systems that do not provide the desired functionality and efficiency.  Accordingly, we made recommendations aimed at improving the  department’s efforts to achieve total asset visibility and enhancing its  efforts to improve its control and accountability over business system  investments. The department agreed with our recommendations.
We also reported that DON had not, among other things, economically  justified its ongoing and planned investment in the Naval Tactical  Command Support System (NTCSS) and had not invested in NTCSS  within the context of a well-defined DOD or DON enterprise architecture.  In addition, we reported that DON had not effectively performed key  measurement, reporting, budgeting, and oversight activities and had not  adequately conducted requirements management and testing activities. We  concluded that, without this information, DON could not determine  whether NTCSS, as defined, and as being developed, is the right solution  to meet its strategic business and technological needs. Accordingly, we  recommended that the department develop the analytical basis to  determine if continued investment in the NTCSS represents prudent use of  limited resources and to strengthen management of the program,  conditional upon a decision to proceed with further investment in the  program. The department largely agreed with these recommendations.
In addition, we reported that the Army had not defined and developed its  Transportation Coordinators’ Automated Information for Movements  System II (TC-AIMS II)—a joint services system with the goal of helping to  manage the movement of forces and equipment within the United States  and abroad—in the context of a DOD enterprise architecture. We also  reported that the Army had not economically justified the program on the  basis of reliable estimates of life cycle costs and benefits and had not  effectively implemented risk management. As a result, we concluded that  the Army did not know if its investment in TC-AIMS II, as planned, is  warranted or represents a prudent use of limited DOD resources.  Accordingly, we recommended that DOD, among other things, develop the  analytical basis needed to determine if continued investment in TC-AIMS  II, as planned, represents prudent use of limited defense resources. In  response, the department largely agreed with our recommendations and  has since reduced the program’s scope by canceling planned investments.
section_title: Key DOD and Related IT Acquisition Management Controls Have Not Been Fully Implemented on GCSS-MC
paragraphs: DOD IT-related acquisition policies and guidance, along with other  relevant guidance, provide an acquisition management control framework  within which to manage business system programs like GCSS-MC.  Effective implementation of this framework can minimize program risks  and better ensure that system investments are defined in a way to  optimally support mission operations and performance, as well as deliver  promised system capabilities and benefits on time and within budget. Thus  far, GCSS-MC has not been managed in accordance with key aspects of  this framework, which has already contributed to more than 3 years in  program schedule delays and about $193 million in cost increases. These  IT acquisition management control weaknesses include    compliance with DOD’s federated BEA not being sufficiently    expected costs not being reliably estimated;    earned value management not being adequately implemented;    system requirements not always being effectively managed, although  this has recently improved;    key program risks not being effectively managed; and    key system quality measures not being used.
The reasons that these key practices have not been sufficiently executed  include limitations in the applicable DOD guidance and tools, and not  collecting relevant data, each of which is described in the applicable  sections of this report. By not effectively implementing these key IT  acquisition management controls, the program has already experienced  sizeable schedule and cost increases, and it is at increased risk of (1) not  being defined in a way that best meets corporate mission needs and  enhances performance and (2) costing more and taking longer than  necessary to complete.
subsections: section_title: GCSS-MC Compliance with DOD’s Federated BEA Has Not Been Sufficiently Demonstrated
paragraphs: DOD and federal guidance recognize the importance of investing in  business systems within the context of an enterprise architecture.  Moreover, the 2005 National Defense Authorization Act requires that  defense business systems be compliant with DOD’s federated BEA. Our  research and experience in reviewing federal agencies show that not  making investments within the context of a well-defined enterprise  architecture often results in systems that are duplicative, are not well  integrated, are unnecessarily costly to interface and maintain, and do not  optimally support mission outcomes.
To its credit, the program office has followed DOD’s BEA compliance  guidance. However, this guidance does not adequately provide for  addressing all relevant aspects of BEA compliance. Moreover, DON’s  enterprise architecture, which is a major component of DOD’s federated  BEA, as well as key aspects of DOD’s corporate BEA, have yet to be  sufficiently defined to permit thorough compliance determinations. In  addition, current policies and guidance do not require DON investments to  comply with its enterprise architecture. This means that the department  does not have a sufficient basis for knowing if GCSS-MC has been defined  to optimize DON and DOD business operations. Each of these architecture  alignment limitations is discussed as follows:    The program’s compliance assessments did not include all relevant  architecture products. In particular, the program did not assess  compliance with the BEA’s technical standards profile, which outlines, for  example, the standards governing how systems physically communicate  with other systems and how they secure data from unauthorized access.  This is particularly important because systems, like GCSS-MC, need to  employ common standards in order to effectively and efficiently share  information with other systems. A case in point is GCSS-MC and the Navy  Enterprise Resource Planning program. Specifically, GCSS-MC has  identified 13 technical standards that are not in the BEA technical  standards profile, and Navy Enterprise Resource Planning has identified  25 technical standards that are not in the profile. Of these, some relate to  networking protocols, which could limit information sharing between  these and other systems.
In addition, the program office did not assess compliance with the BEA  products that describe system characteristics. This is important because  doing so would create a body of information about programs that could be  used to identify common system components and services that could  potentially be shared by the programs, thus avoiding wasteful duplication.  For example, our analysis of GCSS-MC program documentation shows  that they contain such system functions as receiving goods, taking  physical inventories, and returning goods, which are also system functions  cited by the Navy Enterprise Resource Planning program. However,  because compliance with the BEA system products was not assessed, the  extent to which these functions are potentially duplicative was not  considered.
Furthermore, the program office did not assess compliance with BEA  system products that describe data exchanges among systems. As we  previously reported, establishing and using standard system interfaces is a  critical enabler to sharing data. For example, GCSS-MC program  documentation indicates that it is to exchange order and status data with  other systems. However, the program office has not fully developed its  architecture product describing these exchanges and thus does not have  the basis for understanding how its approach to exchanging information  differs from that of other systems that it is to interface with. Compliance  against each of these BEA products was not assessed because DOD’s  compliance guidance does not provide for doing so and, according to BTA  and program officials, some BEA and program-level architecture products  are not sufficiently defined. According to these officials, BTA plans to  continue to define these products as the BEA evolves.
The compliance assessment was not used to identify potential areas of  duplication across programs, which DOD has stated is an explicit goal of  its federated BEA and associated investment review and decision-making  processes. More specifically, even though the compliance guidance  provides for assessing programs’ compliance with the BEA product that  defines DOD operational activities, and GCSS-MC was assessed for  compliance with this product, the results were not used to identify  programs that support the same operational activities and related business  processes. Given that the federated BEA is intended to identify and avoid  not only duplications within DOD components, but also between DOD  components, it is important that such commonality be addressed. For  example, program-level architecture products for GCSS-MC and Navy  Enterprise Resource Planning, as well as two Air Force programs (Defense  Enterprise Accounting and Management System-Air Force and the Air  Force Expeditionary Combat Support System) show that each supports at  least six of the same BEA operational activities (e.g., conducting physical  inventory, delivering property, and services) and three of these four  programs support at least 18 additional operational activities (e.g.,  performing budgeting, managing receipt, and acceptance). As a result,  these programs may be investing in duplicative functionality. Reasons for  not doing so were that compliance guidance does not provide for such  analyses to be conducted, and programs have not been granted access  rights to use this functionality.
The program’s compliance assessment did not address compliance against  the DON’s enterprise architecture, which is one of the biggest members of  the federated BEA. This is particularly important given that DOD’s  approach to fully satisfying the architecture requirements of the 2005  National Defense Authorization Act is to develop and use a federated  architecture in which component architectures are to provide the  additional details needed to supplement the thin layer of corporate  policies, rules, and standards included in the corporate BEA. As we  recently reported, the DON’s enterprise architecture is not mature  because, among other things, it is missing a sufficient description of its  current and future environments in terms of business and  information/data. However, certain aspects of an architecture nevertheless  exist and, according to DON, these aspects will be leveraged in its efforts  to develop a complete enterprise architecture. For example, the  FORCEnet architecture documents DON’s technical infrastructure.  Therefore, opportunities exist for DON to assess its programs in relation  to these architecture products and to understand where its programs are  exposed to risks because products do not exist, are not mature, or are at  odds with other DON programs. According to DOD officials, compliance  with the DON architecture was not assessed because DOD compliance  policy is limited to compliance with the corporate BEA, and the DON  enterprise architecture has yet to be sufficiently developed.
The program’s compliance assessment was not validated by DOD or DON  investment oversight and decision-making authorities. More specifically,  neither the DOD IRBs nor the DBSMC, nor the BTA in supporting both of  these investment oversight and decision-making authorities, reviewed the  program’s assessments. According to BTA officials, under DOD’s tiered  approach to investment accountability, these entities are not responsible  for validating programs’ compliance assessments. Rather, this is a  component responsibility, and thus they rely on the military departments  and defense agencies to validate the assessments.
However, the DON Office of the CIO, which is responsible for  precertifying investments as compliant before they are reviewed by the  IRB, did not evaluate any of the programs’ compliance assessments.  According to CIO officials, they rely on Functional Area Managers to  validate a program’s compliance assessments. However, no DON policy or  guidance exists that describes how the Functional Area Managers should  conduct such validations.
Validation of program assessments is further complicated by the absence  of information captured in the assessment tool about what program  documentation or other source materials were used by the program office  in making its compliance determinations. Specifically, the tool is only  configured, and thus was only used, to capture the results of a program’s  comparison of program architecture products to BEA products. Thus, it  was not used to capture the system products used in making these  determinations.
In addition, the program office did not develop certain program-level  architecture products that are needed to support and validate the  program’s compliance assessment and assertions. According to the  compliance guidance, program-level architecture products, such as those  defining information exchanges and system data requirements are not  required to be used until after the system has been deployed. This is  important because waiting until the system is deployed is too late to avoid  the costly rework required to address areas of noncompliance. Moreover,  it is not consistent with other DOD guidance, which states that program- level architecture products that describe, for example, information  exchanges, should be developed before a program begins system  development.
The limitations in existing BEA compliance-related policy and guidance,  the supporting compliance assessment tool, and the federated BEA, puts  programs like GCSS-MC at increased risk of being defined and  implemented in a way that does not sufficiently ensure interoperability  and avoid duplication and overlap. We currently have a review under way  for the Senate Armed Services Committee, Subcommittee on Readiness  and Management Support, which is examining multiple programs’  compliance with the federated BEA.
section_title: Investment in GCSS-MC Was Not Economically Justified Using Reliable Estimates of Costs
paragraphs: The investment in the first increment of GCSS-MC has not been  economically justified on the basis of reliable analyses of estimated system  costs over the life of the program. According to the program’s economic  analysis, the first increment will have an estimated life cycle cost of about  $442 million and deliver an estimated $1.04 billion in risk-adjusted  estimated benefits during this same life cycle. This equates to a net present  value of about $688 million. While the most recent cost estimate was  derived using some effective estimating practices, it did not make use of  other practices that are essential to having an accurate and credible  estimate. As a result, the Marine Corps does not have a sufficient basis for  deciding whether GCSS-MC, as defined, is the most cost-effective solution  to meeting its mission needs, and it does not have a reliable basis against  which to measure cost performance.
subsections: section_title: The Cost Estimate Was Not Derived Using Practices Necessary for an Accurate and Credible Estimate
paragraphs: A reliable cost estimate is critical to the success of any IT program, as it  provides the basis for informed investment decision making, realistic  budget formulation and program resourcing, meaningful progress  measurement, proactive course correction, and accountability for results.  According to the Office of Management and Budget (OMB), programs  must maintain current and well-documented cost estimates, and these  estimates must encompass the full life cycle of the program. OMB states  that generating reliable cost estimates is a critical function necessary to  support OMB’s capital programming process. Without reliable estimates,  programs are at increased risk of experiencing cost overruns, missed  deadlines, and performance shortfalls.
Our research has identified a number of practices for effective program  cost estimating. We have issued guidance that associates these practices  with four characteristics of a reliable cost estimate. These four  characteristic are specifically defined as follows:    Comprehensive: The cost estimates should include both government and  contractor costs over the program’s full life cycle, from the inception of  the program through design, development, deployment, and operation and  maintenance, to retirement. They should also provide a level of detail  appropriate to ensure that cost elements are neither omitted nor double  counted and include documentation of all cost-influencing ground rules  and assumptions.
Well-documented: The cost estimates should have clearly defined purposes  and be supported by documented descriptions of key program or system  characteristics (e.g., relationships with other systems, performance  parameters). Additionally, they should capture in writing such things as  the source data used and their significance, the calculations performed  and their results, and the rationale for choosing a particular estimating  method or reference. Moreover, this information should be captured in  such a way that the data used to derive the estimate can be traced back to,  and verified against, their sources. The final cost estimate should be  reviewed and accepted by management on the basis of confidence in the  estimating process and the estimate produced by the process.
Accurate: The cost estimates should provide for results that are unbiased  and should not be overly conservative or optimistic (i.e., should represent  the most likely costs). In addition, the estimates should be updated  regularly to reflect material changes in the program, and steps should be  taken to minimize mathematical mistakes and their significance. The  estimates should also be grounded in a historical record of cost estimating  and actual experiences on comparable programs.
Credible: The cost estimates should discuss any limitations in the analysis  performed that are due to uncertainty or biases surrounding data or  assumptions. Further, the estimates’ derivation should provide for varying  any major assumptions and recalculating outcomes based on sensitivity  analyses, and the estimates’ associated risks and inherent uncertainty  should be disclosed. Also, the estimates should be verified based on cross- checks using other estimating methods and by comparing the results with  independent cost estimates.
The $442 million life cycle cost estimate for the first increment reflects  many of the practices associated with a reliable cost estimate, including all  practices associated with being comprehensive and well-documented, and  several related to being accurate and credible. (See table 4.) However,  several important accuracy and credibility practices were not satisfied.
The cost estimate is comprehensive because it includes both the  government and contractor costs specific to development, acquisition  (nondevelopment), implementation, and operations and support over the  program’s 12-year life cycle. Moreover, the estimate clearly describes how  the various subelements are summed to produce the amounts for each  cost category, thereby ensuring that all pertinent costs are included, and  no costs are double counted. Lastly, cost-influencing ground rules and  assumptions, such as the program’s schedule, labor rates, and inflation  rates are documented.
The cost estimate is also well-documented in that the purpose of the cost  estimate was clearly defined, and a technical baseline has been  documented that includes, among others things, the relationships with  other systems and planned performance parameters. Furthermore, the  calculations and results used to derive the estimate are documented,  including descriptions of the methodologies used and traceability back to  source data (e.g., vendor quotes, salary tables). Also, the cost estimate was  reviewed both by the Naval Center for Cost Analysis and the Office of the  Secretary of Defense, Director for Program Analysis and Evaluation,  which ensures a level of confidence in the estimating process and the  estimate produced.
However, the estimate lacks accuracy because not all important practices  related to this characteristic were satisfied. Specifically, while the estimate  is grounded in documented assumptions (e.g., hardware refreshment every  5 years), and periodically updated to reflect changes to the program, it is  not grounded in historical experience with comparable programs. As  stated in our guide, estimates should be based on historical records of cost  and schedule estimates from comparable programs, and such historical  data should be maintained and used for evaluation purposes and future  estimates on other comparable programs. The importance of doing so is  evident by the fact that GCSS-MC’s cost estimate has increased by about  $193 million since July 2005, which program officials attributed to, among  other things, schedule delays, software development complexity, and the  lack of historical data from similar ERP programs. While the program  office did leverage historical cost data from other ERP programs,  including the Navy’s Enterprise Resource Planning Pilot programs and  programs at the Bureau of Prisons and the Department of Agriculture,  program officials told us that these programs’ scopes were not  comparable. For example, none of the programs had to utilize a  communication architecture as complex as the Marine Corps, which  officials cited as a significant factor in the cost increases and a challenge  in estimating costs.
The absence of analogous cost data for large-scale ERP programs is due in  part to the fact that DOD has not established a standardized cost element  structure for ERP programs that can be used to capture actual cost data.  According to officials with the Defense Cost and Resource Center, such  cost element structures are needed, along with a requirement for programs  to report on their costs, but approval and resources have yet to be gained  for either these structures or the reporting of their costs. Until a  standardized data structure exists, programs like GCSS-MC will continue  to lack a historical database containing cost estimates and actual cost  experiences of comparable ERP programs. This means that the current  and future GCSS-MC cost estimates will lack sufficient accuracy for  effective investment decision making and performance measurement.
Compounding the estimate’s limited accuracy are limitations in its  credibility. Specifically, while the estimate satisfies some of the key  practices for a credible cost estimate (e.g., confirming key cost drivers,  performing sensitivity analyses, having an independent cost estimate  prepared by the Naval Center for Cost Analysis that was within 4 percent  of the program’s estimate, and conducting a risk analysis that showed a  range of estimated costs of $411 million to $523 million), no risk analysis  was performed to determine the program schedule’s risks and associated  impact on the cost estimate. As described earlier in this report, the  program has experienced about 3 years in schedule delays and recently  experienced delays in completing the solution design phase. Therefore, the  importance of conducting a schedule risk analysis and using the results to  assess the variability in the cost estimate is critical for ensuring a credible  cost estimate. Program officials agreed that the program’s schedule is  aggressive and risky and that this risk was not assessed in determining the  cost estimate’s variability. Without doing so, the program’s cost estimate is  not credible, and thus the program is at risk of cost overruns as a result of  schedule delays.
section_title: Benefits Estimate Has Been Adjusted to Reflect Questionable Assumptions and Limited Data
paragraphs: Forecasting expected benefits over the life of a program is also a key  aspect of economically justifying an investment. OMB guidance  advocates economically justifying investments on the basis of net present  value. If net present value is positive, then the corresponding benefit-to- cost ratio will be greater than 1 (and vice versa). This guidance also  advocates updating the analyses over the life of the program to reflect  material changes in expected benefits, costs, and risks. Since estimates of  benefits can be uncertain because of the imprecision in both the  underlying data and modeling assumptions used, effects of this uncertainty  should be analyzed and reported. By doing this, informed investment  decision making can occur through the life of the program, and a baseline  can be established against which to compare the accrual of actual benefits  from deployed system capabilities.
The original benefit estimate for the first increment was based on  questionable assumptions and insufficient data from comparable  programs. The most recent economic analysis, dated January 2007,  includes monetized, yearly benefit estimates for fiscal years 2010–2019 in  three key areas—inventory reductions, reductions in inventory carrying  costs, and improvements in maintenance processes. Collectively, these  benefits totaled about $2.89 billion (not risk-adjusted). However, these  calculations were made using questionable assumptions and limited data.  For example,    The total value of the Marine Corps inventory needed to calculate  inventory reductions and reductions in carrying costs could not be  determined because of limitations with existing logistic systems.
The cost savings resulting from improvements in maintenance processes  were calculated based on assumptions from an ERP implementation in the  commercial sector that, according to program officials, is not comparable  in scope to GCSS-MC.
To account for the uncertainty inherent in the benefits estimate, the  program office performed a Monte Carlo simulation. According to the  program office, this risk analysis generated a discounted and risk-adjusted  benefits estimate of $1.04 billion. As a result of the $1.85 billion adjustment  to estimated benefits, the program office has a more realistic benefit  baseline against which to compare the accrual of actual benefits from  deployed system capabilities.
section_title: EVM Has Not Been Adequately Implemented
paragraphs: The program office has elected to implement EVM, which is a proven  means for measuring program progress and thereby identifying potential  cost overruns and schedule delays early, when they can be minimized. In  doing so, it has adopted a tailored EVM approach that focuses on  schedule. However, this schedule-focused approach has not been  effectively implemented because it is based on a baseline schedule that  was not derived using key schedule estimating practices. According to  program officials, the schedule was driven by an aggressive program  completion date established in response to direction from oversight  entities to complete the program as soon as possible. As a result, they said  that following these practices would have delayed this completion date.  Regardless, this means that the schedule baseline is not reliable, and  progress will likely not track to the schedule.
subsections: section_title: A Tailored EVM Approach Is Being Used to Measure Program Progress
paragraphs: The program office has adopted a tailored approach to performing EVM  because of the contract type being used. As noted earlier, the contract  types associated with GCSS-MC integration and implementation vary, and  include, for example, firm-fixed-price contracts and time-and-materials  contracts. Under a firm-fixed-price contract, the price is not subject to any  adjustment on the basis of the contractor’s cost experience in performing  the contract. For a time-and-materials contract, supplies or services are  acquired on the basis of (1) an undefined number of direct labor hours that  are paid at specified fixed hourly rates and (2) actual cost for materials.
According to DOD guidance, EVM is generally not encouraged for firm- fixed-price, level of effort, and time-and-material contracts. In these  situations, the guidance states that programs can use a tailored EVM  approach in which an integrated master schedule (IMS) is exclusively used  to provide visibility into program performance.
DON has chosen to implement this tailored EVM approach on GCSS-MC.  In doing so, it is measuring progress against schedule commitments, and  not cost commitments, using an IMS for each program phase. According to  program officials, the IMS describes and guides the execution of program  activities. Regardless of the approach used, effective implementation  depends on having a reliable IMS.
section_title: Schedule Baseline Was Not Derived in Accordance with Key Estimating Practices
paragraphs: The success of any program depends in part on having a reliable schedule  specifying when the program’s set of work activities will occur, how long  they will take, and how they are related to one another. As such, the  schedule not only provides a road map for the systematic execution of a  program, but it also provides the means by which to gauge progress,  identify and address potential problems, and promote accountability. Our  research has identified nine practices associated with effective schedule  estimating. These practices are (1) capturing key activities, (2)  sequencing key activities, (3) assigning resources to key activities, (4)  integrating key activities horizontally and vertically, (5) establishing the  duration of key activities, (6) establishing the critical path for key  activities, (7) identifying “float time” between key activities, (8)  distributing reserves to high-risk activities, and (9) performing a schedule  risk analysis.
The current IMS for the solution design and transition-to-build phase of  the first increment was developed using some of these practices. However,  it does not reflect several practices that are fundamental to having a  schedule baseline that provides a sufficiently reliable basis for measuring  progress and forecasting slippages. To the program office’s credit, its IMS  captures and sequences key activities required to complete the project,  integrates the tasks horizontally, and identifies the program’s critical path.  However, the program office is not monitoring the actual durations of  scheduled activities so that it can address the impact of any deviations on  later scheduled activities. Moreover, the schedule does not adequately  identify the resources needed to complete the tasks and is not integrated  vertically, meaning that multiple teams executing different aspects of the  program cannot effectively work to the same master schedule. Further, the  IMS does not adequately mitigate schedule risk by identifying float time  between key activities, introducing schedule reserve for high-risk  activities, or including the results of a schedule risk analysis. See table 5  for the results of our analyses relative to each of the nine practices.
According to program officials, they intend to begin monitoring actual  activity start and completion dates so that they can proactively adjust later  scheduled activities that are affected by deviations. However, they do not  plan to perform the three practices related to understanding and managing  schedule risk because doing so would likely extend the program’s  completion date, and they set this date to be responsive to direction from  DOD and DON oversight entities to complete the program as soon as  possible. In our view, not performing these practices does not allow the  inherent risks in meeting this imposed completion date to be proactively  understood and addressed. The consequence of omitting these practices is  a schedule that does not provide a reliable basis for performing EVM.
section_title: Requirements Management Weakness Was Recently Corrected
paragraphs: Well-defined and managed requirements are recognized by DOD guidance  as essential and can be viewed as a cornerstone of effective system  acquisition. One aspect of effective requirements management is  requirements traceability. By tracing requirements both backward from  system requirements to higher level business or operational requirements  and forward to system design specifications and test plans, the chances of  the deployed product satisfying requirements are increased, and the ability  to understand the impact of any requirement changes, and thus make  informed decision about such changes, is enhanced.
The program office recently strengthened its requirements traceability. In  November 2007, and again in February 2008, the program office was  unable to demonstrate for us that it could adequately trace its 1,375 system  requirements to both design specifications and test documentation.  Specifically, the program office was at that time using a tool called  DOORS®, which if implemented properly, allows each requirement to be  linked from its most conceptual definition to its most detailed definition,  as well as to design specifications and test cases. In effect, the tool  maintains the linkages among requirement documents, design documents,  and test cases even if requirements change. However, the system  integration contractor was not using the tool. Instead the contractor was  submitting its 244 work products, accompanied by spreadsheets that  linked each work product to one or more system requirements and test  cases. The program office then had to verify and validate the spreadsheets  and import and link each work product to the corresponding requirement  and test case in DOORS. Because of the sheer number of requirements and  work products and its potential to impact cost, schedule, and  performance, the program designated this approach as a medium risk. It  later closed the risk because the proposed mitigation strategy failed to  mitigate it, and it was realized as a high-priority program issue (i.e.,  problem).
According to program officials, this requirements traceability approach  resulted in time-consuming delays in approving the design work products  and importing and establishing links between these products and the  requirements in DOORS, in part because the work products were not  accompanied by complete spreadsheets that established the traceability.  As a result, about 30 percent of the contractor’s work products had yet to  be validated, approved, and linked to requirements when the design phase  was originally scheduled to be complete. Officials stated that the  contractor was not required to use DOORS because it was not experienced  with this tool and becoming proficient with it would have required time  and resources, thereby increasing both the program’s cost and schedule.  Ironically, however, not investing the time and resources to address the  limitations in the program’s traceability approach contributed to recent  delays in completing the solution design activities, and additional  resources had to be invested to address its requirements traceability  problems.
The program office now reports that it can trace requirements backward  and forward. In April 2008, we verified this by tracing 60 out of 61  randomly sampled requirements backward to system requirements and  forward to approved design specifications and test plans. Program officials  explained that the reason that we could not trace the one requirement was  that the related work products had not yet been approved. In addition,  they stated that there were additional work products that had yet to be  finalized and traced.
Without adequate traceability, the risk of a system not performing as  intended and requiring expensive rework is increased. To address its  requirements traceability weakness, program officials told us that they  now intend to require the contractor to use DOORS during the next phase  of the program (build and test). If implemented effectively, the new  process should address previous requirements traceability weaknesses  and thereby avoid a repeat of past problems.
section_title: Not All Program Risks Have Been Adequately Managed
paragraphs: Proactively managing program risks is a key acquisition management  control and, if done properly, can greatly increase the chances of  programs delivering promised capabilities and benefits on time and within  budget. To the program office’s credit, it has defined a risk management  process that meets relevant guidance. However, it has not effectively  implemented the process for all identified risks. As a result, these risks  have become actual program problems that have impacted the program’s  cost, schedule, and performance commitments.
DOD acquisition management guidance, as well as other relevant  guidance, advocates identifying facts and circumstances that can  increase the probability of an acquisition’s failing to meet cost, schedule,  and performance commitments and then taking steps to reduce the  probability of their occurrence and impact. In brief, effective risk  management consists of: (1) establishing a written plan for managing risks;  (2) designating responsibility for risk management activities; (3)  encouraging project-wide participation in the identification and mitigation  of risks; (4) defining and implementing a process that provides for the  identification, analysis, and mitigation of risks; and (5) examining the  status of identified risks in program milestone reviews.
The program office has developed a written plan for managing risks, and  established a process that together provide for the above cited risk  management practices, and it has followed many key aspects of its plan  and process. For example,    The Program Manager has been assigned overall responsibility for  managing risks. Also, individuals have been assigned ownership of each  risk, to include conducting risk analyses, implementing mitigation  strategies, and working with the risk support team.
The plan and process encourage project-wide participation in the  identification and mitigation of risks by allowing program staff to submit a  risk for inclusion in a risk database and take ownership of the risk and  the strategy for mitigating it. In addition, stakeholders can bring potential  risks to the Program Manager’s attention through interviews, where  potential risks are considered and evaluated.
The program office has thus far identified and categorized individual risks.
As of December 2007, the risk database contained 27 active risks—2 high,  15 medium, and 10 low.
Program risks are considered during program milestone reviews.
Specifically, our review of documentation for the Design Readiness  Review, a key decision point during the system development and  demonstration phase leading up to a Milestone C decision, showed that  key risks were discussed. Furthermore, the Program Manager reviews  program risks’ status through a risk watch list and bimonthly risk  briefings.
However, the program office has not consistently followed other aspects  of its process. For example, it did not perform key practices for identifying  and managing schedule risks, such as conducting a schedule risk  assessment and building in reserve time to its schedule. In addition,  mitigation steps for several key risks were either not performed in  accordance with the risk management strategy, or risks that were closed  as having been mitigated were later found to be actual program issues (i.e.,  problems).
For 25 medium risks in the closed risk database, as of February 2008, 4  were closed because mitigation steps were not performed in accordance  with the strategy and the risks ultimately became actual issues. Examples  from these medium risks are as follows:  In one case, the mitigation strategy was for the contractor to deliver  certain design documents that were traced to system requirements and to  do so before beginning the solution build phase. The design documents,  however, were not received in accordance with the mitigation strategy.  Specifically, program officials told us that the design documents contained  inaccuracies or misinterpretations of the requirements and were not  completed on time because of the lack of resources to correct these  problems. As a result, the program experienced delays in completing its  solution design activities.
In another case, the mitigation strategy included creating the  documentation needed to execute the contract for monitoring the build  phase activities. However, the mitigation steps were not performed due to,  among other things, delays in approving the contractual approach. As a  result, the risk became a high-priority issue in February 2008. According to  a program issue report, the lack of a contract to monitor system  development progress may result in unnecessary rework and thus  additional program cost overruns, schedule delays, and performance  shortfalls.
Four of the same 25 medium risks were retired because key mitigation  steps for each one were implemented, but the strategies proved  ineffective, and the risks became actual program issues. Included in these  4 risks were the following:  In one case, the program office closed a risk regarding data exchange with  another DON system because key mitigation steps to establish exchange  requirements were fully implemented. However, in February 2008, a high- priority issue was identified regarding the exchange of data with this  system. According to program officials, the risk was mitigated to the  fullest extent possible and closed based on the understanding that  continued evaluation of data exchange requirements would be needed.  However, because the risk was retired, this evaluation did not occur.
In another case, a requirements management risk was closed on the basis  of having implemented mitigation steps, which involved establishing a  requirements management process, including having complete  requirements traceability spreadsheets. However, although several of the  mitigation steps were not fully implemented, the risk was closed on the  basis of what program officials described as an understanding reached  with the contractor regarding the requirements management process.  Several months later, a high-priority issue concerning requirements  traceability was identified because the program office discovered that the  contractor was not adhering to the understanding.
Unless risk mitigation strategies are monitored to ensure that they are  fully implemented and that they produce the intended outcomes, and  additional mitigation steps are taken when they are not, the program office  will continue to be challenged in preventing risks from developing into  actual cost, schedule, and performance problems.
section_title: Important Aspects of System Quality and Program Maturity Are Not Being Measured
paragraphs: Effective management of programs like GCSS-MC depends in part on the  ability to measure the quality of the system being acquired and  implemented. Two measures of system quality are trends in (1) the  number of unresolved severe system defects and (2) the number of  unaddressed high-priority system change requests.
GCSS-MC documentation recognizes the importance of monitoring such  trends. Moreover, the program office has established processes for (1)  collecting and tracking data on the status of program issues, including  problems discovered during early test events, and (2) capturing data on  the status of requests for changes to the system. However, its processes do  not provide the full complement of data that are needed to generate a  reliable and meaningful picture of trends in these areas. In particular, data  on problems and change request priority levels and closure dates are  either not captured or not consistently maintained. Further, program  office oversight of contractor-identified issues or defects is limited.  Program officials acknowledged these data limitations, but they stated that  oversight of contractor-identified issues is not their responsibility. Without  tracking trends in key indicators, the program office cannot adequately  understand and report to DOD decision makers whether GCSS-MC’s  quality and stability are moving in the right direction.
subsections: section_title: Data to Understand Trends in Program Problems Are Limited
paragraphs: Program guidance and related best practices encourage trend analysis  and the reporting of system defects and program problems as measures or  indicators of system quality and program maturity. As we have previously  reported, these indicators include trends in the number of unresolved  problems according to their significance or priority.
To the program office’s credit, it collects and tracks what it calls program  issues, which are problems identified by program office staff or the system  integrator that are process, procedure, or management related. These  issues are contained in the program’s Issues-Risk Management  Information System (I-RMIS). Among other things, each issue in I-RMIS is  to have an opened and closed date and an assigned priority level of high,  medium, or low. In addition, the integration contractor tracks issues that  its staff identifies related to such areas as system test defects. These issues  are contained in the contractor’s Marine Corps Issue Tracking System  (MCITS). Each issue in MCITS is to have a date when it was opened and is  to be assigned a priority on a scale of 1-5. According to program officials,  the priority levels are based on guidance from the Institute of Electrical  and Electronics Engineers (IEEE). (See table 6 for a description of each  priority level.)
However, neither I-RMIS nor MCITS contain all the data needed to reliably  produce key measures or indicators of system quality and program  maturity. Examples of these limitations are as follows:    For I-RMIS, the program office has not established a standard definition of  the priority levels used. Rather, according to program officials, each issue  owner is allowed to assign a priority based on the owner’s definition of  what high, medium, and low mean. By not using standard priority  definitions for categorizing issues, the program office cannot ensure that it  has an accurate and useful understanding of the problems it is facing at  any given time, and it will not know if it is addressing the highest priority  issues first.
For MCITS, the integration contractor does not track closure dates for all  issues. For example, as of April 2008, over 30 percent of the closed issues  did not have closure dates. This is important because it limits the  contractor’s ability to understand trends in the number of high-priority  issues that are unresolved. Program officials acknowledged the need to  have closure dates for all closed issues and stated that they intend to  correct this. If it is not corrected, the program office will not be able to  create a reliable measure of system quality and program maturity.
Compounding the above limitations in MCITS data is the program office’s  decision not to use contractor-generated reports that are based on MCITS  data. Specifically, reports summarizing MCITS issues are posted to a  SharePoint site for the program office to review. However, program  officials stated that they do not review these reports because the MCITS  issues are not their responsibility, but the contractor’s. However, without  tracking and monitoring contractor-identified issues, which include such  things as having the right skill-sets and having the resources to track and  monitor issues captured in separate databases, the program office is  missing an opportunity to understand whether proactive action is needed  to address emerging quality shortfalls in a timely manner.
section_title: Data to Understand Trends in Changes to the System Are Limited
paragraphs: Program guidance and related best practices encourage trend reporting  of change requests as measures or indicators of system stability and  quality. These indicators include trends in the number and priority of  approved changes to the system’s baseline functional and performance  capabilities that have yet to be resolved.
To its credit, the program office collects and tracks changes to the system,  which can range from minor or administrative changes to more significant  changes that propose or impact important system functionality. These  changes can be identified by either the program office or the contractor,  and they are captured in a master change request spreadsheet. Further, the  changes are to be prioritized according to the level described in table 7,  and the dates that change requests are opened and closed are to be  recorded.
However, the change request master spreadsheet does not contain the  data needed to reliably produce key measures or indicators of system  stability and quality. Examples of these limitations are as follows:    The program office has not prioritized proposed changes or managed  these changes according to their priorities. For example, of the 572 change  requests as of April 2008, 171 were assigned a priority level, and 401 were  not. Of these 171, 132 were categorized as priority 1. Since then, the  program office has temporarily recategorized the 401 change requests to  priority 3 until each one’s priority can be evaluated. The program office  has yet to establish a time frame for doing so.
The dates that change requests are resolved are not captured in the master  spreadsheet. Rather, program officials said that these dates are in the  program’s IMS and are shown there as target implementation dates. While  the IMS does include the dates changes will be implemented, these dates  are not actual dates, and they are not used to establish trends in  unresolved change requests.
Without the full complement of data needed to monitor and measure  change requests, the program office cannot know and disclose to DOD  decision makers whether the quality and stability of the system are moving  in the right direction.
section_title: Conclusions
paragraphs: DOD’s success in delivering large-scale business systems, such as GCSS- MC, is in large part determined by the extent to which it employs the kind  of rigorous and disciplined IT management controls that are reflected in  DOD policies and related guidance. While implementing these controls  does not guarantee a successful program, it does minimize a program’s  exposure to risk and thus the likelihood that it will fall short of  expectations. In the case of GCSS-MC, living up to expectations is  important because the program is large, complex, and critical to  supporting the department’s warfighting mission.
The department has not effectively implemented a number of essential IT  management controls on GCSS-MC, which has already contributed to  significant cost overruns and schedule delays, and has increased the  program’s risk going forward of not delivering a cost-effective system  solution and not meeting future cost, schedule, capability, and benefit  commitments. Moreover, GCSS-MC could be duplicating the functionality  of related systems and may be challenged in interoperating with these  systems because compliance with key aspects of DOD’s federated BEA  has not been demonstrated. Also, the program’s estimated return on  investment, and thus the economic basis for pursing the proposed system  solution, is uncertain because of limitations in how the program’s cost  estimate was derived, raising questions as to whether the nature and level  of future investment in the program needs to be adjusted. In addition, the  program’s schedule was not derived using several key schedule estimating  practices, which impacts the integrity of the cost estimate and precludes  effective implementation of EVM. Without effective EVM, the program  cannot reliably gauge progress of the work being performed so that  shortfalls can be known and addressed early, when they require less time  and fewer resources to overcome. Another related indicator of progress,  trends in system problems and change requests, also cannot be gauged  because the data needed to do so are not being collected. Collectively,  these weaknesses have already helped to push back the completion of the  program’s first increment by more than 3 years and added about $193  million in costs, and they are introducing a number of risks that, if not  effectively managed, could further impact the program. However, whether  these risks will be effectively managed is uncertain because the program  has not always followed its defined risk management process and, as a  result, has allowed yesterday’s potential problems to become today’s  actual cost, schedule, and performance problems.
While the program office is primarily responsible for ensuring that  effective IT management controls are implemented on GCSS-MC, other  oversight and stakeholder organizations share some responsibility. In  particular, even though the program office has not demonstrated its  alignment with the federated BEA, it nevertheless followed established  DOD architecture compliance guidance and used the related compliance  assessment tool in assessing and asserting its compliance. The root cause  for not demonstrating compliance thus is not traceable to the program  office, but rather is due to, among other things, the compliance guidance  and tool being limited, and the program’s oversight entities not validating  the compliance assessment and assertion. Also, even though the program’s  cost estimate was not informed by the cost experiences of other ERP  programs of the same scope, the program office is not to blame because  the root cause for this is that the Defense Cost and Resource Center has  not maintained a standardized cost element structure for its ERP programs  and a historical database of ERP program costs for program’s like GCSS- MC to use. In contrast, other weaknesses are within the program office’s  control, as evidenced by its positive actions to address the requirements  traceability shortcomings that we brought to its attention during of the  course of our work and its well-defined risk management process.
All told, this means that addressing the GCSS-MC IT management control  weaknesses require the combined efforts of the various DOD  organizations that share responsibility for defining, justifying, managing,  and overseeing the program. By doing so, the department can better  assure itself that GCSS-MC will optimally support its mission operations  and performance goals and will deliver promised capabilities and benefits,  on time and within budget.
section_title: Recommendations for Executive Action
paragraphs: To ensure that each GCSS-MC system increment is economically justified  on the basis of a full and reliable understanding of costs, benefits, and  risks, we recommend that the Secretary of Defense direct the Secretary of  the Navy to ensure that investment in the next acquisition phase of the  program’s first increment is conditional upon fully disclosing to program  oversight and approval entities the steps under way or planned to address  each of the risks discussed in this report, including the risk of not being  architecturally compliant and being duplicative of related programs, not  producing expected mission benefits commensurate with reliably  estimated costs, not effectively implementing EVM, not mitigating known  program risks, and not knowing whether the system is becoming more or  less mature and stable. We further recommend that investment in all  future GCSS-MC increments be limited if the management control  weaknesses that are the source of these risks, and which are discussed in  this report, have not been fully addressed.
To address each of the IT management control weaknesses discussed in  this report, we are also making a number of additional recommendations.  However, we are not making recommendations for the architecture  compliance weaknesses discussed in this report because we have a  broader review of DON program compliance to the BEA and DON  enterprise architecture that will be issued shortly and will contain  appropriate recommendations.
To improve the accuracy of the GCSS-MC cost estimate, as well as other  cost estimates for the department’s ERP programs, we recommend that  the Secretary of Defense direct the appropriate organization within DOD  to collaborate with relevant organizations to standardize the cost element  structure for the department’s ERP programs and to use this standard  structure to maintain cost data for its ERP programs, including GCSS-MC,  and to use this cost data in developing future cost estimates.
To improve the credibility of the GCSS-MC cost estimate, we recommend  that the Secretary of Defense direct the Secretary of the Navy, through the  appropriate chain of command, to ensure that the program’s current  economic analysis is adjusted to reflect the risks associated with it not  reflecting cost data for comparable ERP programs, and otherwise not  having been derived according to other key cost estimating practices, and  that future updates to the GCSS-MC economic analysis similarly do so.
To enhance GCSS-MC’s use of EVM, we recommend that the Secretary of  Defense direct the Secretary of the Navy, through the appropriate chain of  command, to ensure that the program office (1) monitors the actual start  and completion dates of work activities performed so that the impact of  deviations on downstream scheduled work can be proactively addressed;  (2) allocates resources, such as labor hours and material, to all key  activities on the schedule; (3) integrates key activities and supporting  tasks and subtasks; (4) identifies and allocates the amount of float time  needed for key activities to account for potential problems that might  occur along or near the schedule’s critical path; (5) performs a schedule  risk analysis to determine the level of confidence in meeting the program’s  activities and completion date; (6) allocates schedule reserve for high-risk  activities on the critical path; and (7) discloses the inherent risks and  limitations associated with any future use of the program’s EVM reports  until the schedule has been risk-adjusted.
To improve GCSS-MC management of program risks, we recommend that  the Secretary of Defense direct the Secretary of the Navy, through the  appropriate chain of command, to ensure that the program office (1) adds  each of the risks discussed in this report to its active inventory of risks, (2)  tracks and evaluates the implementation of mitigation plans for all risks,  (3) discloses to appropriate program oversight and approval authorities  whether mitigation plans have been fully executed and have produced the  intended outcome(s), and (4) only closes a risk if its mitigation plan has  been fully executed and produced the intended outcome(s).
To strengthen GCSS-MC system quality measurement, we recommend that  the Secretary of Defense direct the Secretary of the Navy, through the  appropriate chain of command, to ensure that the program office (1)  collects the data needed to develop trends in unresolved system defects  and change requests according to their priority and severity and (2)  discloses these trends to appropriate program oversight and approval  authorities.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, signed by the Deputy Under  Secretary of Defense (Business Transformation) and reprinted in appendix  II, the department stated that it concurred with two of our  recommendations and partially concurred with the remaining five. In  general, the department partially concurred because it said that efforts  were either under way or planned that will address some of the  weaknesses that these recommendations are aimed at correcting. For  example, the department stated that GCSS-MC will begin to use a recently  developed risk assessment tool that is expected to assist programs in  identifying and mitigating internal and external risks. Further, it said that  these risks will be reported to appropriate department decision makers.
We support the efforts that DOD described in its comments because they  are generally consistent with the intent of our recommendations and  believe that if they are fully and properly implemented, they will go a long  way in addressing the management control weaknesses that our  recommendations are aimed at correcting. In addition, we have made a  slight modification to one of these five recommendations to provide the  department with greater flexibility in determining which organizations  should provide for the recommendation’s implementation.
We are sending copies of this report to interested congressional  committees; the Director, Office of Management and Budget; the  Congressional Budget Office; the Secretary of Defense; and the  Department of Defense Office of the Inspector General. We also will make  copies available to others upon request. In addition, the report will be  available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-3439 or hiter@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix III.
section_title: Appendix I: Objective, Scope, and Methodology
paragraphs: Our objective was to determine whether the Department of the Navy is  effectively implementing information technology management controls on  the Global Combat Support System-Marine Corps (GCSS-MC). To  accomplish this, we focused on the first increment of GCSS-MC relative to  the following management areas: architectural alignment, economic  justification, earned value management, requirements management, risk  management, and system quality measurement. In doing so, we analyzed a  range of program documentation, such as the acquisition strategy,  program management plan, and Acquisition Program Baseline, and  interviewed cognizant program officials.
To determine whether GCSS-MC was aligned with the Department of  Defense’s (DOD) federated business enterprise architecture (BEA), we  reviewed the program’s BEA compliance assessments and system  architecture products, as well as versions 4.0, 4.1, and 5.0 of the BEA and  compared them with the BEA compliance requirements described in the  Fiscal Year 2005 National Defense Authorization Act and DOD’s BEA  compliance guidance and evaluated the extent to which the compliance  assessments addressed all relevant BEA products. We also determined the  extent to which the program-level architecture documentation supported  the BEA compliance assessments. We obtained documentation, such as  the BEA compliance assessments from the GCSS-MC and Navy Enterprise  Resource Planning programs, as well as the Air Force’s Defense Enterprise  Accounting and Management System and Air Force Expeditionary Combat  Support System programs. We then compared these assessments to  identify potential redundancies or opportunities for reuse and determined  if the compliance assessments examined duplication across programs and  if the tool that supports these assessments is being used to identify such  duplication. In doing so, we interviewed program officials and officials  from the Department of the Navy, Office of the Chief Information Officer,  and reviewed recent GAO reports to determine the extent to which the  programs were assessed for compliance against the Department of the  Navy enterprise architecture. We also interviewed program officials and  officials from the Business Transformation Agency and the Department of  the Navy, including the logistics Functional Area Manager, and obtained  guidance documentation from these officials to determine the extent to  which the compliance assessments were subject to oversight or validation.
To determine whether the program had economically justified its  investment in GCSS-MC, we reviewed the latest economic analysis to  determine the basis for the cost and benefit estimates. This included  evaluating the analysis against Office of Management and Budget guidance  and GAO’s Cost Assessment Guide. In doing so, we interviewed cognizant  program officials, including the Program Manager and cost analysis team,  regarding their respective roles, responsibilities, and actual efforts in  developing and/or reviewing the economic analysis. We also interviewed  officials at the Office of Program Analysis and Evaluation and Naval  Center for Cost Analysis as to their respective roles, responsibilities, and  actual efforts in developing and/or reviewing the economic analysis.
To determine the extent to which the program had effectively  implemented earned value management, we reviewed relevant  documentation, such the contractor’s monthly status reports, Acquisition  Program Baselines, and schedule estimates and compared them with DOD  policies and guidance. We also reviewed the program’s schedule estimates  and compared them with relevant best practices to determine the extent  to which they reflect key estimating practices that are fundamental to  having a reliable schedule. In doing so, we interviewed cognizant program  officials to discuss their use of best practices in creating the program’s  current schedule.
To determine the extent to which the program implemented requirements  management, we reviewed relevant program documentation, such as the  baseline list of requirements and system specifications and evaluated them  against relevant best practices to determine the extent to which the  program has effectively managed the system’s requirements and  maintained traceability backward to high-level business operation  requirements and system requirements, and forward to system design  specifications, and test plans. To determine the extent to which the  requirements were traceable, we randomly selected 61 program  requirements and traced them both backward and forward. This sample  was designed with a 5 percent tolerable error rate at the 95 percent level of  confidence, so that, if we found 0 problems in our sample, we could  conclude statistically that the error rate was less than 5 percent. Based  upon the weight of all other factors included in our evaluation, our  verification of 60 out of 61 requirements was sufficient to demonstrate  traceability. In addition, we interviewed program officials involved in the  requirements management process to discuss their roles and  responsibilities for managing requirements.
To determine the extent to which the program implemented risk  management, we reviewed relevant risk management documentation, such  as risk plans and risk database reports demonstrating the status of the  program’s major risks and compared the program office’s activities with  DOD acquisition management guidance and related best practices. We  also reviewed the program’s mitigation process with respect to key risks,  including 25 medium risks in the retired risk database that were actively  addressed by the program office, to determine the extent to which these  risks were effectively managed. In doing so, we interviewed cognizant  program officials responsible, such as the Program Manager, Risk  Manager, and subject matter experts to discuss their roles and  responsibilities and obtain clarification on the program’s approach to  managing risks associated with acquiring and implementing GCSS-MC.
To determine the extent to which the program is collecting the data and  monitoring trends in the number of unresolved system defects and the  number of unaddressed change requests, we reviewed program  documentation such as the testing strategy, configuration management  policy, test defect reports, change request logs, and issue data logs. We  compared the program’s data collection and analysis practices relative to  these areas to program guidance and best practices to determine the  extent to which the program is measuring important aspects of system  quality. We also interviewed program officials such as system developers,  relevant program management staff, and change control managers to  discuss their roles and responsibilities for system quality measurement.
We conducted our work at DOD offices and contractor facilities in the  Washington, D.C., metropolitan area, and Triangle, Va., from June 2007 to  July 2008, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objective. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objective.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, key contributors to this report  were Neelaxi Lakhmani, Assistant Director; Monica Anatalio; Harold  Brumm; Neil Doherty; Cheryl Dottermusch; Nancy Glover; Mustafa  Hassan; Michael Holland; Ethan Iczkovitz; Anh Le; Josh Leiling; Emily  Longcore; Lee McCracken; Madhav Panwar; Karen Richey; Melissa  Schermerhorn; Karl Seifert; Sushmita Srikanth; Jonathan Ticehurst;  Christy Tyson; and Adam Vodraska.