DOD Business Systems Modernization: Key Marine Corps System Acquisition Needs to Be Better Justified, Defined, and Managed
Released: 2008-07-28T00:00:00

Letter

Background

The Department of the Navy (DON) is a major component of DOD, consisting of two uniformed services: the Navy and the Marine Corps. The Marine Corps’ primary mission is to serve as a “total force in readiness” by responding quickly in a wide spectrum of responsibilities, such as attacks from sea to land in support of naval operations, air combat, and security of naval bases. As the only service that operates in three dimensions—in the air, on land, and at sea, the Marine Corps must be equipped to provide rapid and precise logistics support to operating forces in any environment.

The Marine Corps’ many and dispersed organization components rely heavily on IT to perform their respective mission-critical operations and related business functions, such as logistics and financial management. For fiscal year 2008, the Marine Corps budget for IT business systems is about $1.3 billion, of which $746 million (57 percent) is for operations and maintenance of existing systems and $553 million (43 percent) is for systems development and modernization. Of the approximately 904 systems in DON’s current inventory, the Marine Corps accounts for 81, or about 9 percent, of the total. The GCSS-MC is one such system investment. According to DOD, it is intended to address the Marine Corps’ long- standing problem of stove-piped logistics systems that collectively provide limited data visibility and access, are unable to present a common, integrated logistics picture in support of the warfighter, and do not provide important decision support tools.

GCSS-MC: A Brief Description

In September 2003, the Marine Corps initiated GCSS-MC to (1) deliver integrated functionality across the logistics areas (e.g., supply and maintenance), (2) provide timely and complete logistics information to authorized users for decision making, and (3) provide access to logistics information and applications regardless of location. The system is intended to function in three operational environments—deployed operations (i.e., in theater of war or exercise environment on land or at sea), in-transit, and in garrison. When GCSS-MC is fully implemented, it is to support about 33,000 users located around the world.

GCSS-MC is being developed in a series of large and complex increments using commercially available enterprise resource planning (ERP) software and hardware components. The first increment is currently the only funded portion of the program and is to provide a range of asset management capabilities, including planning inventory requirements to support current and future requesting and tracking the status of products (e.g., supplies and personnel) and services (e.g., maintenance and engineering); allocating resources (e.g., inventory, warehouse capacity, and personnel) to support unit demands for specific products; and scheduling maintenance resources (e.g., manpower, equipment, and supplies) for specific assets, such as vehicles.

Additionally, the first increment is to replace four legacy systems scheduled for retirement in 2010. Table 1 describes these four systems.

Future increments are to provide additional functionality (e.g., transportation and wholesale inventory management), enhance existing functionality, and potentially replace up to 44 additional legacy systems.

The program office estimates the total life cycle cost for the first increment to be about $442 million, including $169 million for acquisition and $273 million for operations and maintenance. The total life cycle cost of the entire program has not yet been determined because future increments are currently in the planning stages and have not been defined. As of April 2008, the program office reported that approximately $125 million has been spent on the first increment.

Program Oversight and Management Roles and Responsibilities

To manage the acquisition and deployment of GCSS-MC, the Marine Corps established a program management office within the Program Executive Office for Executive Information Systems. The program office is led by the Program Manager who is responsible for managing the program’s scope and funding and ensuring that the program meets its objectives. To accomplish this, the program office is responsible for key acquisition management controls, such as architectural alignment, economic justification, EVM, requirements management, risk management, and system quality measurement. In addition, various DOD and DON organizations share program oversight and review activities relative to these and other acquisition management controls. A listing of key entities and their roles and responsibilities is in table 2.

Overview of GCSS-MC’s Status

The program reports that the first increment of GCSS-MC is currently in the system development and demonstration phase of the defense acquisition system (DAS). The DAS consists of five key program life cycle phases and three related milestone decision points. These five phases and related milestones are described along with a summary of key program activities completed during, or planned, for each phase as follows: 1. Concept refinement: The purpose of this phase is to refine the initial system solution (concept) and create a strategy for acquiring the investment solution. During this phase, the program office defined the acquisition strategy and analyzed alternative solutions. The first increment completed this phase on July 23, 2004, which was 1 month later than planned, and the MDA approved a Milestone A decision to move to the next phase. 2. Technology development: The purpose of this phase is to determine the appropriate set of technologies to be integrated into the investment solution by iteratively assessing the viability of various technologies while simultaneously refining user requirements. During this phase, the program office selected Oracle’s E-Business Suite as the commercial off-the-shelf ERP software. In addition, the program office awarded Accenture the system integration contract to, among other things, configure the software, establish system interfaces, and implement the new system. This system integration contract was divided into two phases—Part 1 for the planning, analysis, and conceptual design of the solution and Part 2 for detailed design, build, test, and deployment of the solution. The program office did not exercise the option for Part 2 of the contract to Accenture and shortly thereafter established a new program baseline in June 2006. In November 2006, it awarded a time-and-materials system integration contract valued at $28.4 million for solution design to Oracle. The first increment completed this phase on June 8, 2007, which was 25 months later than planned due in part to contractual performance shortfalls, and the MDA approved a Milestone B decision to move to the next phase. 3. System development and demonstration: The purpose of this phase is to develop the system and demonstrate through developer testing that the system can function in its target environment. During this phase, the program office extended the solution design contract and increased funding to $67.5 million due, in part, to delays in completing the detailed design activities. As a result, the program office has not yet awarded the next contract (which includes both firm-fixed-price and time-and-materials task orders) for build and testing activities, originally planned for July 2007. Instead, it entered what it termed a “transition period” to complete detailed design activities. According to the program’s baseline, the MDA is expected to approve a Milestone C decision to move to the next phase in October 2008. However, program officials stated that Milestone C is now scheduled for April 2009, which is 35 months later than originally planned. 4. Production and deployment: The purpose of this phase is to achieve an operational capability that satisfies the mission needs, as verified through independent operational test and evaluation, and implement the system at all applicable locations. The program office plans to award a separate firm-fixed-price plus award fee contract for these activities with estimated costs yet to be determined. 5. Operations and support: The purpose of this phase is to operationally sustain the system in the most cost-effective manner over its life cycle. The details of this phase have not yet been defined.

Overall, GCSS-MC was originally planned to reach full operational capability (FOC) in fiscal year 2007 at an estimated cost of about $126 million over a 7-year life cycle. This cost estimate was later revised in 2005 to about $249 million over a 13-year life cycle. However, the program now expects to reach FOC in fiscal year 2010 at a cost of about $442 million over a 12-year life cycle. Figures 1 and 2 show the program’s current status against original milestones and original, revised, and current cost estimates.

Use of IT Acquisition Management Best Practices Maximizes Chances for Success

Acquisition best practices are tried and proven methods, processes, techniques, and activities that organizations define and use to minimize program risks and maximize the chances of a program’s success. Using best practices can result in better outcomes, including cost savings, improved service and product quality, and a better return on investment. For example, two software engineering analyses of nearly 200 systems acquisition projects indicate that teams using systems acquisition best practices produced cost savings of at least 11 percent over similar projects conducted by teams that did not employ the kind of rigor and discipline embedded in these practices. In addition, our research shows that best practices are a significant factor in successful acquisition outcomes and increase the likelihood that programs and projects will be executed within cost and schedule estimates.

We and others have identified and promoted the use of a number of best practices associated with acquiring IT systems. See table 3 for a description of several of these activities.

Prior GAO Reviews Have Identified IT Acquisition Management Weaknesses in DOD’s Business System Investments

We have previously reported that DOD has not effectively managed a number of business system investments. Among other things, our reviews of individual system investments have identified weaknesses in such areas as architectural alignment and informed investment decision making, which are also the focus areas of the Fiscal Year 2005 National Defense Authorization Act business system provisions. Our reviews have also identified weaknesses in other system acquisition and investment management areas—such as EVM, economic justification, requirements management, risk management, and test management.

Most recently, for example, we reported that the Army’s approach to investing about $5 billion over the next several years in its General Fund Enterprise Business System, Global Combat Support System-Army Field/Tactical, and Logistics Modernization Program did not include alignment with Army enterprise architecture or use a portfolio-based business system investment review process. Moreover, we reported that the Army did not have reliable analyses, such as economic analyses, to support its management of these programs. We concluded that until the Army adopts a business system investment management approach that provides for reviewing groups of systems and making enterprise decisions on how these groups will collectively interoperate to provide a desired capability, it runs the risk of investing significant resources in business systems that do not provide the desired functionality and efficiency. Accordingly, we made recommendations aimed at improving the department’s efforts to achieve total asset visibility and enhancing its efforts to improve its control and accountability over business system investments. The department agreed with our recommendations.

We also reported that DON had not, among other things, economically justified its ongoing and planned investment in the Naval Tactical Command Support System (NTCSS) and had not invested in NTCSS within the context of a well-defined DOD or DON enterprise architecture. In addition, we reported that DON had not effectively performed key measurement, reporting, budgeting, and oversight activities and had not adequately conducted requirements management and testing activities. We concluded that, without this information, DON could not determine whether NTCSS, as defined, and as being developed, is the right solution to meet its strategic business and technological needs. Accordingly, we recommended that the department develop the analytical basis to determine if continued investment in the NTCSS represents prudent use of limited resources and to strengthen management of the program, conditional upon a decision to proceed with further investment in the program. The department largely agreed with these recommendations.

In addition, we reported that the Army had not defined and developed its Transportation Coordinators’ Automated Information for Movements System II (TC-AIMS II)—a joint services system with the goal of helping to manage the movement of forces and equipment within the United States and abroad—in the context of a DOD enterprise architecture. We also reported that the Army had not economically justified the program on the basis of reliable estimates of life cycle costs and benefits and had not effectively implemented risk management. As a result, we concluded that the Army did not know if its investment in TC-AIMS II, as planned, is warranted or represents a prudent use of limited DOD resources. Accordingly, we recommended that DOD, among other things, develop the analytical basis needed to determine if continued investment in TC-AIMS II, as planned, represents prudent use of limited defense resources. In response, the department largely agreed with our recommendations and has since reduced the program’s scope by canceling planned investments.

Key DOD and Related IT Acquisition Management Controls Have Not Been Fully Implemented on GCSS-MC

DOD IT-related acquisition policies and guidance, along with other relevant guidance, provide an acquisition management control framework within which to manage business system programs like GCSS-MC. Effective implementation of this framework can minimize program risks and better ensure that system investments are defined in a way to optimally support mission operations and performance, as well as deliver promised system capabilities and benefits on time and within budget. Thus far, GCSS-MC has not been managed in accordance with key aspects of this framework, which has already contributed to more than 3 years in program schedule delays and about $193 million in cost increases. These IT acquisition management control weaknesses include compliance with DOD’s federated BEA not being sufficiently expected costs not being reliably estimated; earned value management not being adequately implemented; system requirements not always being effectively managed, although this has recently improved; key program risks not being effectively managed; and key system quality measures not being used.

The reasons that these key practices have not been sufficiently executed include limitations in the applicable DOD guidance and tools, and not collecting relevant data, each of which is described in the applicable sections of this report. By not effectively implementing these key IT acquisition management controls, the program has already experienced sizeable schedule and cost increases, and it is at increased risk of (1) not being defined in a way that best meets corporate mission needs and enhances performance and (2) costing more and taking longer than necessary to complete.

GCSS-MC Compliance with DOD’s Federated BEA Has Not Been Sufficiently Demonstrated

DOD and federal guidance recognize the importance of investing in business systems within the context of an enterprise architecture. Moreover, the 2005 National Defense Authorization Act requires that defense business systems be compliant with DOD’s federated BEA. Our research and experience in reviewing federal agencies show that not making investments within the context of a well-defined enterprise architecture often results in systems that are duplicative, are not well integrated, are unnecessarily costly to interface and maintain, and do not optimally support mission outcomes.

To its credit, the program office has followed DOD’s BEA compliance guidance. However, this guidance does not adequately provide for addressing all relevant aspects of BEA compliance. Moreover, DON’s enterprise architecture, which is a major component of DOD’s federated BEA, as well as key aspects of DOD’s corporate BEA, have yet to be sufficiently defined to permit thorough compliance determinations. In addition, current policies and guidance do not require DON investments to comply with its enterprise architecture. This means that the department does not have a sufficient basis for knowing if GCSS-MC has been defined to optimize DON and DOD business operations. Each of these architecture alignment limitations is discussed as follows: The program’s compliance assessments did not include all relevant architecture products. In particular, the program did not assess compliance with the BEA’s technical standards profile, which outlines, for example, the standards governing how systems physically communicate with other systems and how they secure data from unauthorized access. This is particularly important because systems, like GCSS-MC, need to employ common standards in order to effectively and efficiently share information with other systems. A case in point is GCSS-MC and the Navy Enterprise Resource Planning program. Specifically, GCSS-MC has identified 13 technical standards that are not in the BEA technical standards profile, and Navy Enterprise Resource Planning has identified 25 technical standards that are not in the profile. Of these, some relate to networking protocols, which could limit information sharing between these and other systems.

In addition, the program office did not assess compliance with the BEA products that describe system characteristics. This is important because doing so would create a body of information about programs that could be used to identify common system components and services that could potentially be shared by the programs, thus avoiding wasteful duplication. For example, our analysis of GCSS-MC program documentation shows that they contain such system functions as receiving goods, taking physical inventories, and returning goods, which are also system functions cited by the Navy Enterprise Resource Planning program. However, because compliance with the BEA system products was not assessed, the extent to which these functions are potentially duplicative was not considered.

Furthermore, the program office did not assess compliance with BEA system products that describe data exchanges among systems. As we previously reported, establishing and using standard system interfaces is a critical enabler to sharing data. For example, GCSS-MC program documentation indicates that it is to exchange order and status data with other systems. However, the program office has not fully developed its architecture product describing these exchanges and thus does not have the basis for understanding how its approach to exchanging information differs from that of other systems that it is to interface with. Compliance against each of these BEA products was not assessed because DOD’s compliance guidance does not provide for doing so and, according to BTA and program officials, some BEA and program-level architecture products are not sufficiently defined. According to these officials, BTA plans to continue to define these products as the BEA evolves.

The compliance assessment was not used to identify potential areas of duplication across programs, which DOD has stated is an explicit goal of its federated BEA and associated investment review and decision-making processes. More specifically, even though the compliance guidance provides for assessing programs’ compliance with the BEA product that defines DOD operational activities, and GCSS-MC was assessed for compliance with this product, the results were not used to identify programs that support the same operational activities and related business processes. Given that the federated BEA is intended to identify and avoid not only duplications within DOD components, but also between DOD components, it is important that such commonality be addressed. For example, program-level architecture products for GCSS-MC and Navy Enterprise Resource Planning, as well as two Air Force programs (Defense Enterprise Accounting and Management System-Air Force and the Air Force Expeditionary Combat Support System) show that each supports at least six of the same BEA operational activities (e.g., conducting physical inventory, delivering property, and services) and three of these four programs support at least 18 additional operational activities (e.g., performing budgeting, managing receipt, and acceptance). As a result, these programs may be investing in duplicative functionality. Reasons for not doing so were that compliance guidance does not provide for such analyses to be conducted, and programs have not been granted access rights to use this functionality.

The program’s compliance assessment did not address compliance against the DON’s enterprise architecture, which is one of the biggest members of the federated BEA. This is particularly important given that DOD’s approach to fully satisfying the architecture requirements of the 2005 National Defense Authorization Act is to develop and use a federated architecture in which component architectures are to provide the additional details needed to supplement the thin layer of corporate policies, rules, and standards included in the corporate BEA. As we recently reported, the DON’s enterprise architecture is not mature because, among other things, it is missing a sufficient description of its current and future environments in terms of business and information/data. However, certain aspects of an architecture nevertheless exist and, according to DON, these aspects will be leveraged in its efforts to develop a complete enterprise architecture. For example, the FORCEnet architecture documents DON’s technical infrastructure. Therefore, opportunities exist for DON to assess its programs in relation to these architecture products and to understand where its programs are exposed to risks because products do not exist, are not mature, or are at odds with other DON programs. According to DOD officials, compliance with the DON architecture was not assessed because DOD compliance policy is limited to compliance with the corporate BEA, and the DON enterprise architecture has yet to be sufficiently developed.

The program’s compliance assessment was not validated by DOD or DON investment oversight and decision-making authorities. More specifically, neither the DOD IRBs nor the DBSMC, nor the BTA in supporting both of these investment oversight and decision-making authorities, reviewed the program’s assessments. According to BTA officials, under DOD’s tiered approach to investment accountability, these entities are not responsible for validating programs’ compliance assessments. Rather, this is a component responsibility, and thus they rely on the military departments and defense agencies to validate the assessments.

However, the DON Office of the CIO, which is responsible for precertifying investments as compliant before they are reviewed by the IRB, did not evaluate any of the programs’ compliance assessments. According to CIO officials, they rely on Functional Area Managers to validate a program’s compliance assessments. However, no DON policy or guidance exists that describes how the Functional Area Managers should conduct such validations.

Validation of program assessments is further complicated by the absence of information captured in the assessment tool about what program documentation or other source materials were used by the program office in making its compliance determinations. Specifically, the tool is only configured, and thus was only used, to capture the results of a program’s comparison of program architecture products to BEA products. Thus, it was not used to capture the system products used in making these determinations.

In addition, the program office did not develop certain program-level architecture products that are needed to support and validate the program’s compliance assessment and assertions. According to the compliance guidance, program-level architecture products, such as those defining information exchanges and system data requirements are not required to be used until after the system has been deployed. This is important because waiting until the system is deployed is too late to avoid the costly rework required to address areas of noncompliance. Moreover, it is not consistent with other DOD guidance, which states that program- level architecture products that describe, for example, information exchanges, should be developed before a program begins system development.

The limitations in existing BEA compliance-related policy and guidance, the supporting compliance assessment tool, and the federated BEA, puts programs like GCSS-MC at increased risk of being defined and implemented in a way that does not sufficiently ensure interoperability and avoid duplication and overlap. We currently have a review under way for the Senate Armed Services Committee, Subcommittee on Readiness and Management Support, which is examining multiple programs’ compliance with the federated BEA.

Investment in GCSS-MC Was Not Economically Justified Using Reliable Estimates of Costs

The investment in the first increment of GCSS-MC has not been economically justified on the basis of reliable analyses of estimated system costs over the life of the program. According to the program’s economic analysis, the first increment will have an estimated life cycle cost of about $442 million and deliver an estimated $1.04 billion in risk-adjusted estimated benefits during this same life cycle. This equates to a net present value of about $688 million. While the most recent cost estimate was derived using some effective estimating practices, it did not make use of other practices that are essential to having an accurate and credible estimate. As a result, the Marine Corps does not have a sufficient basis for deciding whether GCSS-MC, as defined, is the most cost-effective solution to meeting its mission needs, and it does not have a reliable basis against which to measure cost performance.

The Cost Estimate Was Not Derived Using Practices Necessary for an Accurate and Credible Estimate

A reliable cost estimate is critical to the success of any IT program, as it provides the basis for informed investment decision making, realistic budget formulation and program resourcing, meaningful progress measurement, proactive course correction, and accountability for results. According to the Office of Management and Budget (OMB), programs must maintain current and well-documented cost estimates, and these estimates must encompass the full life cycle of the program. OMB states that generating reliable cost estimates is a critical function necessary to support OMB’s capital programming process. Without reliable estimates, programs are at increased risk of experiencing cost overruns, missed deadlines, and performance shortfalls.

Our research has identified a number of practices for effective program cost estimating. We have issued guidance that associates these practices with four characteristics of a reliable cost estimate. These four characteristic are specifically defined as follows: Comprehensive: The cost estimates should include both government and contractor costs over the program’s full life cycle, from the inception of the program through design, development, deployment, and operation and maintenance, to retirement. They should also provide a level of detail appropriate to ensure that cost elements are neither omitted nor double counted and include documentation of all cost-influencing ground rules and assumptions.

Well-documented: The cost estimates should have clearly defined purposes and be supported by documented descriptions of key program or system characteristics (e.g., relationships with other systems, performance parameters). Additionally, they should capture in writing such things as the source data used and their significance, the calculations performed and their results, and the rationale for choosing a particular estimating method or reference. Moreover, this information should be captured in such a way that the data used to derive the estimate can be traced back to, and verified against, their sources. The final cost estimate should be reviewed and accepted by management on the basis of confidence in the estimating process and the estimate produced by the process.

Accurate: The cost estimates should provide for results that are unbiased and should not be overly conservative or optimistic (i.e., should represent the most likely costs). In addition, the estimates should be updated regularly to reflect material changes in the program, and steps should be taken to minimize mathematical mistakes and their significance. The estimates should also be grounded in a historical record of cost estimating and actual experiences on comparable programs.

Credible: The cost estimates should discuss any limitations in the analysis performed that are due to uncertainty or biases surrounding data or assumptions. Further, the estimates’ derivation should provide for varying any major assumptions and recalculating outcomes based on sensitivity analyses, and the estimates’ associated risks and inherent uncertainty should be disclosed. Also, the estimates should be verified based on cross- checks using other estimating methods and by comparing the results with independent cost estimates.

The $442 million life cycle cost estimate for the first increment reflects many of the practices associated with a reliable cost estimate, including all practices associated with being comprehensive and well-documented, and several related to being accurate and credible. (See table 4.) However, several important accuracy and credibility practices were not satisfied.

The cost estimate is comprehensive because it includes both the government and contractor costs specific to development, acquisition (nondevelopment), implementation, and operations and support over the program’s 12-year life cycle. Moreover, the estimate clearly describes how the various subelements are summed to produce the amounts for each cost category, thereby ensuring that all pertinent costs are included, and no costs are double counted. Lastly, cost-influencing ground rules and assumptions, such as the program’s schedule, labor rates, and inflation rates are documented.

The cost estimate is also well-documented in that the purpose of the cost estimate was clearly defined, and a technical baseline has been documented that includes, among others things, the relationships with other systems and planned performance parameters. Furthermore, the calculations and results used to derive the estimate are documented, including descriptions of the methodologies used and traceability back to source data (e.g., vendor quotes, salary tables). Also, the cost estimate was reviewed both by the Naval Center for Cost Analysis and the Office of the Secretary of Defense, Director for Program Analysis and Evaluation, which ensures a level of confidence in the estimating process and the estimate produced.

However, the estimate lacks accuracy because not all important practices related to this characteristic were satisfied. Specifically, while the estimate is grounded in documented assumptions (e.g., hardware refreshment every 5 years), and periodically updated to reflect changes to the program, it is not grounded in historical experience with comparable programs. As stated in our guide, estimates should be based on historical records of cost and schedule estimates from comparable programs, and such historical data should be maintained and used for evaluation purposes and future estimates on other comparable programs. The importance of doing so is evident by the fact that GCSS-MC’s cost estimate has increased by about $193 million since July 2005, which program officials attributed to, among other things, schedule delays, software development complexity, and the lack of historical data from similar ERP programs. While the program office did leverage historical cost data from other ERP programs, including the Navy’s Enterprise Resource Planning Pilot programs and programs at the Bureau of Prisons and the Department of Agriculture, program officials told us that these programs’ scopes were not comparable. For example, none of the programs had to utilize a communication architecture as complex as the Marine Corps, which officials cited as a significant factor in the cost increases and a challenge in estimating costs.

The absence of analogous cost data for large-scale ERP programs is due in part to the fact that DOD has not established a standardized cost element structure for ERP programs that can be used to capture actual cost data. According to officials with the Defense Cost and Resource Center, such cost element structures are needed, along with a requirement for programs to report on their costs, but approval and resources have yet to be gained for either these structures or the reporting of their costs. Until a standardized data structure exists, programs like GCSS-MC will continue to lack a historical database containing cost estimates and actual cost experiences of comparable ERP programs. This means that the current and future GCSS-MC cost estimates will lack sufficient accuracy for effective investment decision making and performance measurement.

Compounding the estimate’s limited accuracy are limitations in its credibility. Specifically, while the estimate satisfies some of the key practices for a credible cost estimate (e.g., confirming key cost drivers, performing sensitivity analyses, having an independent cost estimate prepared by the Naval Center for Cost Analysis that was within 4 percent of the program’s estimate, and conducting a risk analysis that showed a range of estimated costs of $411 million to $523 million), no risk analysis was performed to determine the program schedule’s risks and associated impact on the cost estimate. As described earlier in this report, the program has experienced about 3 years in schedule delays and recently experienced delays in completing the solution design phase. Therefore, the importance of conducting a schedule risk analysis and using the results to assess the variability in the cost estimate is critical for ensuring a credible cost estimate. Program officials agreed that the program’s schedule is aggressive and risky and that this risk was not assessed in determining the cost estimate’s variability. Without doing so, the program’s cost estimate is not credible, and thus the program is at risk of cost overruns as a result of schedule delays.

Benefits Estimate Has Been Adjusted to Reflect Questionable Assumptions and Limited Data

Forecasting expected benefits over the life of a program is also a key aspect of economically justifying an investment. OMB guidance advocates economically justifying investments on the basis of net present value. If net present value is positive, then the corresponding benefit-to- cost ratio will be greater than 1 (and vice versa). This guidance also advocates updating the analyses over the life of the program to reflect material changes in expected benefits, costs, and risks. Since estimates of benefits can be uncertain because of the imprecision in both the underlying data and modeling assumptions used, effects of this uncertainty should be analyzed and reported. By doing this, informed investment decision making can occur through the life of the program, and a baseline can be established against which to compare the accrual of actual benefits from deployed system capabilities.

The original benefit estimate for the first increment was based on questionable assumptions and insufficient data from comparable programs. The most recent economic analysis, dated January 2007, includes monetized, yearly benefit estimates for fiscal years 2010–2019 in three key areas—inventory reductions, reductions in inventory carrying costs, and improvements in maintenance processes. Collectively, these benefits totaled about $2.89 billion (not risk-adjusted). However, these calculations were made using questionable assumptions and limited data. For example, The total value of the Marine Corps inventory needed to calculate inventory reductions and reductions in carrying costs could not be determined because of limitations with existing logistic systems.

The cost savings resulting from improvements in maintenance processes were calculated based on assumptions from an ERP implementation in the commercial sector that, according to program officials, is not comparable in scope to GCSS-MC.

To account for the uncertainty inherent in the benefits estimate, the program office performed a Monte Carlo simulation. According to the program office, this risk analysis generated a discounted and risk-adjusted benefits estimate of $1.04 billion. As a result of the $1.85 billion adjustment to estimated benefits, the program office has a more realistic benefit baseline against which to compare the accrual of actual benefits from deployed system capabilities.

EVM Has Not Been Adequately Implemented

The program office has elected to implement EVM, which is a proven means for measuring program progress and thereby identifying potential cost overruns and schedule delays early, when they can be minimized. In doing so, it has adopted a tailored EVM approach that focuses on schedule. However, this schedule-focused approach has not been effectively implemented because it is based on a baseline schedule that was not derived using key schedule estimating practices. According to program officials, the schedule was driven by an aggressive program completion date established in response to direction from oversight entities to complete the program as soon as possible. As a result, they said that following these practices would have delayed this completion date. Regardless, this means that the schedule baseline is not reliable, and progress will likely not track to the schedule.

A Tailored EVM Approach Is Being Used to Measure Program Progress

The program office has adopted a tailored approach to performing EVM because of the contract type being used. As noted earlier, the contract types associated with GCSS-MC integration and implementation vary, and include, for example, firm-fixed-price contracts and time-and-materials contracts. Under a firm-fixed-price contract, the price is not subject to any adjustment on the basis of the contractor’s cost experience in performing the contract. For a time-and-materials contract, supplies or services are acquired on the basis of (1) an undefined number of direct labor hours that are paid at specified fixed hourly rates and (2) actual cost for materials.

According to DOD guidance, EVM is generally not encouraged for firm- fixed-price, level of effort, and time-and-material contracts. In these situations, the guidance states that programs can use a tailored EVM approach in which an integrated master schedule (IMS) is exclusively used to provide visibility into program performance.

DON has chosen to implement this tailored EVM approach on GCSS-MC. In doing so, it is measuring progress against schedule commitments, and not cost commitments, using an IMS for each program phase. According to program officials, the IMS describes and guides the execution of program activities. Regardless of the approach used, effective implementation depends on having a reliable IMS.

Schedule Baseline Was Not Derived in Accordance with Key Estimating Practices

The success of any program depends in part on having a reliable schedule specifying when the program’s set of work activities will occur, how long they will take, and how they are related to one another. As such, the schedule not only provides a road map for the systematic execution of a program, but it also provides the means by which to gauge progress, identify and address potential problems, and promote accountability. Our research has identified nine practices associated with effective schedule estimating. These practices are (1) capturing key activities, (2) sequencing key activities, (3) assigning resources to key activities, (4) integrating key activities horizontally and vertically, (5) establishing the duration of key activities, (6) establishing the critical path for key activities, (7) identifying “float time” between key activities, (8) distributing reserves to high-risk activities, and (9) performing a schedule risk analysis.

The current IMS for the solution design and transition-to-build phase of the first increment was developed using some of these practices. However, it does not reflect several practices that are fundamental to having a schedule baseline that provides a sufficiently reliable basis for measuring progress and forecasting slippages. To the program office’s credit, its IMS captures and sequences key activities required to complete the project, integrates the tasks horizontally, and identifies the program’s critical path. However, the program office is not monitoring the actual durations of scheduled activities so that it can address the impact of any deviations on later scheduled activities. Moreover, the schedule does not adequately identify the resources needed to complete the tasks and is not integrated vertically, meaning that multiple teams executing different aspects of the program cannot effectively work to the same master schedule. Further, the IMS does not adequately mitigate schedule risk by identifying float time between key activities, introducing schedule reserve for high-risk activities, or including the results of a schedule risk analysis. See table 5 for the results of our analyses relative to each of the nine practices.

According to program officials, they intend to begin monitoring actual activity start and completion dates so that they can proactively adjust later scheduled activities that are affected by deviations. However, they do not plan to perform the three practices related to understanding and managing schedule risk because doing so would likely extend the program’s completion date, and they set this date to be responsive to direction from DOD and DON oversight entities to complete the program as soon as possible. In our view, not performing these practices does not allow the inherent risks in meeting this imposed completion date to be proactively understood and addressed. The consequence of omitting these practices is a schedule that does not provide a reliable basis for performing EVM.

Requirements Management Weakness Was Recently Corrected

Well-defined and managed requirements are recognized by DOD guidance as essential and can be viewed as a cornerstone of effective system acquisition. One aspect of effective requirements management is requirements traceability. By tracing requirements both backward from system requirements to higher level business or operational requirements and forward to system design specifications and test plans, the chances of the deployed product satisfying requirements are increased, and the ability to understand the impact of any requirement changes, and thus make informed decision about such changes, is enhanced.

The program office recently strengthened its requirements traceability. In November 2007, and again in February 2008, the program office was unable to demonstrate for us that it could adequately trace its 1,375 system requirements to both design specifications and test documentation. Specifically, the program office was at that time using a tool called DOORS®, which if implemented properly, allows each requirement to be linked from its most conceptual definition to its most detailed definition, as well as to design specifications and test cases. In effect, the tool maintains the linkages among requirement documents, design documents, and test cases even if requirements change. However, the system integration contractor was not using the tool. Instead the contractor was submitting its 244 work products, accompanied by spreadsheets that linked each work product to one or more system requirements and test cases. The program office then had to verify and validate the spreadsheets and import and link each work product to the corresponding requirement and test case in DOORS. Because of the sheer number of requirements and work products and its potential to impact cost, schedule, and performance, the program designated this approach as a medium risk. It later closed the risk because the proposed mitigation strategy failed to mitigate it, and it was realized as a high-priority program issue (i.e., problem).

According to program officials, this requirements traceability approach resulted in time-consuming delays in approving the design work products and importing and establishing links between these products and the requirements in DOORS, in part because the work products were not accompanied by complete spreadsheets that established the traceability. As a result, about 30 percent of the contractor’s work products had yet to be validated, approved, and linked to requirements when the design phase was originally scheduled to be complete. Officials stated that the contractor was not required to use DOORS because it was not experienced with this tool and becoming proficient with it would have required time and resources, thereby increasing both the program’s cost and schedule. Ironically, however, not investing the time and resources to address the limitations in the program’s traceability approach contributed to recent delays in completing the solution design activities, and additional resources had to be invested to address its requirements traceability problems.

The program office now reports that it can trace requirements backward and forward. In April 2008, we verified this by tracing 60 out of 61 randomly sampled requirements backward to system requirements and forward to approved design specifications and test plans. Program officials explained that the reason that we could not trace the one requirement was that the related work products had not yet been approved. In addition, they stated that there were additional work products that had yet to be finalized and traced.

Without adequate traceability, the risk of a system not performing as intended and requiring expensive rework is increased. To address its requirements traceability weakness, program officials told us that they now intend to require the contractor to use DOORS during the next phase of the program (build and test). If implemented effectively, the new process should address previous requirements traceability weaknesses and thereby avoid a repeat of past problems.

Not All Program Risks Have Been Adequately Managed

Proactively managing program risks is a key acquisition management control and, if done properly, can greatly increase the chances of programs delivering promised capabilities and benefits on time and within budget. To the program office’s credit, it has defined a risk management process that meets relevant guidance. However, it has not effectively implemented the process for all identified risks. As a result, these risks have become actual program problems that have impacted the program’s cost, schedule, and performance commitments.

DOD acquisition management guidance, as well as other relevant guidance, advocates identifying facts and circumstances that can increase the probability of an acquisition’s failing to meet cost, schedule, and performance commitments and then taking steps to reduce the probability of their occurrence and impact. In brief, effective risk management consists of: (1) establishing a written plan for managing risks; (2) designating responsibility for risk management activities; (3) encouraging project-wide participation in the identification and mitigation of risks; (4) defining and implementing a process that provides for the identification, analysis, and mitigation of risks; and (5) examining the status of identified risks in program milestone reviews.

The program office has developed a written plan for managing risks, and established a process that together provide for the above cited risk management practices, and it has followed many key aspects of its plan and process. For example, The Program Manager has been assigned overall responsibility for managing risks. Also, individuals have been assigned ownership of each risk, to include conducting risk analyses, implementing mitigation strategies, and working with the risk support team.

The plan and process encourage project-wide participation in the identification and mitigation of risks by allowing program staff to submit a risk for inclusion in a risk database and take ownership of the risk and the strategy for mitigating it. In addition, stakeholders can bring potential risks to the Program Manager’s attention through interviews, where potential risks are considered and evaluated.

The program office has thus far identified and categorized individual risks.

As of December 2007, the risk database contained 27 active risks—2 high, 15 medium, and 10 low.

Program risks are considered during program milestone reviews.

Specifically, our review of documentation for the Design Readiness Review, a key decision point during the system development and demonstration phase leading up to a Milestone C decision, showed that key risks were discussed. Furthermore, the Program Manager reviews program risks’ status through a risk watch list and bimonthly risk briefings.

However, the program office has not consistently followed other aspects of its process. For example, it did not perform key practices for identifying and managing schedule risks, such as conducting a schedule risk assessment and building in reserve time to its schedule. In addition, mitigation steps for several key risks were either not performed in accordance with the risk management strategy, or risks that were closed as having been mitigated were later found to be actual program issues (i.e., problems).

For 25 medium risks in the closed risk database, as of February 2008, 4 were closed because mitigation steps were not performed in accordance with the strategy and the risks ultimately became actual issues. Examples from these medium risks are as follows: In one case, the mitigation strategy was for the contractor to deliver certain design documents that were traced to system requirements and to do so before beginning the solution build phase. The design documents, however, were not received in accordance with the mitigation strategy. Specifically, program officials told us that the design documents contained inaccuracies or misinterpretations of the requirements and were not completed on time because of the lack of resources to correct these problems. As a result, the program experienced delays in completing its solution design activities.

In another case, the mitigation strategy included creating the documentation needed to execute the contract for monitoring the build phase activities. However, the mitigation steps were not performed due to, among other things, delays in approving the contractual approach. As a result, the risk became a high-priority issue in February 2008. According to a program issue report, the lack of a contract to monitor system development progress may result in unnecessary rework and thus additional program cost overruns, schedule delays, and performance shortfalls.

Four of the same 25 medium risks were retired because key mitigation steps for each one were implemented, but the strategies proved ineffective, and the risks became actual program issues. Included in these 4 risks were the following: In one case, the program office closed a risk regarding data exchange with another DON system because key mitigation steps to establish exchange requirements were fully implemented. However, in February 2008, a high- priority issue was identified regarding the exchange of data with this system. According to program officials, the risk was mitigated to the fullest extent possible and closed based on the understanding that continued evaluation of data exchange requirements would be needed. However, because the risk was retired, this evaluation did not occur.

In another case, a requirements management risk was closed on the basis of having implemented mitigation steps, which involved establishing a requirements management process, including having complete requirements traceability spreadsheets. However, although several of the mitigation steps were not fully implemented, the risk was closed on the basis of what program officials described as an understanding reached with the contractor regarding the requirements management process. Several months later, a high-priority issue concerning requirements traceability was identified because the program office discovered that the contractor was not adhering to the understanding.

Unless risk mitigation strategies are monitored to ensure that they are fully implemented and that they produce the intended outcomes, and additional mitigation steps are taken when they are not, the program office will continue to be challenged in preventing risks from developing into actual cost, schedule, and performance problems.

Important Aspects of System Quality and Program Maturity Are Not Being Measured

Effective management of programs like GCSS-MC depends in part on the ability to measure the quality of the system being acquired and implemented. Two measures of system quality are trends in (1) the number of unresolved severe system defects and (2) the number of unaddressed high-priority system change requests.

GCSS-MC documentation recognizes the importance of monitoring such trends. Moreover, the program office has established processes for (1) collecting and tracking data on the status of program issues, including problems discovered during early test events, and (2) capturing data on the status of requests for changes to the system. However, its processes do not provide the full complement of data that are needed to generate a reliable and meaningful picture of trends in these areas. In particular, data on problems and change request priority levels and closure dates are either not captured or not consistently maintained. Further, program office oversight of contractor-identified issues or defects is limited. Program officials acknowledged these data limitations, but they stated that oversight of contractor-identified issues is not their responsibility. Without tracking trends in key indicators, the program office cannot adequately understand and report to DOD decision makers whether GCSS-MC’s quality and stability are moving in the right direction.

Data to Understand Trends in Program Problems Are Limited

Program guidance and related best practices encourage trend analysis and the reporting of system defects and program problems as measures or indicators of system quality and program maturity. As we have previously reported, these indicators include trends in the number of unresolved problems according to their significance or priority.

To the program office’s credit, it collects and tracks what it calls program issues, which are problems identified by program office staff or the system integrator that are process, procedure, or management related. These issues are contained in the program’s Issues-Risk Management Information System (I-RMIS). Among other things, each issue in I-RMIS is to have an opened and closed date and an assigned priority level of high, medium, or low. In addition, the integration contractor tracks issues that its staff identifies related to such areas as system test defects. These issues are contained in the contractor’s Marine Corps Issue Tracking System (MCITS). Each issue in MCITS is to have a date when it was opened and is to be assigned a priority on a scale of 1-5. According to program officials, the priority levels are based on guidance from the Institute of Electrical and Electronics Engineers (IEEE). (See table 6 for a description of each priority level.)

However, neither I-RMIS nor MCITS contain all the data needed to reliably produce key measures or indicators of system quality and program maturity. Examples of these limitations are as follows: For I-RMIS, the program office has not established a standard definition of the priority levels used. Rather, according to program officials, each issue owner is allowed to assign a priority based on the owner’s definition of what high, medium, and low mean. By not using standard priority definitions for categorizing issues, the program office cannot ensure that it has an accurate and useful understanding of the problems it is facing at any given time, and it will not know if it is addressing the highest priority issues first.

For MCITS, the integration contractor does not track closure dates for all issues. For example, as of April 2008, over 30 percent of the closed issues did not have closure dates. This is important because it limits the contractor’s ability to understand trends in the number of high-priority issues that are unresolved. Program officials acknowledged the need to have closure dates for all closed issues and stated that they intend to correct this. If it is not corrected, the program office will not be able to create a reliable measure of system quality and program maturity.

Compounding the above limitations in MCITS data is the program office’s decision not to use contractor-generated reports that are based on MCITS data. Specifically, reports summarizing MCITS issues are posted to a SharePoint site for the program office to review. However, program officials stated that they do not review these reports because the MCITS issues are not their responsibility, but the contractor’s. However, without tracking and monitoring contractor-identified issues, which include such things as having the right skill-sets and having the resources to track and monitor issues captured in separate databases, the program office is missing an opportunity to understand whether proactive action is needed to address emerging quality shortfalls in a timely manner.

Data to Understand Trends in Changes to the System Are Limited

Program guidance and related best practices encourage trend reporting of change requests as measures or indicators of system stability and quality. These indicators include trends in the number and priority of approved changes to the system’s baseline functional and performance capabilities that have yet to be resolved.

To its credit, the program office collects and tracks changes to the system, which can range from minor or administrative changes to more significant changes that propose or impact important system functionality. These changes can be identified by either the program office or the contractor, and they are captured in a master change request spreadsheet. Further, the changes are to be prioritized according to the level described in table 7, and the dates that change requests are opened and closed are to be recorded.

However, the change request master spreadsheet does not contain the data needed to reliably produce key measures or indicators of system stability and quality. Examples of these limitations are as follows: The program office has not prioritized proposed changes or managed these changes according to their priorities. For example, of the 572 change requests as of April 2008, 171 were assigned a priority level, and 401 were not. Of these 171, 132 were categorized as priority 1. Since then, the program office has temporarily recategorized the 401 change requests to priority 3 until each one’s priority can be evaluated. The program office has yet to establish a time frame for doing so.

The dates that change requests are resolved are not captured in the master spreadsheet. Rather, program officials said that these dates are in the program’s IMS and are shown there as target implementation dates. While the IMS does include the dates changes will be implemented, these dates are not actual dates, and they are not used to establish trends in unresolved change requests.

Without the full complement of data needed to monitor and measure change requests, the program office cannot know and disclose to DOD decision makers whether the quality and stability of the system are moving in the right direction.

Conclusions

DOD’s success in delivering large-scale business systems, such as GCSS- MC, is in large part determined by the extent to which it employs the kind of rigorous and disciplined IT management controls that are reflected in DOD policies and related guidance. While implementing these controls does not guarantee a successful program, it does minimize a program’s exposure to risk and thus the likelihood that it will fall short of expectations. In the case of GCSS-MC, living up to expectations is important because the program is large, complex, and critical to supporting the department’s warfighting mission.

The department has not effectively implemented a number of essential IT management controls on GCSS-MC, which has already contributed to significant cost overruns and schedule delays, and has increased the program’s risk going forward of not delivering a cost-effective system solution and not meeting future cost, schedule, capability, and benefit commitments. Moreover, GCSS-MC could be duplicating the functionality of related systems and may be challenged in interoperating with these systems because compliance with key aspects of DOD’s federated BEA has not been demonstrated. Also, the program’s estimated return on investment, and thus the economic basis for pursing the proposed system solution, is uncertain because of limitations in how the program’s cost estimate was derived, raising questions as to whether the nature and level of future investment in the program needs to be adjusted. In addition, the program’s schedule was not derived using several key schedule estimating practices, which impacts the integrity of the cost estimate and precludes effective implementation of EVM. Without effective EVM, the program cannot reliably gauge progress of the work being performed so that shortfalls can be known and addressed early, when they require less time and fewer resources to overcome. Another related indicator of progress, trends in system problems and change requests, also cannot be gauged because the data needed to do so are not being collected. Collectively, these weaknesses have already helped to push back the completion of the program’s first increment by more than 3 years and added about $193 million in costs, and they are introducing a number of risks that, if not effectively managed, could further impact the program. However, whether these risks will be effectively managed is uncertain because the program has not always followed its defined risk management process and, as a result, has allowed yesterday’s potential problems to become today’s actual cost, schedule, and performance problems.

While the program office is primarily responsible for ensuring that effective IT management controls are implemented on GCSS-MC, other oversight and stakeholder organizations share some responsibility. In particular, even though the program office has not demonstrated its alignment with the federated BEA, it nevertheless followed established DOD architecture compliance guidance and used the related compliance assessment tool in assessing and asserting its compliance. The root cause for not demonstrating compliance thus is not traceable to the program office, but rather is due to, among other things, the compliance guidance and tool being limited, and the program’s oversight entities not validating the compliance assessment and assertion. Also, even though the program’s cost estimate was not informed by the cost experiences of other ERP programs of the same scope, the program office is not to blame because the root cause for this is that the Defense Cost and Resource Center has not maintained a standardized cost element structure for its ERP programs and a historical database of ERP program costs for program’s like GCSS- MC to use. In contrast, other weaknesses are within the program office’s control, as evidenced by its positive actions to address the requirements traceability shortcomings that we brought to its attention during of the course of our work and its well-defined risk management process.

All told, this means that addressing the GCSS-MC IT management control weaknesses require the combined efforts of the various DOD organizations that share responsibility for defining, justifying, managing, and overseeing the program. By doing so, the department can better assure itself that GCSS-MC will optimally support its mission operations and performance goals and will deliver promised capabilities and benefits, on time and within budget.

Recommendations for Executive Action

To ensure that each GCSS-MC system increment is economically justified on the basis of a full and reliable understanding of costs, benefits, and risks, we recommend that the Secretary of Defense direct the Secretary of the Navy to ensure that investment in the next acquisition phase of the program’s first increment is conditional upon fully disclosing to program oversight and approval entities the steps under way or planned to address each of the risks discussed in this report, including the risk of not being architecturally compliant and being duplicative of related programs, not producing expected mission benefits commensurate with reliably estimated costs, not effectively implementing EVM, not mitigating known program risks, and not knowing whether the system is becoming more or less mature and stable. We further recommend that investment in all future GCSS-MC increments be limited if the management control weaknesses that are the source of these risks, and which are discussed in this report, have not been fully addressed.

To address each of the IT management control weaknesses discussed in this report, we are also making a number of additional recommendations. However, we are not making recommendations for the architecture compliance weaknesses discussed in this report because we have a broader review of DON program compliance to the BEA and DON enterprise architecture that will be issued shortly and will contain appropriate recommendations.

To improve the accuracy of the GCSS-MC cost estimate, as well as other cost estimates for the department’s ERP programs, we recommend that the Secretary of Defense direct the appropriate organization within DOD to collaborate with relevant organizations to standardize the cost element structure for the department’s ERP programs and to use this standard structure to maintain cost data for its ERP programs, including GCSS-MC, and to use this cost data in developing future cost estimates.

To improve the credibility of the GCSS-MC cost estimate, we recommend that the Secretary of Defense direct the Secretary of the Navy, through the appropriate chain of command, to ensure that the program’s current economic analysis is adjusted to reflect the risks associated with it not reflecting cost data for comparable ERP programs, and otherwise not having been derived according to other key cost estimating practices, and that future updates to the GCSS-MC economic analysis similarly do so.

To enhance GCSS-MC’s use of EVM, we recommend that the Secretary of Defense direct the Secretary of the Navy, through the appropriate chain of command, to ensure that the program office (1) monitors the actual start and completion dates of work activities performed so that the impact of deviations on downstream scheduled work can be proactively addressed; (2) allocates resources, such as labor hours and material, to all key activities on the schedule; (3) integrates key activities and supporting tasks and subtasks; (4) identifies and allocates the amount of float time needed for key activities to account for potential problems that might occur along or near the schedule’s critical path; (5) performs a schedule risk analysis to determine the level of confidence in meeting the program’s activities and completion date; (6) allocates schedule reserve for high-risk activities on the critical path; and (7) discloses the inherent risks and limitations associated with any future use of the program’s EVM reports until the schedule has been risk-adjusted.

To improve GCSS-MC management of program risks, we recommend that the Secretary of Defense direct the Secretary of the Navy, through the appropriate chain of command, to ensure that the program office (1) adds each of the risks discussed in this report to its active inventory of risks, (2) tracks and evaluates the implementation of mitigation plans for all risks, (3) discloses to appropriate program oversight and approval authorities whether mitigation plans have been fully executed and have produced the intended outcome(s), and (4) only closes a risk if its mitigation plan has been fully executed and produced the intended outcome(s).

To strengthen GCSS-MC system quality measurement, we recommend that the Secretary of Defense direct the Secretary of the Navy, through the appropriate chain of command, to ensure that the program office (1) collects the data needed to develop trends in unresolved system defects and change requests according to their priority and severity and (2) discloses these trends to appropriate program oversight and approval authorities.

Agency Comments and Our Evaluation

In written comments on a draft of this report, signed by the Deputy Under Secretary of Defense (Business Transformation) and reprinted in appendix II, the department stated that it concurred with two of our recommendations and partially concurred with the remaining five. In general, the department partially concurred because it said that efforts were either under way or planned that will address some of the weaknesses that these recommendations are aimed at correcting. For example, the department stated that GCSS-MC will begin to use a recently developed risk assessment tool that is expected to assist programs in identifying and mitigating internal and external risks. Further, it said that these risks will be reported to appropriate department decision makers.

We support the efforts that DOD described in its comments because they are generally consistent with the intent of our recommendations and believe that if they are fully and properly implemented, they will go a long way in addressing the management control weaknesses that our recommendations are aimed at correcting. In addition, we have made a slight modification to one of these five recommendations to provide the department with greater flexibility in determining which organizations should provide for the recommendation’s implementation.

We are sending copies of this report to interested congressional committees; the Director, Office of Management and Budget; the Congressional Budget Office; the Secretary of Defense; and the Department of Defense Office of the Inspector General. We also will make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3439 or hiter@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Objective, Scope, and Methodology

Our objective was to determine whether the Department of the Navy is effectively implementing information technology management controls on the Global Combat Support System-Marine Corps (GCSS-MC). To accomplish this, we focused on the first increment of GCSS-MC relative to the following management areas: architectural alignment, economic justification, earned value management, requirements management, risk management, and system quality measurement. In doing so, we analyzed a range of program documentation, such as the acquisition strategy, program management plan, and Acquisition Program Baseline, and interviewed cognizant program officials.

To determine whether GCSS-MC was aligned with the Department of Defense’s (DOD) federated business enterprise architecture (BEA), we reviewed the program’s BEA compliance assessments and system architecture products, as well as versions 4.0, 4.1, and 5.0 of the BEA and compared them with the BEA compliance requirements described in the Fiscal Year 2005 National Defense Authorization Act and DOD’s BEA compliance guidance and evaluated the extent to which the compliance assessments addressed all relevant BEA products. We also determined the extent to which the program-level architecture documentation supported the BEA compliance assessments. We obtained documentation, such as the BEA compliance assessments from the GCSS-MC and Navy Enterprise Resource Planning programs, as well as the Air Force’s Defense Enterprise Accounting and Management System and Air Force Expeditionary Combat Support System programs. We then compared these assessments to identify potential redundancies or opportunities for reuse and determined if the compliance assessments examined duplication across programs and if the tool that supports these assessments is being used to identify such duplication. In doing so, we interviewed program officials and officials from the Department of the Navy, Office of the Chief Information Officer, and reviewed recent GAO reports to determine the extent to which the programs were assessed for compliance against the Department of the Navy enterprise architecture. We also interviewed program officials and officials from the Business Transformation Agency and the Department of the Navy, including the logistics Functional Area Manager, and obtained guidance documentation from these officials to determine the extent to which the compliance assessments were subject to oversight or validation.

To determine whether the program had economically justified its investment in GCSS-MC, we reviewed the latest economic analysis to determine the basis for the cost and benefit estimates. This included evaluating the analysis against Office of Management and Budget guidance and GAO’s Cost Assessment Guide. In doing so, we interviewed cognizant program officials, including the Program Manager and cost analysis team, regarding their respective roles, responsibilities, and actual efforts in developing and/or reviewing the economic analysis. We also interviewed officials at the Office of Program Analysis and Evaluation and Naval Center for Cost Analysis as to their respective roles, responsibilities, and actual efforts in developing and/or reviewing the economic analysis.

To determine the extent to which the program had effectively implemented earned value management, we reviewed relevant documentation, such the contractor’s monthly status reports, Acquisition Program Baselines, and schedule estimates and compared them with DOD policies and guidance. We also reviewed the program’s schedule estimates and compared them with relevant best practices to determine the extent to which they reflect key estimating practices that are fundamental to having a reliable schedule. In doing so, we interviewed cognizant program officials to discuss their use of best practices in creating the program’s current schedule.

To determine the extent to which the program implemented requirements management, we reviewed relevant program documentation, such as the baseline list of requirements and system specifications and evaluated them against relevant best practices to determine the extent to which the program has effectively managed the system’s requirements and maintained traceability backward to high-level business operation requirements and system requirements, and forward to system design specifications, and test plans. To determine the extent to which the requirements were traceable, we randomly selected 61 program requirements and traced them both backward and forward. This sample was designed with a 5 percent tolerable error rate at the 95 percent level of confidence, so that, if we found 0 problems in our sample, we could conclude statistically that the error rate was less than 5 percent. Based upon the weight of all other factors included in our evaluation, our verification of 60 out of 61 requirements was sufficient to demonstrate traceability. In addition, we interviewed program officials involved in the requirements management process to discuss their roles and responsibilities for managing requirements.

To determine the extent to which the program implemented risk management, we reviewed relevant risk management documentation, such as risk plans and risk database reports demonstrating the status of the program’s major risks and compared the program office’s activities with DOD acquisition management guidance and related best practices. We also reviewed the program’s mitigation process with respect to key risks, including 25 medium risks in the retired risk database that were actively addressed by the program office, to determine the extent to which these risks were effectively managed. In doing so, we interviewed cognizant program officials responsible, such as the Program Manager, Risk Manager, and subject matter experts to discuss their roles and responsibilities and obtain clarification on the program’s approach to managing risks associated with acquiring and implementing GCSS-MC.

To determine the extent to which the program is collecting the data and monitoring trends in the number of unresolved system defects and the number of unaddressed change requests, we reviewed program documentation such as the testing strategy, configuration management policy, test defect reports, change request logs, and issue data logs. We compared the program’s data collection and analysis practices relative to these areas to program guidance and best practices to determine the extent to which the program is measuring important aspects of system quality. We also interviewed program officials such as system developers, relevant program management staff, and change control managers to discuss their roles and responsibilities for system quality measurement.

We conducted our work at DOD offices and contractor facilities in the Washington, D.C., metropolitan area, and Triangle, Va., from June 2007 to July 2008, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, key contributors to this report were Neelaxi Lakhmani, Assistant Director; Monica Anatalio; Harold Brumm; Neil Doherty; Cheryl Dottermusch; Nancy Glover; Mustafa Hassan; Michael Holland; Ethan Iczkovitz; Anh Le; Josh Leiling; Emily Longcore; Lee McCracken; Madhav Panwar; Karen Richey; Melissa Schermerhorn; Karl Seifert; Sushmita Srikanth; Jonathan Ticehurst; Christy Tyson; and Adam Vodraska.