DOD Business Systems Modernization: Additional Enhancements Are Needed for Army Business System Schedule and Cost Estimates to Fully Meet Best Practices
Released: 2014-10-30T00:00:00

Letter

Scope and Methodology

We assessed the GCSS-Army schedule that supported DOD’s December 2012 full deployment decision using the GAO Schedule Guide to determine whether it was comprehensive, well-constructed, credible, and controlled. To assess the schedule, we obtained and reviewed documentation, including the integrated master plan, work breakdown structure, and statement of work. To assess the program’s cost estimate, we used the GAO Cost Guide to evaluate the GCSS-Army Program Management Office’s estimating methodologies, assumptions, and results to determine whether the cost estimate was comprehensive, well-documented, accurate, and credible. We obtained and reviewed documentation, including the program office estimate, software cost model, independent cost estimate, and risk and uncertainty analysis. We also met with key program officials, such as the program manager, lead schedulers, and cost estimators to present the preliminary results of our assessment of the program’s schedule and cost estimates best practices and obtained explanations and clarifications.

We conducted this performance audit from October 2011 to September 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Background

GCSS-Army was initiated in December 2003 and is intended to provide all active Army, National Guard, and Army Reserve tactical units with the capability to track supplies, spare parts, and organizational equipment. The system is also intended to track unit maintenance, total cost of ownership, and other financial transactions related to logistics for all Army units—about 160,000 users. GCSS-Army is intended to integrate approximately 40,000 local supply and logistics databases into a single, enterprise-wide system. In December 2012, the Under Secretary of Defense for Acquisition, Technology and Logistics granted full deployment decision approval for GCSS-Army to be deployed to all remaining locations beyond the limited fielding locations of the life-cycle acquisition process. DOD officials reported that the GCSS-Army full deployment will be completed by the fourth quarter of fiscal year 2017. GCSS-Army program functionality is intended to be implemented across the Army in two waves—the first is to include two releases and is to provide supply (warehouse) and financial reporting capabilities, and the second is to include one release, which is to provide property book and maintenance capabilities. DOD has approved the funding for the Army to proceed with the deployment of the GCSS-Army functionality to all intended locations. This funding is approximately $3.7 billion. The Army reported that it had spent about $1.6 billion as of June 30, 2014.

In October 2010, we reported that the Army did not fully follow best practices in developing a reliable schedule and cost estimate for implementing GCSS-Army. In particular, the Army had not developed a fully integrated master schedule that reflected all government and contractor activities and had not performed a sensitivity analysis for the cost estimate. We recommended that the Army develop an integrated master schedule that fully incorporated best practices, such as capturing all activities, sequencing all activities, integrating activities horizontally and vertically, establishing the critical path for all activities, and conducting a schedule risk analysis. In addition, we recommended that the Army update the cost estimate by using actual costs and preparing a sensitivity analysis. DOD concurred with our recommendations, and this report provides the status of the department’s efforts to address our prior recommendations.

GAO’s Cost and Schedule Guides

In March 2009, we published the Cost Guide to address a gap in federal guidance about processes, procedures, and practices needed to ensure reliable cost estimates. The Cost Guide provides a consistent methodology based on best practices that can be used across the federal government to develop, manage, and evaluate capital program cost estimates. The methodology is a compilation of characteristics and associated best practices that federal cost estimating organizations and industry use to develop and maintain reliable cost estimates throughout the life of an acquisition program.

In May 2012, we issued an exposure draft of the Schedule Guide as a companion to the Cost Guide. A consistent methodology for developing, managing, and evaluating capital program cost estimates includes the concept of scheduling the necessary work to a timeline, as discussed in the Cost Guide. Simply put, schedule variances are usually followed by cost variances. Because some program costs, such as labor, supervision, rented equipment, and facilities, cost more if the program takes longer, a reliable schedule can contribute to an understanding of the cost impact if the program does not finish on time. In addition, management tends to respond to schedule delays by adding more resources or authorizing overtime. Further, a schedule risk analysis allows for program management to account for the cost effects of schedule slippage when developing the life-cycle cost estimate. A cost estimate cannot be considered fully credible if it does not account for the cost effects of schedule slippage.

GCSS-Army Program Schedule and Cost Estimates Did Not Fully Meet Best Practices

We found that the program schedule and cost estimates for the GCSS- Army did not fully meet best practices. Specifically, the GCSS-Army schedule supporting the December 2012 full deployment decision partially met the comprehensiveness and construction characteristics and substantially met the credibility and control characteristics for developing a high-quality and reliable schedule. In addition, the cost estimate fully met the comprehensiveness characteristic, substantially met the documentation and accuracy characteristics, and partially met the credibility characteristic for developing a high-quality and reliable cost estimate. It is important that the schedule and cost estimates are continually updated throughout the program’s life cycle so that management has the best information available to make decisions. By incorporating best practices for developing reliable schedule and cost estimates, DOD would increase the probability of GCSS-Army successfully achieving full deployment by the fourth quarter of fiscal year 2017 to provide needed functionality for financial improvement and audit readiness.

Program Schedule Did Not Fully Meet Best Practices

Our analysis found that the GCSS-Army program substantially met two and partially met the other two characteristics of a reliable schedule estimate and therefore did not provide the information needed to support the December 2012 full deployment decision (see table 1). Appendix I contains our detailed analysis of the GCSS-Army schedule estimate. The success of any program depends on having a reliable schedule of the program’s work activities that will occur, how long they will take, and how the activities are related to one another. As such, the schedule not only provides a road map for systematic execution of a program, but also provides the means by which to gauge progress, identify and address potential problems, and promote accountability.

Comprehensive. A schedule should reflect all activities as defined in the program’s work breakdown structure, including activities to be performed by the government and the contractor; the resources (e.g., labor, materials, and overhead) needed to complete each activity; and how long each activity will take.

We found that the GCSS-Army schedule partially met the comprehensive characteristic. The schedule used to support the full deployment decision reflected all activities to be performed by both the government and contractor for the program. However, resources were not loaded into the schedule software and were not assigned to specific activities in the schedule. GCSS-Army program management officials told us that the contractor used a separate system outside the schedule to manage the resources needed for the program. Information on resource needs and availability in each work period assists the program office in forecasting the likelihood that activities will be completed as scheduled. If the current schedule does not allow insight into the current or projected allocation of resources, the risk of the program’s schedule slipping is significantly increased. Our analysis also determined that activity durations were not manageable and reasonably estimated in the schedule. We found that 30 percent of the remaining activities in the schedule exceeded the standard best practice for activity duration, which should be shorter than approximately 44 working days, or 2 working months. For example, audit support activities had durations over 100 working days. Durations should be as short as possible to facilitate the objective measurement of accomplished effort. If activities are too long, the schedule may not have enough detail for effective progress measurement and reporting.

Well-constructed. A schedule should be planned so that critical project dates can be met. To meet this objective, all activities should be logically sequenced—that is, listed in the order in which they are to be carried out. In particular, activities that must finish prior to the start of other activities (i.e., predecessor activities), as well as activities that cannot begin until other activities are completed (i.e., successor activities), should be identified and their relationships established. The schedule should identify the project’s critical path. Establishing a valid critical path is necessary for examining the effects of any activity slipping along this path. The calculation of a critical path determines which activities drive the project’s earliest completion date. The schedule should also identify total float so that the schedule’s flexibility can be accurately determined.

We found that the GCSS-Army schedule was partially well-constructed. The majority of logic used to sequence the activities within the schedule was generally error free, clearly indicating to program management the order of activities that must be accomplished. However, the schedule’s critical path was not valid because it included level of effort activities and date constraints. Level of effort activities, such as program management, should not define the critical path because they are nondiscrete support activities that do not produce a definite end product; therefore, level of effort activities cannot determine the length of the project. In addition, date constraints prevent the critical path from being a continuous sequence of events from the current to finish dates of the project. Rather than relying on such constraints, the schedule should use logic and durations in order to reflect realistic start and completion dates for activities. Successfully identifying the critical path relies on several factors, such as capturing all activities; properly sequencing activities; and assigning resources, which, as noted earlier, had not been completely done. Without a valid critical path, management cannot focus on activities that will have detrimental effects on the key project milestones and deliverables if they slip. Further, our analysis found that 28 percent of remaining schedule activities had more than 100 working days of total float, meaning that those activities could slip almost 5 working months and not affect the estimated finish date of the program. Based on the remaining duration of the program, 100 working days of float would not appear to be reasonable. The GCSS-Army Program Management Office stated that total float was not reliable at the time of the full deployment decision because the schedule was being updated to reflect a modification to the system. Without accurate values of total float for a program activity, management cannot determine the flexibility of tasks and therefore cannot properly reallocate resources from tasks that can safely slip to tasks that cannot slip without adversely affecting the estimated program completion date.

Credible. A schedule should be horizontally and vertically integrated. A horizontally integrated schedule links products and outcomes with other associated sequenced activities, which helps verify that activities are arranged in the right order to achieve aggregated products or outcomes. A vertically integrated schedule ensures that the start and completion dates for activities are aligned with such dates on subsidiary schedules supporting tasks and subtasks. A schedule risk analysis should also be performed using statistical techniques to predict the level of confidence in meeting a program’s completion date.

We found that the GCSS-Army schedule was substantially credible. The schedule was substantially horizontally integrated, which means that outcomes were aligned with sequenced activities. The schedule was also substantially vertically integrated; we were able to trace varying levels of activities and supporting subactivities. Such mapping or alignment among subsidiary schedules enables different groups—such as government teams and contractors—to work to the same master schedule, and provides assurance that the representation of the schedule to different audiences is consistent and accurate. However, our analysis found that a schedule risk analysis had not been fully conducted. GCSS-Army program management officials provided documentation for a schedule risk analysis, but we noted that risk analyses were not performed for all supporting activities because program management officials stated that the program fielding schedule was not finalized at the time of the full deployment decision. If a schedule risk analysis is not conducted, program management cannot determine (1) the likelihood that the project completion date will occur, (2) how much schedule risk contingency is needed to provide an acceptable level of certainty for completion by a specific date, (3) risks most likely to delay the project, (4) how much contingency reserve each risk requires, and (5) the activities that are most likely to delay the project.

Controlled. A schedule should be continually updated using logic, durations, and actual progress to realistically forecast dates for program activities. A schedule narrative should accompany the updated schedule to provide decision makers and auditors a log of changes and their effect, if any, on the schedule time frame. The schedule should be analyzed continually for variances to determine when forecasted completion dates differ from planned dates. This analysis is especially important for those variations that affect activities identified as being in a program’s critical path and that can affect a scheduled completion date. A baseline schedule should be used to manage the program scope, the time period for accomplishing it, and the required resources.

We found that the GCSS-Army schedule was substantially controlled. GCSS-Army program management officials stated that they met weekly to discuss proposed schedule changes and update the schedule’s progress, and management also prepared a schedule narrative document that contained a list of custom fields and assumptions. In addition, we found no anomalies throughout the schedule (e.g., activities with planned start dates scheduled to occur in the past and activities with actual finish dates scheduled to occur in the future). However, we found that there was not a documented baseline schedule to measure program performance against, which would allow management to monitor any schedule variances that affect the completion of work. Without a formally established baseline schedule to measure performance against, management cannot identify or mitigate the effect of unfavorable performance.

In our October 2010 report, we recommended that the Army develop an integrated master schedule that fully incorporated best practices, such as capturing all activities, sequencing all activities, integrating activities horizontally and vertically, establishing the critical path for all activities, and conducting a schedule risk analysis. The Army’s December 2012 GCSS-Army schedule used to support the full deployment decision addressed several of the best practices that were an issue in our prior report, including capturing all activities, sequencing all activities, and integrating activities horizontally and vertically. However, as discussed, we continued to identify several best practices that were not yet fully addressed and also identified several new areas where the 2012 schedule did not incorporate best practices, such as activity durations and baseline schedule. Although GCSS-Army is in full deployment, without fully addressing best practices for scheduling, program managers will not have the best information available to make decisions related to issues such as the sequencing of activities and the flexibility of the schedule according to available resources.

Program Cost Estimate Did Not Fully Meet Best Practices

We found that the GCSS-Army program fully met one, substantially met two, and partially met one of the characteristics of a reliable cost estimate and therefore did not provide the information needed to support the full deployment decision, as shown in table 2. Appendix II contains our detailed analysis of the GCSS-Army cost estimate. A reliable cost estimate is critical to the success of any program and is updated continually throughout its life cycle. Such an estimate provides the basis for informed investment decision making, realistic budget formulation and program resourcing, meaningful progress measurement, proactive course correction when warranted, and accountability for results.

Comprehensive. A cost estimate should include costs of the program over its full life cycle, provide a level of detail appropriate to ensure that cost elements are neither omitted nor double-counted, and document all cost- influencing ground rules and assumptions. The cost estimate should also completely define the program and be technically reasonable.

We found that the cost estimate for GCSS-Army was fully comprehensive. The cost estimate included both government and contractor costs of the program over its life cycle—from the inception of the program through design, development, deployment, and operation and maintenance. The cost estimate also included an appropriate level of detail, which provided assurance that cost elements were neither omitted nor double-counted, and included documentation of all cost-influencing ground rules and assumptions. The cost estimate documentation included the purpose of the cost estimate, a technical description of the program, and technical risks (e.g., the resolution for any identified deficiencies).

Well-documented. A cost estimate should be supported by detailed documentation that describes how it was derived and how the expected funding will be spent in order to achieve a given objective. The documentation should capture such things as the source data used, the calculations performed, the results of the calculations, the estimating methodology used to derive each work breakdown structure element’s cost, and evidence that the estimate was approved by management. The documentation should discuss the technical baseline description, and the data in the technical baseline should be consistent with the cost estimate.

We found that the cost estimate for GCSS-Army was substantially well- documented. The cost estimate captured such things as the calculations performed to derive each element’s cost and the results of the calculations. The documentation also included a technical baseline description that provided data consistent with the cost estimate. Further, the GCSS-Army Program Management Office presented evidence of receiving approval of the estimate through briefings to management. Although program management officials did not provide us with written documentation of the source data, the Office of the Deputy Assistant Secretary of the Army for Cost and Economics (DASA-CE) did provide us with a full deployment decision briefing, which showed each major cost element and listed the methodology and sources of the data. However, the briefing documents included a limited amount of the actual source data, and we could not determine their reliability. Without sufficient background information about the source data and reliability of the data, the GCSS-Army cost estimator cannot know with any confidence whether the data collected can be used directly or need to be modified before use in the cost estimate.

Accurate. A cost estimate should provide for results that are unbiased, are not overly conservative or optimistic, and contain no major mistakes. A cost estimate should be based on an assessment of most likely costs (adjusted properly for inflation), updated to reflect significant changes and grounded in a historical record of cost estimating and actual experiences on other comparable programs. In addition, variances between planned and actual costs should be documented, explained, and reviewed, and estimating techniques for each cost element should be used appropriately.

We found that the cost estimate for GCSS-Army was substantially accurate. The GCSS-Army cost model detailed the inflation indexes and properly applied the indexes to each relevant cost element and included time phasing of the costs. The GCSS-Army cost model did not include any major mistakes, and all its cost elements summed up properly and were consistent with the cost estimate. In addition, the estimating techniques (i.e., engineering build-up) used to create the estimate were used appropriately. The cost model documentation did not explain whether the cost estimate was updated to reflect changes in technical or program assumptions. The program management officials provided documentation that reflected the technical changes for the major deployment decisions, but the documentation did not include details on how the costs were updated. Unless such documentation is available to verify that the cost estimate is properly updated on a regular basis, management will not have reasonable assurance that the cost estimate provides accurate information to make informed decisions about the program.

Credible. A cost estimate should discuss any limitations of the analysis because of uncertainty or biases surrounding data or assumptions. The cost estimate should include a sensitivity analysis that identifies a range of possible costs based on varying major assumptions and data. A risk and uncertainty analysis should be conducted to determine the level of risk associated with the cost estimate and identify the effects of changing key cost driver assumptions and factors. In addition, the estimate’s results should be cross-checked and reconciled to an independent cost estimate to determine whether other estimating methods produce similar results.

We found that the cost estimate was partially credible. The Army Cost Review Board developed an independent cost estimate that was reconciled to the program management officials’ cost estimate. The program management officials’ cost estimate mentioned results of a risk analysis; however the risk and uncertainty analysis was not documented. Further, since the cost estimate that was provided discussed risk only at a summary level, it is unclear how management considered risk related to the program. Without a fully documented risk and uncertainty analysis, the estimate will lose credibility and management’s decision-making ability will be impaired because it will not know the level of confidence associated with achieving the cost estimate. In addition, program management officials provided a cost estimate that identified major cost drivers, including system deployment and training. The cost estimate documentation contained a reference that a sensitivity analysis was completed on these cost drivers, but results of this analysis were not documented. As a result, the GCSS-Army cost estimator will not have a clear understanding of how each major cost driver is affected by a change in a single assumption and thus which cost driver most affects the cost estimate. Further, GCSS-Army program officials provided us with one example of evidence that indicated that some cross-checking was performed using cost models; however, the results of this cross-checking were not documented. The purpose of cross-checking is to determine whether alternative methods would produce similar results, which would increase the credibility of the estimate.

In our October 2010 report, we recommended that the Army update the GCSS-Army cost estimate by using actual costs and preparing a sensitivity analysis. For the 2012 cost estimate, we found that the Army had made progress, but we continued to identify deficiencies in documentation related to the sensitivity analysis, risk and uncertainty analysis, and cross-checking of major cost elements for reasonableness.

Conclusions

While the Army made some improvements to the schedule and cost estimates that supported the full deployment decision, the Army did not fully meet best practices in developing schedule and cost estimates for the GCSS-Army program. The Army made progress in incorporating schedule best practices, such as capturing and sequencing all activities and integrating activities horizontally and vertically, but we identified other deficiencies in schedule and cost best practices. For example, GCSS- Army did not meet best practices related to schedule durations, a valid critical path, and a cost sensitivity analysis. It is critical to correct the deficiencies identified with the schedule and cost estimates to help ensure that the projected spending for this program is being used in the most efficient and effective manner. By incorporating best practices for developing reliable schedule and cost estimates, DOD would increase the probability of GCSS-Army successfully achieving full deployment by the fourth quarter of fiscal year 2017 to provide needed functionality for financial improvement and audit readiness.

Recommendations for Executive Action

To help improve the implementation of GCSS-Army, we recommend that the Secretary of the Army take the following two actions:

Ensure that the Under Secretary of the Army, in his capacity as the Chief Management Officer, directs the GCSS-Army Program Management Office to develop an updated schedule that fully incorporates best practices, including assigning resources to all activities, establishing durations of all activities, confirming that the critical path is valid, and ensuring reasonable total float.

Ensure that the Under Secretary of the Army, in his capacity as the Chief Management Officer, directs the GCSS-Army Program Management Office to update the cost estimate to fully incorporate best practices by documenting the results of a risk and uncertainty analysis, the cross-checking of major cost elements to see if results are similar, and a sensitivity analysis.

Agency Comments and Our Evaluation

We provided a draft of this report to DOD for review and comment. In its written comments, reprinted in appendix III, DOD concurred with our recommendation to update the schedule to fully incorporate best practices and described planned and ongoing actions that the department is taking to address the recommendation. In particular, DOD indicated that the Army has taken steps to help ensure that (1) all activities are assigned resources in the schedule software, (2) all schedule activities with long durations have been detailed, (3) level of effort activities and date constraints have been removed from the schedule so that they do not define the critical path, and (4) the majority of the schedule activities associated with high total float have been removed. If effectively implemented, these actions should address the intent of our recommendation.

DOD also concurred with our recommendation to update the cost estimate to fully incorporate best practices by documenting the results of a risk and uncertainty analysis, the cross-checking of major cost elements to see if results are similar, and a sensitivity analysis. DOD described completed actions that the department has taken to address the recommendation. DOD stated that GCSS-Army achieved Milestone C in August 2011 and a full deployment decision in December 2012, and that it prepared a cost estimate per DOD acquisition rules and guidelines. DOD also stated that the Army (1) followed all Army directed best practices and approvals from the Office of the Deputy Assistant Secretary of the Army for Cost and Economics and (2) prepared a sensitivity analysis, a risk analysis, and cross-checked major cost elements for similar results, but that those documented analyses and results were not included in the formal cost estimates as directed by the Army. DOD commented that these documented analyses and results are part of the formal working papers and were provided to GAO in February 2013.

However, these actions do not fully address the intent of our recommendation. As stated in our report, we focused on the extent to which GCSS-Army’s schedule and cost estimates were prepared consistent with GAO’s Schedule and Cost Guides. We reviewed the cost estimate documentation provided by the Army in February 2013 and additional information provided in February 2014 and determined that the documentation did not fully meet best practices for a risk and uncertainty analysis, a sensitivity analysis, and cross-checking of major cost elements for similar results. As stated in our report, GCSS-Army program management officials provided a cost estimate that mentioned the results of a risk and uncertainty analysis, and contained a reference that a sensitivity analysis was completed. Also, GCSS-Army program management officials provided us with one example of evidence that indicated some cross-checking was performed using cost models. However, the results of the risk and uncertainty and sensitivity analyses, as well as the cross-checking were not documented consistent with best practices. As stated in our report, incorporating best practices for a reliable cost estimate would help ensure that DOD has a reliable cost estimate that provides the basis for effective resource allocation, proactive course correction when warranted, and accountability for results.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees; the Secretary of Defense; the Secretary of the Army; the Assistant Secretary of Defense (Acquisition); the Acting Deputy Chief Management Officer; the Under Secretary of Defense (Comptroller); the Under Secretary of the Army, in his capacity as the Chief Management Officer of the Army; and the Program Manager for GCSS- Army. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact Asif A. Khan at (202) 512-9869 or khana@gao.gov or Nabajyoti Barkakati at (202) 512-4499 or barkakatin@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made key contributions to this report are listed in appendix IV.

Appendix I: Assessment of the Global Combat Support System-Army Program’s Schedule Estimate

This appendix provides the results of our analysis of the extent to which the Global Combat Support System-Army schedule supporting the December 2012 full deployment decision met the characteristics of a high-quality, reliable schedule. Table 3 provides the detailed results of our analysis.

GAO’s methodology includes five levels of compliance with its best practices. “Not met” means the program provided no evidence that satisfies any of the criterion. “Minimally met” means the program provided evidence that satisfies a small portion of the criterion. “Partially met” means the program provided evidence that satisfies about half of the criterion. “Substantially met” means the program provided evidence that satisfies a large portion of the criterion. “Fully met” means the program provided evidence that completely satisfies the criterion.

Appendix II: Assessment of the Global Combat Support System-Army Program’s Cost Estimate

This appendix provides the results of our analysis of the extent to which the Global Combat Support System-Army cost estimate supporting the December 2012 full deployment decision met the characteristics of a high-quality cost estimate. Table 4 provides the detailed results of our analysis.

GAO’s methodology includes five levels of compliance with its best practices. “Not met” means the program provided no evidence that satisfies any of the criterion. “Minimally met” means the program provided evidence that satisfies a small portion of the criterion. “Partially met” means the program provided evidence that satisfies about half of the criterion. “Substantially met” means the program provided evidence that satisfies a large portion of the criterion. “Fully met” means the program provided evidence that completely satisfies the criterion.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to the contacts named above, Arkelga Braxton (Assistant Director), Karen Richey (Assistant Director), Beatrice Alff, Tisha Derricotte, Jennifer Echard, Emile Ettedgui, Patrick Frey, and Jason Lee made key contributions to this report.