DOD Business Systems Modernization: Air Force Business System Schedule and Cost Estimates
Released: 2014-03-10T00:00:00

Letter

Scope and Methodology

We assessed the DEAMS schedule using the GAO Schedule Guide to determine whether it was comprehensive, well-constructed, credible, and controlled. To assess the schedule, we obtained and reviewed documentation, including the integrated master plan and work breakdown structure. In assessing the program’s cost estimate, we used the GAO Cost Guide to evaluate the DEAMS Program Management Office’s estimating methodologies, assumptions, and results to determine whether the cost estimate was comprehensive, well-documented, accurate, and credible. We obtained and reviewed documentation, including the program office estimate, software cost model, independent cost estimate, and risk and uncertainty analysis. We also interviewed key program officials, such as the Program Manager, lead schedulers, and cost estimators, to obtain information, such as explanations to resolve identified discrepancies.

After we briefed DEAMS program officials on the results of our assessment, they provided an updated schedule dated October 2012. For this updated schedule, we determined the extent to which it met certain best practices for the comprehensive, well-constructed, and credible characteristics, because not implementing these best practices would affect the reliability of the entire schedule. In May 2013, program management officials provided another updated DEAMS schedule, which they acknowledged contained issues that prevented the schedule from meeting best practices. Although we did not independently assess the May 2013 schedule, we did confirm that it included certain information needed for long-term planning.

We conducted this performance audit from May 2012 to February 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Background

DEAMS was initiated in August 2003 and is intended to provide the Air Force with the entire spectrum of financial management capabilities, including collections, commitments and obligations, cost accounting, general ledger, funds control, receipt and acceptance, accounts payable and disbursement, billing, and financial reporting for the general fund. In February 2012, the DOD Deputy Chief Management Officer granted Milestone B approval for DEAMS to enter the Engineering Development Phase of the acquisition life cycle, which is considered the official start of the program. DEAMS program functionality is intended to be implemented across the Air Force in a series of releases in two increments—Increment 1 will include six releases and Increment 2 will include two releases. DOD has approved the funding for the Air Force to proceed with the acquisition of the functionality for the first increment of DEAMS. This funding is approximately $1.6 billion, with deployment scheduled to occur during the fourth quarter of fiscal year 2016. The Air Force reported that it had spent about $427.5 million as of September 30, 2013, on the program.

As stated earlier, in October 2010, we reported that although the Air Force met best practices in developing a cost estimate, it did not meet best practices in developing the schedule estimate for implementing DEAMS. In particular, the Air Force had not developed a fully integrated master schedule that reflected all government and contractor activities. We recommended that the Air Force develop an integrated master schedule that fully incorporated best practices, such as capturing all activities, sequencing all activities, integrating activities horizontally and vertically, establishing the critical path for all activities, identifying float between activities, conducting a schedule risk analysis, and updating the schedule using logic and durations to determine dates. DOD concurred with our recommendation, and we discuss later in this report the status of DOD’s efforts to address this recommendation.

GAO’s Cost and Schedule Guides

In March 2009, we published the Cost Guide to address a gap in federal guidance about processes, procedures, and practices needed for ensuring reliable cost estimates. The Cost Guide provides a consistent methodology based on best practices that can be used across the federal government to develop, manage, and evaluate capital program cost estimates. The methodology is a compilation of characteristics and associated best practices that federal cost estimating organizations and industry use to develop and maintain reliable cost estimates throughout the life of an acquisition program.

In May 2012, we issued the Schedule Guide as a companion to the Cost Guide. A consistent methodology for developing, managing, and evaluating capital program cost estimates includes the concept of scheduling the necessary work to a timeline, as discussed in the Cost Guide. Simply put, schedule variances are usually followed by cost variances. Because some program costs, such as labor, supervision, rented equipment, and facilities, cost more if the program takes longer, a reliable schedule can contribute to an understanding of the cost impact if the program does not finish on time. In addition, management tends to respond to schedule delays by adding more resources or authorizing overtime. Further, a schedule risk analysis allows for program management to account for the cost effects of schedule slippage when developing the life-cycle cost estimate. A cost estimate cannot be considered fully credible if it does not account for the cost effects of schedule slippage.

A well-planned schedule is a fundamental management tool that can help government programs use public funds effectively by specifying when work will be performed in the future and measuring program performance against an approved plan. Moreover, as a model of time, an integrated and reliable schedule can show when major events are expected to occur as well as the completion dates for all activities leading up to them, which can help determine if the program’s parameters are realistic and achievable. A program’s success depends in part on the quality of its schedule.

DEAMS Program Schedule Did Not Meet Best Practices, but the Cost Estimate Did Meet Best Practices

We found that the schedule for the DEAMS program did not meet best practices. The cost estimate did meet best practices, but the issues associated with the schedule could negatively affect the cost estimate. Specifically, the DEAMS schedule supporting the February 2012 Milestone B decision partially or minimally met the four characteristics for developing a high-quality and reliable schedule—it was not comprehensive, well-constructed, credible, or controlled. In addition, our assessment of the October 2012 updated schedule found that it was not comprehensive, well-constructed, and credible and thus was also not reliable. In contrast, the DEAMS cost estimate fully or substantially met the four characteristics of a high-quality and reliable cost estimate—it was comprehensive, well-documented, accurate, and credible. However, because the cost estimate is based on the schedule, the unreliability of the schedule could affect the cost estimate. For example, if there are schedule slippages, the costs for the program could be greater than currently estimated.

Program Schedule Was Not Developed in Accordance with Best Practices

Our analysis found that the DEAMS program partially met three and minimally met one of the characteristics of a reliable schedule estimate and therefore did not provide the information needed to support the February 2012 Milestone B decision (see table 1). Appendix I contains our detailed analysis of the DEAMS schedule estimate. The success of any program depends on having a reliable schedule of the program’s work activities that will occur, how long they will take, and how the activities are related to one another. As such, the schedule not only provides a roadmap for systematic execution of a program, but also provides the means by which to gauge progress, identify and address potential problems, and promote accountability.

Comprehensive. A schedule should reflect all activities as defined in the program’s work breakdown structure, including activities to be performed by the government and the contractor; the resources (e.g., labor, materials, and overhead) needed to do the work; and how long each activity will take.

We found that the schedule used to support the Milestone B decision included the activities to be performed by both the government and contractor for Releases 1 through 3 of Increment 1. However, the schedule did not reflect activities to be performed for Releases 4 through 6 of Increment 1 and for Releases 1 and 2 of Increment 2. The DEAMS Program Manager stated that a comprehensive schedule for Increment 1 that included the activities for all six releases would not be completed until mid-2014. The Program Manager also stated that Increment 2 had not been included because program officials did not know the detailed activities to be performed that far in advance. To address this issue, the DEAMS program office developed a roadmap depicting Releases 1 through 6 of Increment 1 and Releases 1 and 2 of Increment 2 with a full deployment date of fiscal year 2017. However, the program office did not provide a schedule that supported the estimated dates in the roadmap.

A comprehensive schedule should reflect all of a program’s activities and recognize that uncertainties and unknown factors in schedule estimates can stem from, among other things, data limitations. As such, a schedule incorporates different levels of detail depending on the information available at any point in time. That is, near-term effort will be planned in greater detail than long-term effort. Effort beyond the near term that is less well defined is represented within the schedule as long-term planning packages. Planning packages are a summarization of the work to be performed in the distant future with less specificity. Planning packages are planned at higher levels such that a single activity may represent several months of effort, generic work to be accomplished, or even a future contract or phase. Planning packages can be used as long as they are defined and estimated as well as possible. By not including all work for all deliverables for both increments and all releases, the DEAMS program could incur difficulties resulting from an incomplete understanding of the plan and what constitutes a successful conclusion for the program. DEAMS program officials provided a draft of the Schedule Management Plan that documented their intent to use a planning package approach when updating the DEAMS schedule in the future.

Resources were identified in the schedule; however, the resources were not assigned to specific activities in the schedule. Although our analysis determined that activity durations were manageable and reasonably estimated, resource availability affects estimates of work and its duration, as well as resources that will be available for subsequent activities. DEAMS program management officials told us that government resource allocations are determined by management as needed. These officials told us that management does not necessarily take into consideration the resource information captured in the schedule when determining resource allocations. However, DEAMS officials did not provide any documentation that specific resources were being mapped to the schedule. As mentioned above, the estimates of work required and duration for an activity are tied to the availability of resources; therefore, the lack of such information could hinder management’s ability to compute total labor and equipment hours, calculate total project and per-period cost, resolve resource conflicts, and establish the reasonableness of the plan.

Well-constructed. A schedule should be planned so that critical project dates can be met. To meet this objective, all activities should be logically sequenced—that is, listed in the order in which they are to be carried out. In particular, activities that must finish prior to the start of other activities (i.e., predecessor activities), as well as activities that cannot begin until other activities are completed (i.e., successor activities), should be identified and their relationships established. The establishment of a critical path is necessary for examining the effects of any activity slipping along this path. The calculation of a critical path determines which activities drive the project’s earliest completion date. The schedule should also identify total float so that the schedule’s flexibility can be accurately determined.

We found that the majority of logic used to sequence the activities within the schedule was generally error-free with a minimal use of lags, clearly indicating to program management the order of activities that must be accomplished. Although we found few missing logic relationships for Release 3 of Increment 1, approximately 25 percent of the remaining activities for Releases 1 and 2 of Increment 1 were missing logic relationships. Because interdependencies among activities were not identified, the DEAMS program management officials’ ability to properly calculate dates and predict changes in the future is impaired. We found a significant number of constraints for activities throughout the schedule. A schedule is intended to be a dynamic, proactive planning and risk mitigation tool that models the program and can be used to track progress toward important milestones. Schedules with constrained dates can portray an artificial or unrealistic view of the project. Constraints should be minimized because they can create false dates in a schedule.

Further, the schedule did not have a valid critical path and identified critical activities more by their constraints than by logic. Rather than relying on constraints, the schedule should use logic and durations in order to reflect realistic start and completion dates for activities. Successfully identifying the critical path relies on several factors, such as capturing all activities, properly sequencing activities, and assigning resources, which, as noted earlier, had not been done. Without a valid critical path, management cannot focus on activities that will have detrimental effects on the key project milestones and deliveries if they slip.

We found that total float was not reasonable, and that in some instances unreasonable float was a direct result of improper sequencing or missing logic. Releases 1 and 2 of Increment 1 showed that 25 percent of program activities had total float equal to or greater than 392 working days, meaning that those activities could slip almost 2 working years and not affect the end date of the program. Without knowledge of the reason float exists for a program activity, management cannot determine the flexibility of tasks and therefore cannot properly reallocate resources from tasks that can safely slip to tasks that cannot slip without adversely affecting the estimated program completion date.

Credible. A schedule should be horizontally and vertically integrated. A horizontally integrated schedule links products and outcomes with other associated sequenced activities, which helps verify that activities are arranged in the right order to achieve aggregated products or outcomes. A vertically integrated schedule ensures that the start and completion dates for activities are aligned with such dates on subsidiary schedules supporting tasks and subtasks. Such mapping or alignment among subsidiary schedules enables different groups—such as government teams and contractors—to work to the same master schedule, and provides assurance that the representation of the schedule to different audiences is consistent and accurate. A schedule risk analysis should also be performed using statistical techniques to predict the level of confidence in meeting a program’s completion date.

We found that Release 3 of Increment 1 exhibited horizontal integration, but Releases 1 and 2 of Increment 1 did not because date constraints prevented forecasted dates from being calculated realistically for future activities. If the schedule lacks horizontal integration, activities whose durations are greatly extended will have no effect on key milestones reflected in the schedule. We further found that Releases 1 and 2 of Increment 1 did not demonstrate vertical integration. For example, we found instances where the start dates for the same activities differed by 1 day, 1 week, and 1 month between the government and contractor schedules. Unless the schedule is vertically integrated, lower-level schedules will not be consistent with upper-level schedule milestones, affecting the integrity of the entire schedule and the ability of different teams to work to the same schedule expectations.

DEAMS program management officials stated that a schedule risk analysis had not been conducted because the schedule had not been approved to be used as a baseline schedule—the target schedule against which program performance can be measured, monitored, and reported. These officials stated that although this analysis had not been conducted, they were collecting best-case and worst-case durations from the contractor with their periodic schedule delivery. These data can be used by program management to calculate more reliable estimates of durations for future activities. However, we found that the schedule did not contain best- or worst-case duration data for 600 of 605 detailed activities. For the five instances where duration data were contained in the schedule, we determined that four were questionable because two activities were already completed and two had already exceeded the worst-case estimate. If a schedule risk analysis is not conducted, program management cannot determine the likelihood of the project’s completion date, how much schedule risk contingency is needed to provide an acceptable level of certainty for completion by a specific date, risks most likely to delay the project, how much contingency reserve each risk requires, and the paths or activities that are most likely to delay the project. As discussed later, the lack of a schedule risk analysis can affect the credibility of the cost estimate.

Controlled. A schedule should be continuously updated using logic, durations, and actual progress to realistically forecast dates for program activities. A schedule narrative should accompany the updated schedule to provide decision makers and auditors a log of changes and their effect, if any, on the schedule time frame. The schedule should be analyzed continuously for variances to determine when forecasted completion dates differ from planned dates. This analysis is especially important for those variations that affect activities identified as being in a program’s critical path and that can affect a scheduled completion date. A baseline schedule should be used to manage the program scope, the time period for accomplishing it, and the required resources.

We found that DEAMS program management met weekly to discuss proposed schedule changes and updated the schedule’s progress. However, a schedule narrative was not prepared by DEAMS program management. In addition, we found a number of date anomalies throughout the schedule, including activities with planned start dates scheduled to occur in the past and activities with actual finish dates scheduled to occur in the future. We also found a number of out-of- sequence activities in the schedule—activities that started before their predecessors finished, in contradiction to the planned sequence. If the schedule is not continually monitored to determine when forecasted completion dates differ from planned dates, then it cannot be used to determine whether schedule variances will affect work needed to be accomplished at a future date.

We also found that there was no baseline schedule that could be used to measure program performance. DEAMS program management officials did maintain a schedule narrative document that contained a list of custom fields and assumptions; however, the document did not explain ground rules and assumptions, justifications for logic, and other unique features of the schedule. These officials stated that other process documents were being developed. Without a formally established baseline schedule to measure performance against, management cannot identify or mitigate the effect of unfavorable performance.

Updated Schedule Also Did Not Follow Best Practices

Our assessment of the updated schedule dated October 2012 found that it was not comprehensive, well-constructed and credible. Although the DEAMS Program Manager stated that the government and contractor activities for Releases 1 through 3 of Increment 1 had been integrated in the October 2012 schedule, this schedule was not comprehensive. Specifically, it excluded activities for both the government and contractor related to Releases 4 through 6 of Increment 1 and Releases 1 and 2 of Increment 2. If activities are missing from the schedule, then other best practices will not be met. The schedule was also missing relationships for a significant number of the remaining milestones and activities. In addition, the October 2012 schedule included a significant number of date constraints with little or no justification for their use in the schedule. Similar to the previous schedule, the updated schedule presented unreasonable float throughout and did not include a schedule risk analysis. As a result of these shortcomings, the updated schedule was not reliable. Further, program officials could not rely on this schedule as a baseline to effectively manage and monitor program performance.

In May 2013, program management officials provided another updated DEAMS schedule that they stated included some improvements, but they acknowledged that it contained issues that prevented the schedule from meeting best practices. For example, these officials stated that the May 2013 schedule included long-term planning packages for activities related to Releases 4 through 6 of Increment 1 and Releases 1 and 2 of Increment 2, integrated government and contractor activities, and reduced the number of constraints and out-of-sequence activities in the schedule. However, the officials acknowledged that several outstanding issues remained related to, for example, vertical and horizontal integration, missing logic relationships, and the lack of a schedule risk analysis. Although we did not independently assess the May 2013 schedule to determine whether it met the four schedule characteristics, we did confirm that it included long-term planning packages, which are needed to create a complete picture of the program from start to finish and to allow the monitoring of a program’s critical path. The results of our analyses of the schedule that supported the February 2012 Milestone B decision and October 2012 DEAMS schedule reflect similar weaknesses to those we reported in October 2010. Therefore, given the findings of this review, our prior recommendation for improving the DEAMS schedule remains valid.

Program Cost Estimate Met Best Practices

We found that the DEAMS program fully or substantially met the four characteristics of a reliable cost estimate to support the Milestone B decision, as shown in table 2. However, because the cost estimate relies on dates derived from the schedule and we are questioning the reliability of the forecasted program dates, the credibility of the cost estimate can be affected. Appendix II contains our detailed analysis of the DEAMS cost estimate. A reliable cost estimate is critical to the success of any program. Such an estimate provides the basis for informed investment decision making, realistic budget formulation and program resourcing, meaningful progress measurement, proactive course correction when warranted, and accountability for results.

Comprehensive. A cost estimate should include costs of the program over its full life cycle, provide a level of detail appropriate to ensure that cost elements are neither omitted nor double-counted, and document all cost- influencing ground rules and assumptions. The cost estimate should also completely define the program and be technically reasonable.

We found that the cost estimate for DEAMS was comprehensive. The cost estimate included both government and contractor costs of the program over its life cycle—from the inception of the program through design, development, deployment, and operation and maintenance—as outlined in the roadmap prepared by program officials. As stated earlier, the roadmap provided an overall summary of the program’s key phases (increments and releases) and the expected milestones for completion. The cost estimate also included an appropriate level of detail, which provided assurance that cost elements were neither omitted nor duplicated, and included documentation of all cost-influencing ground rules and assumptions. The cost estimate documentation included the purpose of the cost estimate, a technical description of the program, and technical risks (e.g., the resolution for any identified deficiencies).

Well-documented. A cost estimate should be supported by detailed documentation that describes how it was derived and how the expected funding will be spent in order to achieve a given objective. Therefore, the documentation should capture such things as the source data used, the calculations performed, the results of the calculations, the estimating methodology used to derive each work breakdown structure element’s cost, and evidence that the estimate was approved by management.

We found that the cost estimate for DEAMS was well-documented. The cost estimate captured such things as the source data used, the calculations performed and the results of the calculations, and the rationale for choosing a particular estimating methodology. This information was captured in such a way that the data used to derive the estimate can be traced back to, and verified against, the sources so that the estimate can be easily replicated. However, there was no discussion of efforts taken, if any, to ensure the reliability of the data used. The DEAMS Program Management Office presented evidence of receiving approval of the estimate through briefings to management.

Accurate. A cost estimate should be based on an assessment of most likely costs (adjusted properly for inflation), updated to reflect significant changes and grounded in a historical record of cost estimating and actual experiences on other comparable programs.

We found that the cost estimate for DEAMS was accurate. The cost estimate provided results that were substantially unbiased, and the cost model detailed the calculations and inflation indexes underlying the estimate. Calculations within the model could be traced back to supporting documentation. The cost estimate was updated regularly to reflect significant changes in the program and updated annually to incorporate actual costs expended in prior fiscal years. Further, the cost estimate was based on historical data. However, the cost estimate did not discuss variances between planned and actual costs, which would enable estimators to assess how well they are estimating program costs and to identify lessons learned.

Credible. A cost estimate should discuss any limitations of the analysis because of uncertainty or biases surrounding data or assumptions. In addition, the estimate’s results should be cross-checked and reconciled to an independent cost estimate to determine whether other estimating methods produce similar results.

We found that the cost estimate was credible. The DEAMS Program Management Office conducted a risk and uncertainty analysis by identifying the cost elements with the greatest degree of uncertainty, determining the cost drivers for the program, and identifying the impact of changing major ground rules and cost driver assumptions. An independent cost estimate developed by the Air Force Cost Analysis Agency was reconciled to the program’s estimate. However, a sensitivity analysis was not completed for each of the major cost drivers. As a result, the cost estimator will not have a clear understanding of how each major cost driver is affected by a change in a single assumption and thus which scenario most affects the cost estimate. Further, as discussed previously, because a schedule risk analysis was not performed as required by best practices, the cost estimate does not include a contingency amount to account for any schedule slippage that could occur. To the extent that a schedule slippage does occur, there could ultimately be an impact on the cost estimate.

Conclusions

The Air Force did not meet best practices in developing a schedule for the DEAMS program. As a result, this raises questions about the credibility of the deadline for acquiring and implementing DEAMS to provide needed functionality for financial improvement and audit readiness. Because of these questions, the cost estimate, while following best practices, may not fully capture all costs associated with the program, particularly if there is significant schedule slippage. Moreover, Air Force management did not have a reliable schedule estimate when making its decision to invest in the DEAMS program. It is critical to correct the deficiencies identified with the schedule estimate to help ensure that the projected spending for this program is being used in the most efficient and effective manner.

Recommendation for Executive Action

To help provide for the successful implementation of DEAMS, we recommend that the Secretary of the Air Force direct the Under Secretary of the Air Force, in his capacity as the Chief Management Officer, to consider and make any necessary adjustments to the DEAMS cost estimate after addressing our prior recommendation to adopt scheduling best practices.

Agency Comments

We provided a draft of this report to DOD for review and comment. In its written comments, reprinted in appendix III, DOD concurred with our recommendation. DOD also provided a technical comment, which we incorporated.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Defense; the Secretary of the Air Force; the Assistant Secretary of Defense (Acquisition); the Deputy Chief Management Officer; the Under Secretary of Defense (Comptroller); the Under Secretary of the Air Force, in his capacity as the Chief Management Officer of the Air Force; and the Program Manager for DEAMS. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact Asif A. Khan at (202) 512-9869 or khana@gao.gov or Nabajyoti Barkakati at (202) 512-4499 or barkakatin@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made key contributions to this report are listed in appendix IV.

Appendix I: Assessment of the Defense Enterprise Accounting and Management System Program’s Schedule Estimate

This appendix provides the results of our analysis of the extent to which the Defense Enterprise Accounting and Management System (DEAMS) schedule estimate supporting the February 2012 Milestone B decision met the characteristics of a high-quality, reliable schedule. Table 3 provides the detailed results of our analysis.

GAO’s methodology includes five levels of compliance with its best practices. “Not met” means the program provided no evidence that satisfies any of the criterion. “Minimally met” means the program provided evidence that satisfies a small portion of the criterion. “Partially met” means the program provided evidence that satisfies about half of the criterion. “Substantially met” means the program provided evidence that satisfies a large portion of the criterion. “Fully met” means the program provided evidence that completely satisfies the criterion.

Appendix II: Assessment of the Defense Enterprise Accounting and Management System Program’s Cost Estimate

This appendix provides the results of our analysis of the extent to which the Defense Enterprise Accounting and Management System (DEAMS) cost estimate supporting the February 2012 Milestone B decision met the characteristics of a high-quality cost estimate. Table 4 provides the detailed results of our analysis.

GAO’s methodology includes five levels of compliance with its best practices. “Not met” means the program provided no evidence that satisfies any of the criterion. “Minimally met” means the program provided evidence that satisfies a small portion of the criterion. “Partially met” means the program provided evidence that satisfies about half of the criterion. “Substantially met” means the program provided evidence that satisfies a large portion of the criterion. “Fully met” means the program provided evidence that completely satisfies the criterion.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to the contacts named above, Cynthia Jackson (Director), Karen Richey (Assistant Director), Beatrice Alff, Jennifer Echard, Patrick Frey, and Jason Lee made key contributions to this report.