Defense Budget: Independent Review Is Needed to Ensure DOD's Use of Cost Estimating Tool for Contingency Operations Follows Best Practices
Released: 2008-09-15T00:00:00

Letter

Background

Cost estimation is a difficult process that requires both data and judgment, and seldom, if ever, are estimates precise—the goal is to find a “reasonable” estimate of future needs. Cost estimates are necessary for government programs for many reasons: for example, to support decisions about whether to fund one program over another, develop annual budget requests, or evaluate resource requirements at key decision points. As discussed in our Cost Assessment Guide, developing a good cost estimate requires stable program requirements, access to detailed documentation and historical data, well-trained and experienced cost analysts, a risk and uncertainty analysis, and the identification of a range of confidence levels. The guide also outlines 12 steps for a high-quality cost estimation process, which are to: Define the estimate’s purpose Develop the estimating plan Define the program Determine the estimating approach Identify ground rules and assumptions Obtain the data Develop the point estimate Conduct sensitivity analysis Conduct risk and uncertainty analysis Document the estimate Present estimate to management for approval Update the estimate to reflect actual costs and changes.

It is important that cost estimators and independent organizations validate that all cost elements are credible and can be justified by acceptable estimating methods, adequate data, and detailed documentation. Hence, in addition to the 12 steps of a high-quality cost estimation process, the guide also describes four best practice characteristics of a high-quality, reliable estimate generated by a sound cost estimation process. Specifically, the estimate should be well documented, comprehensive, accurate, and credible. Table 1 describes these characteristics in more detail. Adherence to these best practices can help ensure that a cost estimation process provides a reasonable estimate of how much it will cost to accomplish all tasks related to a program and that the estimate is traceable, accurate, and reflects realistic assumptions.

Appendix II presents the 12 steps of a high-quality cost estimation process mapped to the four characteristics of reliable, high-quality estimates.

Because DOD’s cost estimates for military operations in Bosnia during the 1990s were consistently well below the actual costs, DOD contracted with IDA to develop a tool to assist in developing preliminary and detailed cost estimates for contingency operations. By 1998, IDA had developed the first version of COST. The tool generates a cost estimate for a contingency operation on the basis of type of mission, duration, operational tempo or intensity, number of personnel and equipment, transportation needs, subsistence for personnel, and the originating and destination site. Accordingly, the tool contains data relating to geographic locations, military unit types, military equipment types, management and cost factors, and adjustment factors pertaining to climate, terrain, and operational intensity. Formulas within the tool draw on these data and user-defined inputs, such as the number of personnel or equipment and duration of operations, to develop cost estimates for many types of costs associated with contingency operations from predeployment to reconstitution, up to 250 line items or types of costs depending on the operation in question. The tool cannot estimate every type of cost that might be incurred for a contingency operation. Rather, it can only estimate certain incremental costs from the personnel, personnel support, operating, and transportation cost categories. However, some types of costs within those categories—such as depot-level maintenance of equipment and certain contracts, as well as all costs in the investment cost category, including procurement and military construction—are outside the scope of COST’s estimating capabilities.

Military components, including the services, are primary sources of data for the tool. Table 2 illustrates the primary sources of data from each component that uses COST to develop estimates for GWOT budget requests. Cost factors and management factors are key types of data that the tool uses to develop estimates. Cost factors function as variables in the tool, and a few examples are hardship duty pay, cost for operating and support of a ship, or average cost per ton mile for equipment airlift. Cost factors are developed from four main sources; DOD databases of record, service models for various statistics such as Air Force flying hours, DOD’s Cost of War reports, and other information provided by the service budget offices. Management factors include information such as the average metric tons per person of materiel for deployment or redeployment of the monthly flying hours of aircraft.

Services Vary in Their Reliance on COST to Develop GWOT Budget Estimates

The services use COST as required as part of their process to develop a GWOT budget request, but the degree to which each service relies on the COST results differs. Both DOD officials and IDA representatives state that COST is better suited for Army ground forces and was primarily intended to develop incremental personnel and operations cost estimates for operations with discreet phases and time frames. They noted it was not built to estimate costs for the long-term nature and broad scope of activities and needs related to operations such as GWOT. Therefore, service officials report varying degrees of confidence in the tool’s functionality and accuracy for their specific service. The Army relies on COST’s results when developing estimates related to personnel and operations, while Air Force, Navy, and Marine Corps officials often rely more on historical obligation data and other information than the COST-generated estimates. COST does not currently have the capability to estimate other types of costs, such as procurement, most equipment maintenance, and some contract costs. As a result, all the services develop this portion of GWOT budget requests using historical obligation data and other information.

Use of COST Is Required as Part of DOD’s GWOT Budget Request Process

DOD components are required to use COST as part of the GWOT budget request process. The DOD financial management regulation that provides financial policy and procedures for small-, medium-, and large-scale campaign level military contingency operations requires that COST be used to develop a cost estimate for the deployment of military personnel and equipment. The regulation further states that the DOD Comptroller will issue specific guidance providing factors and cost criteria necessary to develop an estimate, and that the COST estimate will address the funding requirements for operations and maintenance and military personnel costs. The DOD Comptroller issues guidance that directs the development of a fiscal year’s GWOT budget request. This guidance specifically directs the use of COST to calculate operations costs related to GWOT, and also provides information guiding COST use, such as the level and intensity of operations to be assumed. Much of the guidance details the type and level of detail that must be provided in supporting materials that should accompany components’ estimated GWOT budget requests.

COST does not develop estimates for items that are not attributable to the deployment or sustainment of personnel and equipment, such as procurement, most types of equipment maintenance, and certain major contracted needs and services. As discussed later, the services must use other processes to develop this portion of their GWOT budget requests. Neither DOD Comptroller guidance nor DOD financial management regulations prescribe the use of any particular method of developing estimates for these categories. A 2006 memo from the Deputy Secretary of Defense expanded allowable costs for GWOT in several categories, especially reset-related procurement and equipment maintenance. These types of costs accounted for about 70 percent of the total GWOT budget request for fiscal year 2008, which is a significant increase over previous years. Due to this increase in costs outside the tool, the COST-related portion of GWOT budget requests fell from about 80 percent to about 30 percent of the total, although the amount of the COST-generated estimate remains stable at between $40 billion and $55 billion per year.

The Army Relies More on COST to Develop Personnel and Operations Estimates, but the Other Services Rely on Their Own Estimates

The Army uses COST as intended by relying on the tool to generate an estimate for many personnel and operations costs for GWOT budget requests. During the Army’s development of an estimate using COST, minor adjustments to COST’s standard settings are made to better match realities on the ground. For example, an official developing an estimate in COST might reduce the costs for the transportation of equipment if a unit will be using equipment already in theater instead of taking equipment with them. DOD officials and IDA representatives stated that COST is better suited for Army ground forces; therefore, the Army relies on the final estimate developed by COST and submits this information to the DOD Comptroller as part of its GWOT budget request. About 40 to 45 percent of the Army’s final GWOT budget requests are typically for the operation and maintenance category of appropriations, and an Army budget office official stated that the majority of this portion is estimated by COST. Army officials further stated that COST is an effective tool for cost estimation because it is frequently updated with cost data the Army submits to IDA. An Army model and database that contain cost information for personnel and equipment are the sources for much of the Army-related data used by COST and also are primary sources for developing the Army’s base budget requests.

The Air Force, Marine Corps, and Navy all fulfill the requirement to develop an estimate for their respective service’s personnel and operations funding requirements for GWOT using COST; however, these services significantly alter the COST results to match estimates they have developed outside COST, using historical obligation data and other information. Officials from the Air Force, Marine Corps, and Navy budget offices reported various concerns regarding the functionality and accuracy of COST as reasons for relying more on historical obligation data and other information. Several budget officials from each of these services reported that COST routinely overestimated some costs. As a result, most of the changes they make to COST results, based on historical obligation data, are decreases in the amount estimated by COST. For example, a service budget official stated that for one fiscal year’s GWOT budget request, the total COST-developed estimate was $100 million more than the estimate developed by the service for the same types of costs using historical obligation data. Specifically, COST overestimated transportation costs by about $275 million, while underestimating certain personnel support costs by about $200 million, among other discrepancies. Navy officials stated that COST often overestimated some types of transportation costs and the results had to be manually adjusted to match historical obligation data or other information. Similarly, Marine Corps officials reported that most adjustments they make to the COST output result in a decrease of the estimate. An Air Force budget official stated that COST overestimated some transportation costs by about $1 billion in a prior year’s estimate, while other costs were not captured and therefore underestimated. While these discrepancies were adjusted prior to submission as a GWOT budget request, this official stated that an IDA representative had since suggested strategies to develop a more accurate transportation estimate in future uses of the tool. Furthermore, a Navy official stated that COST is unable to project certain costs associated with civilians, and hence might underestimate the total costs due to this exclusion. For example, the official stated that COST does not automatically estimate costs for civilian support positions associated with an operational unit, such as a ship or ground unit. Aside from accuracy concerns, officials from the Air Force, Marine Corps, and Navy reported that COST and the cost breakdown structure that forms the basis of COST’s organization and resulting estimations better represent Army ground forces than the unique characteristics of the other forces. Navy officials stated that COST automatically estimates food, ice, and water for all units because deployed ground forces require these items. However, the Navy funds these items for sailors on deployed ships through base budget funding because these costs are incurred regardless of a ship’s location. The tool has not been refined to accurately estimate these costs; therefore, Navy officials must manually remove these types of costs from an estimate for GWOT funding. Several service officials stated that, because of the limitations to COST, the required process of using COST to develop a cost estimate was duplicative of their preferred method of using historical obligation data and other information better suited to their specific service to develop a GWOT budget request.

Service officials stated that COST is a useful tool for estimating costs for small-scale and short-duration operations or for situations for which information is unknown or new, such as the recent troop surge or for other general rough-order-of-magnitude estimates produced early in operation planning while options are being weighed by decision makers. However, officials stated the tool does not perform as well for estimating costs for the lengthy deployment and sustainment phases associated with a large campaign such as GWOT. Furthermore, COST is not able to estimate all costs associated with GWOT.

Military Services Use Other Methods to Determine Cost Estimates for Procurement, Most Equipment Maintenance, and Some Contract Costs

Because COST does not have the capability to estimate costs such as procurement, reset-related equipment maintenance, and contracted needs and services, service budget officials report using historical obligation data, other models and formulas, and other information, such as deployment information, to estimate these costs. Reset-related procurement estimates for GWOT are devised in multiple ways. For example, to estimate costs to procure new equipment to replace lost or damaged equipment, officials stated that incident reports are tracked to provide information on how many pieces of equipment are needed to replace battle losses. For procurement to replace equipment that has reached the end of its useful life because of GWOT’s higher operating tempo, formulas, based on historical data, provide information on the normal extent of wear and tear for certain types of equipment, and wear above the normal extent is attributed to GWOT. Most types of equipment maintenance are not estimated by COST, such as intermediate- and depot- level maintenance; therefore the services again rely on historical obligation data to develop estimates for the GWOT-related costs. For example, Army logistics officials track the units that are scheduled for redeployment and develop estimates for the cost of resetting a particular unit’s equipment based on the type of brigade the unit is part of, such as a heavy brigade combat team or Stryker brigade combat team, and the average cost of resetting that type of brigade unit developed from historical obligation data, adjusted for inflation. For other items outside the scope of COST—such as the Logistics Civil Augmentation Program costs for the Army, intelligence needs, and contracts for other needs such as linguists—functional experts within each service provide the service budget offices with information to develop estimates. This information is based on contract task orders or needs assessments developed by in- theater commanders. For example, in-theater commanders submit requests for additional linguists to Army intelligence officials, and the estimated cost for these linguists is developed based on historical costs for the same type of linguist. Army budget officials stated that contracting costs, including the Logistics Civil Augmentation Program, linguist, and security services, are one of the most expensive cost categories that falls outside of COST. Officials from all services stated that, after many years of ongoing operations in support of GWOT, they believe few requirements are truly unknown or based on emerging needs, so therefore they are comfortable relying on historical obligation data and other information to develop estimates for these types of costs. For an example of how DOD develops a GWOT budget request and the use of COST and other methods of developing an estimate, see appendix III.

COST Is Frequently Updated and Refined, but Could Benefit from a Review of Adherence to Best Practices

DOD has taken steps to improve the performance and reliability of COST; however, COST could benefit from a review of the tool’s adherence to best practices for high-quality cost estimation as outlined in our Cost Assessment Guide. Revisions have been made to the tool to improve its performance, and frequent updates are made to the data used by the tool. However, a review of COST according to the best practices for cost estimation could provide decision makers with information on the extent to which the tool generates reliable estimates and identify opportunities for improvement. Our guide defines high-quality cost estimates as well documented, comprehensive, accurate, and credible. While we did not undertake a full assessment of COST against best practices, during the course of our review we identified features of COST’s estimation process that meet best practices and other features that would benefit from further review. For example, COST adheres to several best practices for a comprehensive and accurate cost estimate, such as frequent updates to the structure of COST and the data that COST uses to generate estimates. While COST appears to largely encompass the types of costs that are incurred to deploy and sustain Army ground forces and their related equipment, COST may not comprehensively and accurately estimate costs for the other services. COST also relies on GWOT obligation data from DOD’s Supplemental and Cost of War Execution Reports that we have identified as being of questionable reliability, and DOD is taking steps to improve. These might be areas for which a thorough and full review of the tool could improve the resulting estimates that are used to develop GWOT budget requests.

COST Revised and Frequently Updated in an Effort to Improve Effectiveness

IDA has made changes to the structure of COST, either at the request of the DOD Comptroller or the services, or on its own initiative, and has refined the tool many times over the past several years to improve functionality or performance. Recent refinements to COST included changes that provide the ability to alter the percentage of officer and enlisted personnel within a unit and the types of diagnostic and summary reports that the tool can create. A 3-year development effort culminated in June 2007 with the release of a new version of COST supported by new software and hardware that increased functionality and performance. This new version allows the user to simultaneously develop estimates for multiple operations within the same contingency. The services and others frequently submit new information for tool updates. For example, components are asked to submit updated cost factor data prior to the development of any request for supplemental emergency funding. Additionally, the DOD Comptroller has asked IDA to review the inputs, assumptions, and processes the services used to generate COST estimates for GWOT budget requests since fiscal year 2005. The reviews revealed issues that were consistent across the services or significant enough to warrant attention in future use of COST. For example, a review found that COST users estimated an excessive use of airlift for the movement of cargo with no scheduled cargo for the return flight. Additionally, a review identified confusion regarding the use of different operational tempo factors and pay offsets. IDA consolidated these and other issues identified in the review of fiscal years 2007 and 2008 into a lessons learned briefing and checklist to assist the services as they use COST in the development of future GWOT budget requests.

COST Could be Assessed for Adherence to Cost Estimation Best Practices

While DOD has taken steps to revise and update COST to improve effectiveness, COST has not been assessed according to best practices for cost estimation that define reliable, high-quality cost estimates. A review of regulations, guidance, and best practices for cost estimation and best practices established by professional cost analysts, and compiled in our Cost Assessment Guide, identified four characteristics of high-quality, reliable cost estimates. As shown in table 1, cost estimates should be well documented, comprehensive, accurate, and credible. DOD Comptroller officials stated they are confident in the tool’s ability to provide reasonable estimates because COST is frequently updated. However, neither DOD, nor any other entity, has assessed COST and its resulting cost estimates against these best practices. While we did not perform a full assessment of COST against best practices, during the course of our work we identified some features of COST that meet best practices for cost estimation and other areas that could benefit from further review.

Well Documented: A well documented cost estimate is based on data that have been gathered from actual historical costs and technical experts, analyzed for cost drivers, and collected from primary sources. These best practices appear to be met by COST. Additionally, any adjustments made to COST’s standard settings are flagged and must be accompanied by an explanatory note that details why information was changed, and these situations are reviewed by DOD Comptroller officials. Furthermore, best practices also require that previous cost factors and data are stored after updates so that an estimation process is repeatable and can be later verified. The newest version of COST does have this capability and IDA maintains records of cost factors and other data. However, best practices also require that data used in a model should be traced back to the source documentation and any normalization steps should be documented. The services are responsible for ensuring data are reliable and neither IDA nor DOD, including the services, traces all data back to the source documents. Additionally, IDA officials stated that unusually high or low data might be removed and no record of these actions would be kept, and best practices require that these sorts of steps should be documented. Further review could reveal if these or other areas might need more work to ensure the estimate is well documented.

Comprehensive: Estimates for personnel and operations costs developed by COST appear to meet several, but not all, of the criteria for comprehensive cost estimates. For example, the cost breakdown structure, which defines the cost elements within COST and forms the foundation of formulas within the tool, has more than three levels of detail, the structure is updated as changes occur, and each element is defined in a cost breakdown structure dictionary included in the financial management regulation for contingency operations. These steps are all considered best practices for a comprehensive cost estimate. However, our analysis of the cost breakdown structure in the financial management regulation revealed that there are some errors and ambiguities in the structure’s definitions that might allow for double counting of costs. Furthermore, while COST appears to largely encompass the types of costs that are incurred to deploy and sustain Army ground forces and their related equipment, it may not comprehensively estimate some of the types of costs incurred by the other services. For example, COST does not develop an estimate for the costs of certain Navy ground support units, such as intelligence, that are not associated with a naval fleet. Moreover, COST is not used by the Air Force to develop estimates for the cost of transporting people in non-combat situations, such as the transport of military or civilian personnel from the International Zone to a forward operating base, for example. An Air Force official stated that this is because the tool automatically assumes that any flying hour-related expenses are operational or combat-related and these costs should instead be attributed to transportation-related expenses. Further review could identify cost data or formulas within the tool that could be refined to better suit the other services or might reveal some types of costs for which COST should not be used to develop an estimate. Additionally, the tool does not comprehensively estimate all GWOT costs, such as procurement and many types of equipment maintenance. The GWOT cost estimate presented in appendix III illustrates the revisions made to COST results and the types of costs that were estimated outside of COST for a particular estimate that was to be comprised primarily of military personnel and operations costs.

Accurate: Similarly, the tool’s estimates appear to meet some, but not all, of the best practices for accuracy. For example, an accurate estimate should be based on cost factors that reflect updates and changes. IDA does ask the components to submit updated cost factors, which are sometimes based on historical obligation data, prior to every run of the tool for a fiscal year’s GWOT request, but the components are not required to update the factors. IDA reviews the cost factor submission to ensure general consistency across years, but the services and other components that submit data are ultimately responsible for the data and are not required to validate the data prior to their inclusion into the tool. According to IDA representatives, IDA does not validate the data sources or obtain assurances that the data submitted are reliable, because this requirement is not included in its contract with DOD. The services do not validate all data submitted to IDA for updates, nor do they submit updates for all factors that need updating. Additionally, some cost factors are developed from the Cost of War reports and other data are compared against these reports as a validation check. These practices might benefit from review since our previous work has raised concerns about the reliability of reported GWOT obligation data. For example, we have reported that there is a lack of transparency over certain obligations in the Cost of War reports and we have identified inaccuracies in these reports. Consequently, we were unable to ensure that DOD’s reported obligations for GWOT are complete, reliable, and accurate, and believe that they should therefore be considered approximations. However, we acknowledge that DOD has taken steps to address our recommendations and the department has several initiatives underway to further improve the reliability of GWOT obligation data. Additionally, according to service officials, many adjustments to COST results for transportation and other types of costs are made to decrease estimates. This raises concerns regarding the accuracy or applicability of some data in COST.

Credible: Many of the best practices associated with a credible cost estimate require sensitivity analysis, risk and uncertainty analysis, and a comparison against an independent cost estimate to be performed. Sensitivity analysis has been performed to understand cost drivers and this type of analysis can be performed during estimate development as characteristics of the operation change. However, IDA representatives stated that risk analysis is unnecessary since the largest sources of risk stem from the changing and unpredictable nature of warfare and policy changes that also cannot be predicted. Additionally, uncertainly analysis can assess the impact that the variability of certain unknown factors will have on resulting estimates. For example, uncertainty analysis of possible fuel price changes could result in various cost scenarios that might occur depending on future fuel prices. Finally, cost results should be compared to an independent cost estimate which is another best practice for a credible cost estimation process. A Joint Staff or other official will often develop a COST estimate to compare against an estimate developed by a service official to ensure the appropriate assumptions were used. A thorough assessment against best practices would reveal if these or other areas might need more work to ensure the estimate is sufficiently credible.

Conclusions

Estimating needs and costs that will occur in the future is not an exact science, and the use of cost estimation tools or historical obligation data, along with other factors, can be reasonable means of projecting future costs. However, due to the significant and ever-increasing size of GWOT budget requests, every attempt should be made to ensure that a cost estimation process is sound. COST has been used to generate hundreds of billions of dollars in budget requests for GWOT over the past several years and the services are required to use it. While the tool and underlying data have been refined and updated, COST’s overall effectiveness for estimating GWOT costs has not been assessed. Best practices criteria to measure COST’s extent of sufficient documentation, comprehensiveness, accuracy, and credibility could provide additional information about the tool’s effectiveness at generating high-quality cost estimates and steps of the process that could be further improved. Without a thorough review of COST according to these best practices, decision makers within both DOD and Congress cannot be assured that estimates generated by the tool are developed using valid data and sound processes. Officials from across DOD and the services report that COST performs well for predicting budget needs for ground forces and for small-scale operations of short duration, or for situations in which detailed information is unknown. However, in light of service officials’ concerns that COST does not perform as well for their needs and might generate estimates that are too high in certain areas, it is important that DOD review the applicability of COST for all of the services or investigate ways in which to make the tool better suit the needs of all the services.

Recommendations for Executive Action

To ensure that DOD budget requests for GWOT are based on a sound cost estimation process, we recommend that the Secretary of Defense direct the Under Secretary of Defense (Comptroller) to Arrange for an independent review of COST against best practices for cost estimation.

Based on that review, and taking into consideration how each service uses COST, we recommend that the Secretary of Defense direct the Under Secretary of Defense (Comptroller) to Consider options for refining COST, determine the appropriate items or types of costs for which COST should be applied, and identify methods to be used when COST is not appropriate.

Agency Comments and Our Evaluation

In written comments on a draft of this report, DOD agreed with both of our recommendations for executive action. These written comments additionally provided examples of steps DOD has taken or plans to take that it considers actions that address aspects of our recommendations. Also, DOD provided us with technical comments which we incorporated in the report where appropriate. DOD’s comments are reprinted in appendix IV.

DOD agreed with our recommendation that the DOD Comptroller arrange for an independent review of the Contingency Operations Support Tool (COST) against best practices for cost estimation. In its comments on this recommendation, DOD agreed with the concept of an independent review of the tool against best practices, and further noted that the Air Force Studies and Analyses Agency (AFSAA) conducted a review of COST’s use in developing cost estimates for the air war over Serbia. This review compared the tool’s output against the actual reported costs compiled by the Defense Finance and Accounting Service. DOD stated that the recommendations for improvement identified in the review were incorporated into the tool. While the review and the incorporation of its recommendations into the tool are positive steps, the review did not assess COST’s use in estimating the broad scope of costs that are associated with a large-scale campaign such as GWOT. For example, the scope of the AFSAA review of COST was limited to the 3-month air war over Serbia and primarily reviewed costs of the Air Force. Furthermore, the AFSAA review was not a thorough review against best practices for cost estimation, which requires cost estimates to be well documented, comprehensive, accurate, and credible. Therefore, we continue to recommend that an independent and thorough review of the tool against best practices for cost estimation be pursued. This type of review would include an assessment of the risk and uncertainty associated with the inputs to the tool and the accuracy of the underlying equations and data the tool relies on to estimate costs. DOD additionally stated that (1) COST’s factors, processes, and algorithms are updated as needed and the tool is updated to reflect changes to congressionally determined factors, such as military pay rates, and (2) COST relies on many of the same service-specific cost factors that are used during the development of baseline budgets. While we acknowledge in this report that factors and other aspects of the model are regularly updated, these actions are a check for accuracy by the users of the data and our recommendation specifically calls for an independent review of COST. Finally, DOD stated that, due to our recommendation, the DOD Comptroller will issue guidance that will be incorporated into DOD’s financial management regulation that includes a process for updating COST to ensure it reflects the most current budgetary assumptions and a process for evaluating the functionality of the model to determine if adjustments are needed. As this revision of the financial management regulation has not been finalized, we did not assess this planned action. However, this positive step, once completed, should be taken into account as part of an independent and thorough review of COST against best practices for cost estimation.

DOD agreed with our recommendation that, based on an independent review, it should consider options for refining COST, determine the appropriate items or types of costs for which COST should be applied, and identify methods to be used when COST is not appropriate. In its comments, DOD stated that it uses every opportunity to refine and improve the COST model, such as the multi-level review of COST results and assumptions during the development of GWOT budget requests. Additionally, changes to COST are made based on training sessions and feedback from COST users. Finally, DOD stated that an extensive review process is in place for each non-COST line item of GWOT budget requests. While we did not assess the process DOD uses to review non-COST line items, our report acknowledges that DOD has refined and updated COST many times as information has changed and the needs of the department have evolved. However, we reiterate our view that the tool should be subject to an independent review, and COST should be further refined based on the findings of that review, as needed.

We are sending copies of this report to other interested congressional committees; the Secretary of Defense; the Under Secretary of Defense (Comptroller), and the Director, Office of Management and Budget. Copies of this report will also be made available to others upon request. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you have any questions regarding this report, please contact Sharon Pickup at (202) 512-9619 or pickups@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Scope and Methodology

To assess how the Department of Defense (DOD) uses the Contingency Operations Support Tool (COST) and other processes to develop Global War on Terror (GWOT) budget requests, we reviewed and analyzed relevant documents and interviewed key DOD and service officials and representatives from the Institute for Defense Analyses (IDA). Documents that we used for our review included, but were not limited to, (1) relevant DOD directives, instructions, and memoranda related to budgeting processes; (2) DOD financial management regulations that provide policy and procedures for contingency operations; (3) DOD guidance for the preparation and submission of requests for incremental funding for GWOT; and (4) service budget office guidance for the preparation and submission of GWOT budget requests. We obtained testimonial evidence from officials representing the Office of the Under Secretary of Defense (Comptroller) and the Joint Staff regarding the processes used to develop GWOT budget requests and the role of COST in that process. Specifically, we obtained their perspectives on COST’s effectiveness and accuracy, as well as the processes employed to develop estimates for costs that are outside the scope of the tool’s estimating capabilities. We similarly interviewed key service officials in the financial management or budget office responsible for developing GWOT cost estimates for contingency operations in the Air Force, Army, Marine Corps, and Navy to understand their experiences using COST during the development of prior years’ GWOT budget requests, including strengths and weaknesses of the tool. We additionally interviewed service officials that were identified as functional experts for the types of costs that must be estimated outside of the tool, such as procurement, reset-level equipment maintenance, and intelligence needs. We discussed what processes they use to develop estimates for these types of costs. We attended several briefings regarding COST presented by IDA and attended training sessions on the tool. We interviewed IDA representatives about the tool and reviewed numerous briefings about COST’s use in cost estimation, the structure of the tool, and the tool’s development. Finally, in order to understand how DOD developed an actual GWOT budget request and the use of COST and other methods to develop that estimate, we asked DOD to demonstrate how an estimate was developed for a case study, which was the $6.3 billion estimate for military operations that was included as part of DOD’s October 2007 $42.3 billion amendment to the Fiscal Year 2008 GWOT supplemental request for emergency funding. We chose this example as our case study because it was a recent estimate and was assumed to be comprised primarily of military personnel and operations costs, due to the description of this estimate in the justification document for the amendment to the Fiscal Year 2008 GWOT budget request. DOD provided the initial and approved estimate for this request by service and by appropriation category. We discussed this estimate with DOD and service officials, including the assumptions and processes that were used to develop this estimate, the reasons for changes between the initial and approved estimates, and the types of costs estimated by COST or outside of COST. We did not validate the assumptions used to generate this estimate or the data DOD presented in this example.

To assess what actions DOD has taken to ensure COST adheres to best practices for cost estimation, we reviewed applicable best practices and compared DOD’s efforts against those best practices we found to be consistently associated with reliable, high-quality cost estimation. We reviewed DOD guidance regarding requirements for the verification, validation, and accreditation of tools and simulations used by DOD. We reviewed documents obtained from DOD and IDA and had discussions with DOD officials and IDA representatives about COST; for example, we reviewed numerous briefings about COST and the training manual that documents the tool’s specifications, development, and use in detail. We interviewed DOD Comptroller and IDA representatives about improvements and updates that have been made to COST and the purpose of those updates. We obtained testimonial evidence from DOD Comptroller officials and IDA representatives about the steps to develop COST and the processes that surround the tool’s use as part of developing a GWOT budget request, and identified steps that appeared to meet certain criteria of established best practices and those that appeared to warrant further review. We did not perform a full review of COST against all best practices, but presented examples of how the tool meets certain best practices to provide some context to decision makers about what might be considered strengths of the tool and we highlighted some areas that might benefit from a full and independent review of COST. These examples are meant to serve as illustrative detail to provide more information to decision makers, but should not be considered the results of a complete, thorough, and independent review of the tool.

We conducted this performance audit from July 2007 through September 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: The 12 Steps of High-Quality Cost Estimating Mapped to Best Practice Criteria

The estimate is thoroughly documented, including source data and

Define the estimate’s purpose significance, clearly detailed calculations and results, and explanations for choosing a particular method or reference.

Data have been traced back to the source documentation.

A technical baseline description is included.

All steps in developing the estimate are documented, so that a cost analyst unfamiliar with the program can recreate it quickly with the same result.

All data sources for how the data were normalized are documented.

The estimating methodology and rationale used to derive each cost breakdown structure element’s cost are described in detail.

The estimate’s level of detail ensures that cost elements are neither omitted All cost-influencing ground rules and assumptions are detailed.

The cost breakdown structure is defined and each element is described in a The estimate is unbiased, not overly conservative or overly optimistic, and based on an assessment of most likely costs.

It has few, if any, mathematical mistakes; those it has are minor.

It has been validated for errors like double counting and omitted costs.

It has been compared to the independent cost estimate for differences.

Cross-checks have been made on cost drivers to see if results are similar.

It is updated to reflect changes in technical or program assumptions and new phases or milestones.

Estimates are replaced with the earned value management estimate at completion and the independent estimate at completion from the integrated earned value management system.

Major assumptions are varied and other outcomes recomputed to determine how sensitive outcomes are to changes in the assumptions.

Risk and uncertainty analysis is performed to determine the level of risk The results are cross-checked and an independent cost estimate is developed to determine if other estimating methods produce similar results.

To better understand how the Department of Defense (DOD) develops a Global War on Terrorism (GWOT) budget request and the use of the Contingency Operations Support Tool (COST) and other methods of developing an estimate, we asked DOD to demonstrate how the services used COST and other methods to develop the $6.3 billion budget estimate for military operations that was included as part of DOD’s October 2007 $42.3 billion amendment to the fiscal year 2008 GWOT supplemental request for emergency funding. DOD justification materials for the amendment describe this $6.3 billion portion as needed to support the continued sustainment and redeployment of the five Army brigades and two Marine Corps infantry battalions that were considered part of the troop surge. This portion of the amendment was also requested to support the simultaneous deployment of combat support forces augmenting these combat forces and other costs related to the presence of the troop surge. Table 3 presents DOD’s estimate for this $6.3 billion portion of the amendment meant to fund additional operations, broken into the COST- generated portion and the portion that was estimated by other means. Typically, service budget officials develop cost estimates for GWOT, but officials stated that time constraints for this specific estimate required that Joint Staff and DOD Comptroller officials develop the initial estimate using both COST and other processes outside of COST as necessary. After discussions with service budget officials, and the September 10, 2007, testimony of the Commander of the Multi-National Force-Iraq before Congress, the estimates were revised. This is reflected in the approved estimate for each service, again broken into the COST portion and the “non-COST” portion. Each service’s estimate is further broken down into estimates for Military Personnel appropriations and Operation and Maintenance appropriations.

As shown in table 3, the Army’s estimate of nearly $5.8 billion comprises the majority of the $6.3 billion total estimate. COST-generated estimates for military personnel or operation and maintenance did not change significantly for the Army from its initial estimate to the approved final estimate. Substantial changes were made, however, to the portions of the estimate that are outside the scope of COST. Army budget officials reported that the large decrease in requested military personnel funding was due to lower mobilization levels than originally predicted, adjusted overstrength levels, and reduced permanent change of station and subsistence costs. The COST portion for both military personnel and operations for the Army decreased slightly from the initial to the approved estimate. Army budget officials reported that the approximately $2 billion increase in operation and maintenance costs that was derived outside of COST was due to force protection and other equipment or services needed to support the troop surge and the additional operations performed by surge troops. Table 4 provides more details on these costs.

Similarly, major adjustments were made to the non-COST portions of the Air Force, Marine Corps, and Navy estimates for either military personnel costs, operation and maintenance costs, or both appropriation categories, depending on the service. These changes were due to changed assumptions regarding deployment and specialized needs. For example, the Navy’s non-COST personnel estimate was increased for costs associated with permanent change of station and active duty for special work needs. The Marine Corps increased its non-COST personnel request by $163 million in anticipation of increased requirements for reserve and Individual Ready Reserve activations to active duty. The adjustments to the Marine Corps and Air Force COST-generated portions were likewise substantial. The Marine Corps’ increase for personnel costs reflected in the COST-attributed portion reflects an anticipated increase in counter- insurgency operations in Afghanistan, while the Air Force estimate was refined to exclude a KC-10 and the Air National Guard and Air Force Reserve personnel that would accompany the KC-10. From this case study, it is clear that changing the assumptions regarding the deployment of personnel and equipment can have substantial impact on the COST-related portion of a GWOT budget request. Also, significant changes were made to the non-COST portion of the request, which in this example increased from $775 million to about $2.1 billion, including the nearly $1 billion offset in Army personnel cost estimates. The information regarding the non-COST portion of this example reveals the significance and size of estimates generated outside the tool, even in a situation where the majority, if not all, of the costs would be assumed to be related to personnel and operations.

We did not validate any of the data DOD presented in the above discussion, or any of the assumptions or other information used by DOD to develop this estimate.

Appendix IV: Comments from the Department of Defense

Appendix V: GAO Contact and Staff Acknowledgments

Acknowledgements

In addition to the contact named above, Ann Borseth (Assistant Director), Grace Coleman, Susan Ditto, Linda Keefer, Lonnie McAllister II, Lisa McMillen, Charles Perdue, Suzanne Perkins, Karen Richey, and Karen Werner made key contributions to this report.