Defense Budget: Independent Review Is Needed to Ensure DOD's Use of Cost Estimating Tool for Contingency Operations Follows Best Practices
Published: 2008-09-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since the September 2001 terrorist attacks, Congress has provided about $800 billion as of July 2008 to the Department of Defense (DOD) for military operations in support of the Global War on Terrorism (GWOT). GWOT budget requests have grown in scope and the amount requested has increased every year. DOD uses various processes and the Contingency Operations Support Tool (COST) to estimate costs for these operations and to develop budget requests. GAO assessed (1) how DOD uses COST and other processes to develop GWOT budget requests and (2) what actions DOD has taken to ensure COST adheres to best practices for cost estimation. GAO interviewed DOD officials and others to determine how the services develop GWOT budget requests using COST and other processes. GAO also used its Cost Assessment Guide as criteria for best practices for cost estimation.
section_title: What GAO Found
paragraphs: The services use COST as part of their process to develop a GWOT budget request. While the Army relies more on the estimate resulting from COST, the other services adjust the results of COST to reflect estimates they generate outside of COST, based on historical obligation data and other information. DOD's financial management regulation and other guidance require components to use COST to develop an estimate for the deployment and sustainment of military personnel and equipment for ongoing operations in support of GWOT. While all services use COST to develop an initial estimate, Air Force, Marine Corps, and Navy budget officials alter the results of the tool to match information provided by lower level commands and historical obligation data that they believe are more accurate than the COST-generated estimate. These officials stated that the tool routinely overestimates some costs and therefore most changes made are decreases in the amount estimated by COST. These officials believe that the requirement to use COST to develop a GWOT budget request is a duplicative process to their preferred method of using historical obligation data and other information better suited to their specific service. For example, they stated that COST better represents the needs of Army ground forces and the tool has not been refined to be as effective for estimating needs for their service's mission. These officials also mentioned that COST is better suited for developing estimates for smaller-scale contingency operations than for the lengthy deployments and sustainment phases associated with a large campaign such as GWOT. To develop estimates for items that are outside the scope of COST, such as procurement and certain contracts, the military services rely primarily on needs assessments developed by commanders and historical obligation data. DOD has taken steps to improve the performance and reliability of COST; however, COST could benefit from an independent review of the tool's adherence to best practices for high-quality cost estimation as described in GAO's Cost Assessment Guide. COST has been refined many times and cost factors are routinely updated in an effort to use the most current information available to develop an estimate. DOD officials stated they are confident in the tool's ability to provide reasonable estimates because COST is frequently updated. However, COST has not been assessed against best practices for cost estimation to determine whether COST can provide high-quality estimates that are well documented, comprehensive, accurate, and credible. While GAO did not undertake a full assessment of COST against best practices, it determined that some features of the tool meet best practices while other features would benefit from further review. For example, the tool adheres to several best practices for a comprehensive and accurate cost estimate, such as frequent updates to the structure of COST and the data the tool uses to generate estimates. However, COST relies on GWOT obligation data that GAO has identified as being of questionable reliability. A thorough, independent review of COST against best practices could provide decision makers with information about whether the tool creates cost estimates for GWOT expenses that are well documented, comprehensive, accurate, and credible.

section_title: Letter
paragraphs: After the terrorist attacks of September 11, 2001, the President announced a global war on  terrorism requiring the collective instruments of the entire federal government to counter  the threat of terrorism. Ongoing military and diplomatic operations overseas, especially in  Iraq and Afghanistan, constitute a key part of GWOT. These operations involve a wide  variety of activities, such as combating insurgents, civil affairs, capacity building,  infrastructure reconstruction, and training military forces of other nations.
DOD, Financial Management Regulation, 7000.14-R, vol. 12, ch. 23 (September 2007).
A January 2007 DOD memorandum defines reset as actions to restore units to a desired  level of combat capability, including maintenance and supply activities that restore and  enhance equipment that was destroyed, damaged, stressed, or worn out beyond economic  repair due to combat operations by repairing, rebuilding, or procuring replacement  equipment. equipment, and certain contract costs. For the first few years of GWOT,  COST-generated estimates for personnel and operating costs ranged from  around 80 to 50 percent of the total budget request for GWOT for fiscal  years 2003 to 2005, respectively. However, this COST-generated portion of  the total has continued to decrease every year. For the fiscal year 2008  request submitted in February 2007, the COST-generated portion of the  request was about 30 percent due to a significant increase in certain types  of estimated costs, such as procurement and reset-related maintenance,  which are outside the scope of COST. Estimates for these types of costs,  or the remaining 70 percent of the fiscal year 2008 request, are developed  using other models and formulas, historical obligation data, and requests  submitted by commanders. Each component submits its estimate for its  GWOT budget request, including the COST-related portion for military  personnel and operations needs and the portions developed by other  means, to the Office of the Under Secretary of Defense (Comptroller).  These estimates are reviewed and revised by the DOD Comptroller and the  Office of Management and Budget before they are submitted to Congress.
Because federal guidelines are limited on the processes, procedures, and  practices for ensuring credible cost estimates, we reviewed existing best  practices and developed the Cost Assessment Guide to fill that gap. The  best practices identified in our guide were compiled from a thorough  review of relevant legislation, regulations, policy, and guidance for the  criteria that most pertained to cost estimating. Our guide defines cost  estimation as the process of using established methods and valid data to  estimate the future costs of a program based on what is known today, such  as the use of COST to develop a portion of GWOT budget requests. DOD  and the Office of Management and Budget have issued the majority of  guidance and regulations regarding cost estimating, and the best practices  in our guide were compiled from these and other sources. The  characteristics of high-quality cost estimation support a process that is  well documented, comprehensive, accurate, and credible.
Over the past several years, we have conducted a series of reviews under  the authority of the Comptroller General examining the funding and  reported obligations for operations in support of GWOT. We have  previously reported that there are reliability concerns regarding DOD’s  reported GWOT obligations, and problems exist with transparency over  certain costs. We have made a series of recommendations to the Secretary  of Defense intended to improve the transparency and reliability of DOD’s  reporting of GWOT obligations and to adjust GWOT funding requests.  DOD has implemented many of these prior recommendations and  continues to improve its cost reporting efforts. Because of broad  congressional interest in issues related to GWOT, we prepared this report  under the Comptroller General’s authority to conduct evaluations on his  own initiative. Due to increasing budget requests for DOD’s ongoing  operations in support of GWOT and to assist members of Congress in their  oversight role as they consider future GWOT funding, we evaluated DOD’s  efforts to develop its GWOT budget requests. Specifically, we assessed    (1) how DOD uses COST and other processes to develop GWOT budget  requests and (2) what actions DOD has taken to ensure COST adheres to  best practices for cost estimation.
To examine how DOD uses COST and other processes to develop budget  requests for GWOT, we met with key officials from the DOD Comptroller,  the Joint Chiefs of Staff, and various service officials from the Air Force,  Army, Marine Corps, and Navy to understand their use of COST to develop  estimates for the deployment and sustainment of troops and equipment  and about other processes used to develop portions of GWOT budget  requests that are outside the scope of COST, such as procurement and  certain types of contracts. We also obtained their perspectives on their  experiences using COST and other processes to develop a cost estimate  for yearly emergency GWOT budget requests or amendments to those  requests. We interviewed representatives from the Institute for Defense  Analyses (IDA)—the contractor that developed and maintains COST—to  understand the development, function, and role of COST. We reviewed  DOD guidance and regulations as well as COST-related documents such as  briefings and the COST training manual. We also analyzed an example of  how the cost estimate for the operations portion of an amendment to a  yearly GWOT supplemental request was developed. To assess the actions  DOD has taken to ensure COST adheres to best practices for cost  estimation, we reviewed COST-related documents, such as the manual for  COST use and financial management regulations. Our Cost Assessment  Guide provides best practices for cost estimating processes. We  interviewed DOD and service officials and representatives from IDA about  steps that have been taken to improve COST’s effectiveness and  functionality. Finally, we analyzed documents and testimonial evidence  from DOD officials and IDA representatives and compared this  information to the four characteristics of high-quality cost estimation.  While we did not conduct a full assessment of COST against best  practices, we identified some features of the tool that either met best  practices or appeared to not meet best practices.
We conducted this performance audit from July 2007 through      September 2008 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Cost estimation is a difficult process that requires both data and judgment,  and seldom, if ever, are estimates precise—the goal is to find a  “reasonable” estimate of future needs. Cost estimates are necessary for  government programs for many reasons: for example, to support decisions  about whether to fund one program over another, develop annual budget  requests, or evaluate resource requirements at key decision points. As  discussed in our Cost Assessment Guide, developing a good cost estimate  requires stable program requirements, access to detailed documentation  and historical data, well-trained and experienced cost analysts, a risk and  uncertainty analysis, and the identification of a range of confidence levels.  The guide also outlines 12 steps for a high-quality cost estimation process,  which are to:    Define the estimate’s purpose    Develop the estimating plan    Define the program    Determine the estimating approach   Identify ground rules and assumptions    Obtain the data    Develop the point estimate    Conduct sensitivity analysis    Conduct risk and uncertainty analysis    Document the estimate    Present estimate to management for approval    Update the estimate to reflect actual costs and changes.
It is important that cost estimators and independent organizations validate  that all cost elements are credible and can be justified by acceptable  estimating methods, adequate data, and detailed documentation. Hence, in  addition to the 12 steps of a high-quality cost estimation process, the guide  also describes four best practice characteristics of a high-quality, reliable  estimate generated by a sound cost estimation process. Specifically, the  estimate should be well documented, comprehensive, accurate, and  credible. Table 1 describes these characteristics in more detail. Adherence  to these best practices can help ensure that a cost estimation process  provides a reasonable estimate of how much it will cost to accomplish all  tasks related to a program and that the estimate is traceable, accurate, and  reflects realistic assumptions.
Appendix II presents the 12 steps of a high-quality cost estimation process  mapped to the four characteristics of reliable, high-quality estimates.
Because DOD’s cost estimates for military operations in Bosnia during the  1990s were consistently well below the actual costs, DOD contracted with  IDA to develop a tool to assist in developing preliminary and detailed cost  estimates for contingency operations. By 1998, IDA had developed the first  version of COST. The tool generates a cost estimate for a contingency  operation on the basis of type of mission, duration, operational tempo or  intensity, number of personnel and equipment, transportation needs,  subsistence for personnel, and the originating and destination site.  Accordingly, the tool contains data relating to geographic locations,  military unit types, military equipment types, management and cost  factors, and adjustment factors pertaining to climate, terrain, and  operational intensity. Formulas within the tool draw on these data and  user-defined inputs, such as the number of personnel or equipment and  duration of operations, to develop cost estimates for many types of costs  associated with contingency operations from predeployment to  reconstitution, up to 250 line items or types of costs depending on the  operation in question. The tool cannot estimate every type of cost that  might be incurred for a contingency operation. Rather, it can only estimate  certain incremental costs from the personnel, personnel support,  operating, and transportation cost categories. However, some types of  costs within those categories—such as depot-level maintenance of  equipment and certain contracts, as well as all costs in the investment cost  category, including procurement and military construction—are outside  the scope of COST’s estimating capabilities.
Military components, including the services, are primary sources of data  for the tool. Table 2 illustrates the primary sources of data from each  component that uses COST to develop estimates for GWOT budget  requests. Cost factors and management factors are key types of data that  the tool uses to develop estimates. Cost factors function as variables in the  tool, and a few examples are hardship duty pay, cost for operating and  support of a ship, or average cost per ton mile for equipment airlift. Cost  factors are developed from four main sources; DOD databases of record,  service models for various statistics such as Air Force flying hours, DOD’s  Cost of War reports, and other information provided by the service budget  offices. Management factors include information such as the average  metric tons per person of materiel for deployment or redeployment of the  monthly flying hours of aircraft.
section_title: Services Vary in Their Reliance on COST to Develop GWOT Budget Estimates
paragraphs: The services use COST as required as part of their process to develop a  GWOT budget request, but the degree to which each service relies on the  COST results differs. Both DOD officials and IDA representatives state  that COST is better suited for Army ground forces and was primarily  intended to develop incremental personnel and operations cost estimates  for operations with discreet phases and time frames. They noted it was not  built to estimate costs for the long-term nature and broad scope of  activities and needs related to operations such as GWOT. Therefore,  service officials report varying degrees of confidence in the tool’s  functionality and accuracy for their specific service. The Army relies on  COST’s results when developing estimates related to personnel and  operations, while Air Force, Navy, and Marine Corps officials often rely  more on historical obligation data and other information than the       COST-generated estimates. COST does not currently have the capability to  estimate other types of costs, such as procurement, most equipment  maintenance, and some contract costs. As a result, all the services develop  this portion of GWOT budget requests using historical obligation data and  other information.
subsections: section_title: Use of COST Is Required as Part of DOD’s GWOT Budget Request Process
paragraphs: DOD components are required to use COST as part of the GWOT budget  request process. The DOD financial management regulation that provides  financial policy and procedures for small-, medium-, and large-scale  campaign level military contingency operations requires that COST be  used to develop a cost estimate for the deployment of military personnel  and equipment. The regulation further states that the DOD Comptroller  will issue specific guidance providing factors and cost criteria necessary to  develop an estimate, and that the COST estimate will address the funding  requirements for operations and maintenance and military personnel  costs. The DOD Comptroller issues guidance that directs the development  of a fiscal year’s GWOT budget request. This guidance specifically directs  the use of COST to calculate operations costs related to GWOT, and also  provides information guiding COST use, such as the level and intensity of  operations to be assumed. Much of the guidance details the type and level  of detail that must be provided in supporting materials that should  accompany components’ estimated GWOT budget requests.
COST does not develop estimates for items that are not attributable to the  deployment or sustainment of personnel and equipment, such as  procurement, most types of equipment maintenance, and certain major  contracted needs and services. As discussed later, the services must use  other processes to develop this portion of their GWOT budget requests.  Neither DOD Comptroller guidance nor DOD financial management  regulations prescribe the use of any particular method of developing  estimates for these categories. A 2006 memo from the Deputy Secretary of  Defense expanded allowable costs for GWOT in several categories,  especially reset-related procurement and equipment maintenance. These  types of costs accounted for about 70 percent of the total GWOT budget  request for fiscal year 2008, which is a significant increase over previous  years. Due to this increase in costs outside the tool, the COST-related  portion of GWOT budget requests fell from about 80 percent to about        30 percent of the total, although the amount of the COST-generated  estimate remains stable at between $40 billion and $55 billion per year.
section_title: The Army Relies More on COST to Develop Personnel and Operations Estimates, but the Other Services Rely on Their Own Estimates
paragraphs: The Army uses COST as intended by relying on the tool to generate an  estimate for many personnel and operations costs for GWOT budget  requests. During the Army’s development of an estimate using COST,  minor adjustments to COST’s standard settings are made to better match  realities on the ground. For example, an official developing an estimate in  COST might reduce the costs for the transportation of equipment if a unit  will be using equipment already in theater instead of taking equipment  with them. DOD officials and IDA representatives stated that COST is  better suited for Army ground forces; therefore, the Army relies on the  final estimate developed by COST and submits this information to the  DOD Comptroller as part of its GWOT budget request. About 40 to 45  percent of the Army’s final GWOT budget requests are typically for the  operation and maintenance category of appropriations, and an Army  budget office official stated that the majority of this portion is estimated  by COST. Army officials further stated that COST is an effective tool for  cost estimation because it is frequently updated with cost data the Army  submits to IDA. An Army model and database that contain cost  information for personnel and equipment are the sources for much of the  Army-related data used by COST and also are primary sources for  developing the Army’s base budget requests.
The Air Force, Marine Corps, and Navy all fulfill the requirement to  develop an estimate for their respective service’s personnel and operations  funding requirements for GWOT using COST; however, these services  significantly alter the COST results to match estimates they have  developed outside COST, using historical obligation data and other  information. Officials from the Air Force, Marine Corps, and Navy budget  offices reported various concerns regarding the functionality and accuracy  of COST as reasons for relying more on historical obligation data and  other information. Several budget officials from each of these services  reported that COST routinely overestimated some costs. As a result, most  of the changes they make to COST results, based on historical obligation  data, are decreases in the amount estimated by COST. For example, a  service budget official stated that for one fiscal year’s GWOT budget  request, the total COST-developed estimate was $100 million more than  the estimate developed by the service for the same types of costs using  historical obligation data. Specifically, COST overestimated transportation  costs by about $275 million, while underestimating certain personnel  support costs by about $200 million, among other discrepancies. Navy  officials stated that COST often overestimated some types of  transportation costs and the results had to be manually adjusted to match  historical obligation data or other information. Similarly, Marine Corps  officials reported that most adjustments they make to the COST output  result in a decrease of the estimate. An Air Force budget official stated  that COST overestimated some transportation costs by about $1 billion in  a prior year’s estimate, while other costs were not captured and therefore  underestimated. While these discrepancies were adjusted prior to  submission as a GWOT budget request, this official stated that an IDA  representative had since suggested strategies to develop a more accurate  transportation estimate in future uses of the tool. Furthermore, a Navy  official stated that COST is unable to project certain costs associated with  civilians, and hence might underestimate the total costs due to this  exclusion. For example, the official stated that COST does not  automatically estimate costs for civilian support positions associated with  an operational unit, such as a ship or ground unit. Aside from accuracy  concerns, officials from the Air Force, Marine Corps, and Navy reported  that COST and the cost breakdown structure that forms the basis of  COST’s organization and resulting estimations better represent Army  ground forces than the unique characteristics of the other forces. Navy  officials stated that COST automatically estimates food, ice, and water for  all units because deployed ground forces require these items. However,  the Navy funds these items for sailors on deployed ships through base  budget funding because these costs are incurred regardless of a ship’s  location. The tool has not been refined to accurately estimate these costs;  therefore, Navy officials must manually remove these types of costs from  an estimate for GWOT funding. Several service officials stated that,  because of the limitations to COST, the required process of using COST to  develop a cost estimate was duplicative of their preferred method of using  historical obligation data and other information better suited to their  specific service to develop a GWOT budget request.
Service officials stated that COST is a useful tool for estimating costs for  small-scale and short-duration operations or for situations for which  information is unknown or new, such as the recent troop surge or for  other general rough-order-of-magnitude estimates produced early in  operation planning while options are being weighed by decision makers.  However, officials stated the tool does not perform as well for estimating  costs for the lengthy deployment and sustainment phases associated with  a large campaign such as GWOT. Furthermore, COST is not able to  estimate all costs associated with GWOT.
section_title: Military Services Use Other Methods to Determine Cost Estimates for Procurement, Most Equipment Maintenance, and Some Contract Costs
paragraphs: Because COST does not have the capability to estimate costs such as  procurement, reset-related equipment maintenance, and contracted needs  and services, service budget officials report using historical obligation  data, other models and formulas, and other information, such as  deployment information, to estimate these costs. Reset-related  procurement estimates for GWOT are devised in multiple ways. For  example, to estimate costs to procure new equipment to replace lost or  damaged equipment, officials stated that incident reports are tracked to  provide information on how many pieces of equipment are needed to  replace battle losses. For procurement to replace equipment that has  reached the end of its useful life because of GWOT’s higher operating  tempo, formulas, based on historical data, provide information on the  normal extent of wear and tear for certain types of equipment, and wear  above the normal extent is attributed to GWOT. Most types of equipment  maintenance are not estimated by COST, such as intermediate- and depot- level maintenance; therefore the services again rely on historical  obligation data to develop estimates for the GWOT-related costs. For  example, Army logistics officials track the units that are scheduled for  redeployment and develop estimates for the cost of resetting a particular  unit’s equipment based on the type of brigade the unit is part of, such as a  heavy brigade combat team or Stryker brigade combat team, and the  average cost of resetting that type of brigade unit developed from  historical obligation data, adjusted for inflation. For other items outside  the scope of COST—such as the Logistics Civil Augmentation Program  costs for the Army, intelligence needs, and contracts for other needs such  as linguists—functional experts within each service provide the service  budget offices with information to develop estimates. This information is  based on contract task orders or needs assessments developed by in- theater commanders. For example, in-theater commanders submit  requests for additional linguists to Army intelligence officials, and the  estimated cost for these linguists is developed based on historical costs for  the same type of linguist. Army budget officials stated that contracting  costs, including the Logistics Civil Augmentation Program, linguist, and  security services, are one of the most expensive cost categories that falls  outside of COST. Officials from all services stated that, after many years of  ongoing operations in support of GWOT, they believe few requirements  are truly unknown or based on emerging needs, so therefore they are  comfortable relying on historical obligation data and other information to  develop estimates for these types of costs. For an example of how DOD  develops a GWOT budget request and the use of COST and other methods  of developing an estimate, see appendix III.
section_title: COST Is Frequently Updated and Refined, but Could Benefit from a Review of Adherence to Best Practices
paragraphs: DOD has taken steps to improve the performance and reliability of COST;  however, COST could benefit from a review of the tool’s adherence to best  practices for high-quality cost estimation as outlined in our Cost  Assessment Guide. Revisions have been made to the tool to improve its  performance, and frequent updates are made to the data used by the tool.  However, a review of COST according to the best practices for cost  estimation could provide decision makers with information on the extent  to which the tool generates reliable estimates and identify opportunities  for improvement. Our guide defines high-quality cost estimates as well  documented, comprehensive, accurate, and credible. While we did not  undertake a full assessment of COST against best practices, during the  course of our review we identified features of COST’s estimation process  that meet best practices and other features that would benefit from further  review. For example, COST adheres to several best practices for a  comprehensive and accurate cost estimate, such as frequent updates to  the structure of COST and the data that COST uses to generate estimates.  While COST appears to largely encompass the types of costs that are  incurred to deploy and sustain Army ground forces and their related  equipment, COST may not comprehensively and accurately estimate costs  for the other services. COST also relies on GWOT obligation data from  DOD’s Supplemental and Cost of War Execution Reports that we have  identified as being of questionable reliability, and DOD is taking steps to  improve. These might be areas for which a thorough and full review of  the tool could improve the resulting estimates that are used to develop  GWOT budget requests.
subsections: section_title: COST Revised and Frequently Updated in an Effort to Improve Effectiveness
paragraphs: IDA has made changes to the structure of COST, either at the request of  the DOD Comptroller or the services, or on its own initiative, and has  refined the tool many times over the past several years to improve  functionality or performance. Recent refinements to COST included  changes that provide the ability to alter the percentage of officer and  enlisted personnel within a unit and the types of diagnostic and summary  reports that the tool can create. A 3-year development effort culminated in  June 2007 with the release of a new version of COST supported by new  software and hardware that increased functionality and performance. This  new version allows the user to simultaneously develop estimates for  multiple operations within the same contingency. The services and others  frequently submit new information for tool updates. For example,  components are asked to submit updated cost factor data prior to the  development of any request for supplemental emergency funding.  Additionally, the DOD Comptroller has asked IDA to review the inputs,  assumptions, and processes the services used to generate COST estimates  for GWOT budget requests since fiscal year 2005. The reviews revealed  issues that were consistent across the services or significant enough to  warrant attention in future use of COST. For example, a review found that  COST users estimated an excessive use of airlift for the movement of  cargo with no scheduled cargo for the return flight. Additionally, a review  identified confusion regarding the use of different operational tempo  factors and pay offsets. IDA consolidated these and other issues identified  in the review of fiscal years 2007 and 2008 into a lessons learned briefing  and checklist to assist the services as they use COST in the development  of future GWOT budget requests.
section_title: COST Could be Assessed for Adherence to Cost Estimation Best Practices
paragraphs: While DOD has taken steps to revise and update COST to improve  effectiveness, COST has not been assessed according to best practices for  cost estimation that define reliable, high-quality cost estimates. A review  of regulations, guidance, and best practices for cost estimation and best  practices established by professional cost analysts, and compiled in our  Cost Assessment Guide, identified four characteristics of high-quality,  reliable cost estimates. As shown in table 1, cost estimates should be well  documented, comprehensive, accurate, and credible. DOD Comptroller  officials stated they are confident in the tool’s ability to provide reasonable  estimates because COST is frequently updated. However, neither DOD, nor  any other entity, has assessed COST and its resulting cost estimates  against these best practices. While we did not perform a full assessment of  COST against best practices, during the course of our work we identified  some features of COST that meet best practices for cost estimation and  other areas that could benefit from further review.
Well Documented: A well documented cost estimate is based on data  that have been gathered from actual historical costs and technical  experts, analyzed for cost drivers, and collected from primary sources.  These best practices appear to be met by COST. Additionally, any  adjustments made to COST’s standard settings are flagged and must be  accompanied by an explanatory note that details why information was  changed, and these situations are reviewed by DOD Comptroller  officials. Furthermore, best practices also require that previous cost  factors and data are stored after updates so that an estimation process  is repeatable and can be later verified. The newest version of COST  does have this capability and IDA maintains records of cost factors and  other data. However, best practices also require that data used in a  model should be traced back to the source documentation and any  normalization steps should be documented. The services are  responsible for ensuring data are reliable and neither IDA nor DOD,  including the services, traces all data back to the source documents.  Additionally, IDA officials stated that unusually high or low data might  be removed and no record of these actions would be kept, and best  practices require that these sorts of steps should be documented.  Further review could reveal if these or other areas might need more  work to ensure the estimate is well documented.
Comprehensive: Estimates for personnel and operations costs  developed by COST appear to meet several, but not all, of the criteria  for comprehensive cost estimates. For example, the cost breakdown  structure, which defines the cost elements within COST and forms the  foundation of formulas within the tool, has more than three levels of  detail, the structure is updated as changes occur, and each element is  defined in a cost breakdown structure dictionary included in the  financial management regulation for contingency operations. These  steps are all considered best practices for a comprehensive cost  estimate. However, our analysis of the cost breakdown structure in the  financial management regulation revealed that there are some errors  and ambiguities in the structure’s definitions that might allow for  double counting of costs. Furthermore, while COST appears to largely  encompass the types of costs that are incurred to deploy and sustain  Army ground forces and their related equipment, it may not  comprehensively estimate some of the types of costs incurred by the  other services. For example, COST does not develop an estimate for  the costs of certain Navy ground support units, such as intelligence,  that are not associated with a naval fleet. Moreover, COST is not used  by the Air Force to develop estimates for the cost of transporting  people in non-combat situations, such as the transport of military or  civilian personnel from the International Zone to a forward operating  base, for example. An Air Force official stated that this is because the  tool automatically assumes that any flying hour-related expenses are  operational or combat-related and these costs should instead be  attributed to transportation-related expenses. Further review could  identify cost data or formulas within the tool that could be refined to  better suit the other services or might reveal some types of costs for  which COST should not be used to develop an estimate. Additionally,  the tool does not comprehensively estimate all GWOT costs, such as  procurement and many types of equipment maintenance. The GWOT  cost estimate presented in appendix III illustrates the revisions made to  COST results and the types of costs that were estimated outside of  COST for a particular estimate that was to be comprised primarily of  military personnel and operations costs.
Accurate: Similarly, the tool’s estimates appear to meet some, but not  all, of the best practices for accuracy. For example, an accurate  estimate should be based on cost factors that reflect updates and  changes. IDA does ask the components to submit updated cost factors,  which are sometimes based on historical obligation data, prior to every  run of the tool for a fiscal year’s GWOT request, but the components  are not required to update the factors. IDA reviews the cost factor  submission to ensure general consistency across years, but the services  and other components that submit data are ultimately responsible for  the data and are not required to validate the data prior to their  inclusion into the tool. According to IDA representatives, IDA does not  validate the data sources or obtain assurances that the data submitted  are reliable, because this requirement is not included in its contract  with DOD. The services do not validate all data submitted to IDA for  updates, nor do they submit updates for all factors that need updating.  Additionally, some cost factors are developed from the Cost of War  reports and other data are compared against these reports as a  validation check. These practices might benefit from review since our  previous work has raised concerns about the reliability of reported  GWOT obligation data. For example, we have reported that there is a  lack of transparency over certain obligations in the Cost of War reports  and we have identified inaccuracies in these reports. Consequently, we  were unable to ensure that DOD’s reported obligations for GWOT are  complete, reliable, and accurate, and believe that they should therefore  be considered approximations. However, we acknowledge that DOD  has taken steps to address our recommendations and the department  has several initiatives underway to further improve the reliability of  GWOT obligation data. Additionally, according to service officials,  many adjustments to COST results for transportation and other types  of costs are made to decrease estimates. This raises concerns regarding  the accuracy or applicability of some data in COST.
Credible: Many of the best practices associated with a credible cost  estimate require sensitivity analysis, risk and uncertainty analysis, and  a comparison against an independent cost estimate to be performed.  Sensitivity analysis has been performed to understand cost drivers and  this type of analysis can be performed during estimate development as  characteristics of the operation change. However, IDA representatives  stated that risk analysis is unnecessary since the largest sources of risk  stem from the changing and unpredictable nature of warfare and policy  changes that also cannot be predicted. Additionally, uncertainly  analysis can assess the impact that the variability of certain unknown  factors will have on resulting estimates. For example, uncertainty  analysis of possible fuel price changes could result in various cost  scenarios that might occur depending on future fuel prices. Finally,  cost results should be compared to an independent cost estimate which  is another best practice for a credible cost estimation process. A Joint  Staff or other official will often develop a COST estimate to compare  against an estimate developed by a service official to ensure the  appropriate assumptions were used. A thorough assessment against  best practices would reveal if these or other areas might need more  work to ensure the estimate is sufficiently credible.
section_title: Conclusions
paragraphs: Estimating needs and costs that will occur in the future is not an exact  science, and the use of cost estimation tools or historical obligation data,  along with other factors, can be reasonable means of projecting future  costs. However, due to the significant and ever-increasing size of GWOT  budget requests, every attempt should be made to ensure that a cost  estimation process is sound. COST has been used to generate hundreds of  billions of dollars in budget requests for GWOT over the past several years  and the services are required to use it. While the tool and underlying data  have been refined and updated, COST’s overall effectiveness for estimating  GWOT costs has not been assessed. Best practices criteria to measure  COST’s extent of sufficient documentation, comprehensiveness, accuracy,  and credibility could provide additional information about the tool’s  effectiveness at generating high-quality cost estimates and steps of the  process that could be further improved. Without a thorough review of  COST according to these best practices, decision makers within both DOD  and Congress cannot be assured that estimates generated by the tool are  developed using valid data and sound processes. Officials from across  DOD and the services report that COST performs well for predicting  budget needs for ground forces and for small-scale operations of short  duration, or for situations in which detailed information is unknown.  However, in light of service officials’ concerns that COST does not  perform as well for their needs and might generate estimates that are too  high in certain areas, it is important that DOD review the applicability of  COST for all of the services or investigate ways in which to make the tool  better suit the needs of all the services.
section_title: Recommendations for Executive Action
paragraphs: To ensure that DOD budget requests for GWOT are based on a sound cost  estimation process, we recommend that the Secretary of Defense direct  the Under Secretary of Defense (Comptroller) to    Arrange for an independent review of COST against best practices for  cost estimation.
Based on that review, and taking into consideration how each service uses  COST, we recommend that the Secretary of Defense direct the Under  Secretary of Defense (Comptroller) to    Consider options for refining COST, determine the appropriate items or  types of costs for which COST should be applied, and identify methods to  be used when COST is not appropriate.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, DOD agreed with both of  our recommendations for executive action. These written comments  additionally provided examples of steps DOD has taken or plans to take  that it considers actions that address aspects of our recommendations.  Also, DOD provided us with technical comments which we incorporated in  the report where appropriate. DOD’s comments are reprinted in     appendix IV.
DOD agreed with our recommendation that the DOD Comptroller arrange  for an independent review of the Contingency Operations Support Tool  (COST) against best practices for cost estimation. In its comments on this  recommendation, DOD agreed with the concept of an independent review  of the tool against best practices, and further noted that the Air Force  Studies and Analyses Agency (AFSAA) conducted a review of COST’s use  in developing cost estimates for the air war over Serbia. This review  compared the tool’s output against the actual reported costs compiled by  the Defense Finance and Accounting Service. DOD stated that the  recommendations for improvement identified in the review were  incorporated into the tool. While the review and the incorporation of its  recommendations into the tool are positive steps, the review did not  assess COST’s use in estimating the broad scope of costs that are  associated with a large-scale campaign such as GWOT. For example, the  scope of the AFSAA review of COST was limited to the 3-month air war  over Serbia and primarily reviewed costs of the Air Force. Furthermore,  the AFSAA review was not a thorough review against best practices for  cost estimation, which requires cost estimates to be well documented,  comprehensive, accurate, and credible. Therefore, we continue to  recommend that an independent and thorough review of the tool against  best practices for cost estimation be pursued. This type of review would  include an assessment of the risk and uncertainty associated with the  inputs to the tool and the accuracy of the underlying equations and data  the tool relies on to estimate costs. DOD additionally stated that               (1) COST’s factors, processes, and algorithms are updated as needed and  the tool is updated to reflect changes to congressionally determined  factors, such as military pay rates, and (2) COST relies on many of the  same service-specific cost factors that are used during the development of  baseline budgets. While we acknowledge in this report that factors and  other aspects of the model are regularly updated, these actions are a check  for accuracy by the users of the data and our recommendation specifically  calls for an independent review of COST. Finally, DOD stated that, due to  our recommendation, the DOD Comptroller will issue guidance that will  be incorporated into DOD’s financial management regulation that includes  a process for updating COST to ensure it reflects the most current  budgetary assumptions and a process for evaluating the functionality of  the model to determine if adjustments are needed. As this revision of the  financial management regulation has not been finalized, we did not assess  this planned action. However, this positive step, once completed, should  be taken into account as part of an independent and thorough review of  COST against best practices for cost estimation.
DOD agreed with our recommendation that, based on an independent  review, it should consider options for refining COST, determine the  appropriate items or types of costs for which COST should be applied, and  identify methods to be used when COST is not appropriate. In its  comments, DOD stated that it uses every opportunity to refine and  improve the COST model, such as the multi-level review of COST results  and assumptions during the development of GWOT budget requests.  Additionally, changes to COST are made based on training sessions and  feedback from COST users. Finally, DOD stated that an extensive review  process is in place for each non-COST line item of GWOT budget requests.  While we did not assess the process DOD uses to review non-COST line  items, our report acknowledges that DOD has refined and updated COST  many times as information has changed and the needs of the department  have evolved. However, we reiterate our view that the tool should be  subject to an independent review, and COST should be further refined  based on the findings of that review, as needed.
We are sending copies of this report to other interested congressional  committees; the Secretary of Defense; the Under Secretary of Defense  (Comptroller), and the Director, Office of Management and Budget. Copies  of this report will also be made available to others upon request. In  addition, this report will be available at no charge on the GAO Web site at  http://www.gao.gov.
If you have any questions regarding this report, please contact Sharon  Pickup at (202) 512-9619 or pickups@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this report. GAO staff who made major contributions to this  report are listed in appendix V.
section_title: Appendix I: Scope and Methodology
paragraphs: To assess how the Department of Defense (DOD) uses the Contingency  Operations Support Tool (COST) and other processes to develop Global  War on Terror (GWOT) budget requests, we reviewed and analyzed  relevant documents and interviewed key DOD and service officials and  representatives from the Institute for Defense Analyses (IDA). Documents  that we used for our review included, but were not limited to, (1) relevant  DOD directives, instructions, and memoranda related to budgeting  processes; (2) DOD financial management regulations that provide policy  and procedures for contingency operations; (3) DOD guidance for the  preparation and submission of requests for incremental funding for  GWOT; and (4) service budget office guidance for the preparation and  submission of GWOT budget requests. We obtained testimonial evidence  from officials representing the Office of the Under Secretary of Defense  (Comptroller) and the Joint Staff regarding the processes used to develop  GWOT budget requests and the role of COST in that process. Specifically,  we obtained their perspectives on COST’s effectiveness and accuracy, as  well as the processes employed to develop estimates for costs that are  outside the scope of the tool’s estimating capabilities. We similarly  interviewed key service officials in the financial management or budget  office responsible for developing GWOT cost estimates for contingency  operations in the Air Force, Army, Marine Corps, and Navy to understand  their experiences using COST during the development of prior years’  GWOT budget requests, including strengths and weaknesses of the tool.  We additionally interviewed service officials that were identified as  functional experts for the types of costs that must be estimated outside of  the tool, such as procurement, reset-level equipment maintenance, and  intelligence needs. We discussed what processes they use to develop  estimates for these types of costs. We attended several briefings regarding  COST presented by IDA and attended training sessions on the tool. We  interviewed IDA representatives about the tool and reviewed numerous  briefings about COST’s use in cost estimation, the structure of the tool,  and the tool’s development. Finally, in order to understand how DOD  developed an actual GWOT budget request and the use of COST and other  methods to develop that estimate, we asked DOD to demonstrate how an  estimate was developed for a case study, which was the $6.3 billion  estimate for military operations that was included as part of DOD’s  October 2007 $42.3 billion amendment to the Fiscal Year 2008 GWOT  supplemental request for emergency funding. We chose this example as  our case study because it was a recent estimate and was assumed to be  comprised primarily of military personnel and operations costs, due to the  description of this estimate in the justification document for the  amendment to the Fiscal Year 2008 GWOT budget request. DOD provided  the initial and approved estimate for this request by service and by  appropriation category. We discussed this estimate with DOD and service  officials, including the assumptions and processes that were used to  develop this estimate, the reasons for changes between the initial and  approved estimates, and the types of costs estimated by COST or outside  of COST. We did not validate the assumptions used to generate this  estimate or the data DOD presented in this example.
To assess what actions DOD has taken to ensure COST adheres to best  practices for cost estimation, we reviewed applicable best practices and  compared DOD’s efforts against those best practices we found to be  consistently associated with reliable, high-quality cost estimation. We  reviewed DOD guidance regarding requirements for the verification,  validation, and accreditation of tools and simulations used by DOD. We  reviewed documents obtained from DOD and IDA and had discussions  with DOD officials and IDA representatives about COST; for example, we  reviewed numerous briefings about COST and the training manual that  documents the tool’s specifications, development, and use in detail. We  interviewed DOD Comptroller and IDA representatives about  improvements and updates that have been made to COST and the purpose  of those updates. We obtained testimonial evidence from DOD  Comptroller officials and IDA representatives about the steps to develop  COST and the processes that surround the tool’s use as part of developing  a GWOT budget request, and identified steps that appeared to meet certain  criteria of established best practices and those that appeared to warrant  further review. We did not perform a full review of COST against all best  practices, but presented examples of how the tool meets certain best  practices to provide some context to decision makers about what might be  considered strengths of the tool and we highlighted some areas that might  benefit from a full and independent review of COST. These examples are  meant to serve as illustrative detail to provide more information to  decision makers, but should not be considered the results of a complete,  thorough, and independent review of the tool.
We conducted this performance audit from July 2007 through      September 2008 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
section_title: Appendix II: The 12 Steps of High-Quality Cost Estimating Mapped to Best Practice Criteria
subsections: section_title: The estimate is thoroughly documented, including source data and
paragraphs: Define the estimate’s purpose  significance, clearly detailed calculations and results, and explanations for  choosing a particular method or reference.
Data have been traced back to the source documentation.
A technical baseline description is included.
All steps in developing the estimate are documented, so that a cost analyst  unfamiliar with the program can recreate it quickly with the same result.
All data sources for how the data were normalized are documented.
The estimating methodology and rationale used to derive each cost  breakdown structure element’s cost are described in detail.
The estimate’s level of detail ensures that cost elements are neither omitted    All cost-influencing ground rules and assumptions are detailed.
The cost breakdown structure is defined and each element is described in a    The estimate is unbiased, not overly conservative or overly optimistic, and  based on an assessment of most likely costs.
It has few, if any, mathematical mistakes; those it has are minor.
It has been validated for errors like double counting and omitted costs.
It has been compared to the independent cost estimate for differences.
Cross-checks have been made on cost drivers to see if results are similar.
It is updated to reflect changes in technical or program assumptions and new  phases or milestones.
Estimates are replaced with the earned value management estimate at  completion and the independent estimate at completion from the integrated  earned value management system.
Major assumptions are varied and other outcomes recomputed to determine  how sensitive outcomes are to changes in the assumptions.
Risk and uncertainty analysis is performed to determine the level of risk    The results are cross-checked and an independent cost estimate is  developed to determine if other estimating methods produce similar results.
To better understand how the Department of Defense (DOD) develops a  Global War on Terrorism (GWOT) budget request and the use of the  Contingency Operations Support Tool (COST) and other methods of  developing an estimate, we asked DOD to demonstrate how the services  used COST and other methods to develop the $6.3 billion budget estimate  for military operations that was included as part of DOD’s October 2007  $42.3 billion amendment to the fiscal year 2008 GWOT supplemental  request for emergency funding. DOD justification materials for the  amendment describe this $6.3 billion portion as needed to support the  continued sustainment and redeployment of the five Army brigades and  two Marine Corps infantry battalions that were considered part of the  troop surge. This portion of the amendment was also requested to support  the simultaneous deployment of combat support forces augmenting these  combat forces and other costs related to the presence of the troop surge.  Table 3 presents DOD’s estimate for this $6.3 billion portion of the  amendment meant to fund additional operations, broken into the COST- generated portion and the portion that was estimated by other means.  Typically, service budget officials develop cost estimates for GWOT, but  officials stated that time constraints for this specific estimate required that  Joint Staff and DOD Comptroller officials develop the initial estimate using  both COST and other processes outside of COST as necessary. After  discussions with service budget officials, and the September 10, 2007,  testimony of the Commander of the Multi-National Force-Iraq before  Congress, the estimates were revised. This is reflected in the approved  estimate for each service, again broken into the COST portion and the  “non-COST” portion. Each service’s estimate is further broken down into  estimates for Military Personnel appropriations and Operation and  Maintenance appropriations.
As shown in table 3, the Army’s estimate of nearly $5.8 billion comprises  the majority of the $6.3 billion total estimate. COST-generated estimates  for military personnel or operation and maintenance did not change  significantly for the Army from its initial estimate to the approved final  estimate. Substantial changes were made, however, to the portions of the  estimate that are outside the scope of COST. Army budget officials  reported that the large decrease in requested military personnel funding  was due to lower mobilization levels than originally predicted, adjusted  overstrength levels, and reduced permanent change of station and  subsistence costs. The COST portion for both military personnel and  operations for the Army decreased slightly from the initial to the approved  estimate. Army budget officials reported that the approximately $2 billion  increase in operation and maintenance costs that was derived outside of  COST was due to force protection and other equipment or services needed  to support the troop surge and the additional operations performed by  surge troops. Table 4 provides more details on these costs.
Similarly, major adjustments were made to the non-COST portions of the  Air Force, Marine Corps, and Navy estimates for either military personnel  costs, operation and maintenance costs, or both appropriation categories,  depending on the service. These changes were due to changed  assumptions regarding deployment and specialized needs. For example,  the Navy’s non-COST personnel estimate was increased for costs  associated with permanent change of station and active duty for special  work needs. The Marine Corps increased its non-COST personnel request  by $163 million in anticipation of increased requirements for reserve and  Individual Ready Reserve activations to active duty. The adjustments to  the Marine Corps and Air Force COST-generated portions were likewise  substantial. The Marine Corps’ increase for personnel costs reflected in  the COST-attributed portion reflects an anticipated increase in counter- insurgency operations in Afghanistan, while the Air Force estimate was  refined to exclude a KC-10 and the Air National Guard and Air Force  Reserve personnel that would accompany the KC-10. From this case study,  it is clear that changing the assumptions regarding the deployment of  personnel and equipment can have substantial impact on the COST-related  portion of a GWOT budget request. Also, significant changes were made to  the non-COST portion of the request, which in this example increased  from $775 million to about $2.1 billion, including the nearly $1 billion  offset in Army personnel cost estimates. The information regarding the  non-COST portion of this example reveals the significance and size of  estimates generated outside the tool, even in a situation where the  majority, if not all, of the costs would be assumed to be related to  personnel and operations.
We did not validate any of the data DOD presented in the above  discussion, or any of the assumptions or other information used by DOD  to develop this estimate.
section_title: Appendix IV: Comments from the Department of Defense
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: Acknowledgements
paragraphs: In addition to the contact named above, Ann Borseth (Assistant Director),  Grace Coleman, Susan Ditto, Linda Keefer, Lonnie McAllister II, Lisa  McMillen, Charles Perdue, Suzanne Perkins, Karen Richey, and Karen  Werner made key contributions to this report.