Information Technology Cost Estimation: Agencies Need to Address Significant Weaknesses in Policies and Practices
Released: 2012-07-11T00:00:00

Letter

Background

Given the size and significance of the government’s investment in IT, it is important that projects be managed effectively to ensure that public resources are wisely invested. Effectively managing projects entails, among other things, developing reliable and high-quality cost estimates that project realistic life-cycle costs. A life-cycle cost estimate provides an exhaustive and structured accounting of all resources and associated cost elements required to develop, produce, deploy, and sustain a particular program. In essence, life cycle can be thought of as a “cradle to grave” approach to managing a program throughout its useful life. Because a life-cycle cost estimate encompasses all past (or sunk), present, and future costs for every aspect of the program, regardless of funding source, it provides a wealth of information about how much programs are expected to cost over time.

We have previously reported that a reliable cost estimate is critical to the success of any government acquisition program, as it provides the basis for informed investment decision making, realistic budget formulation and program resourcing, meaningful progress measurement, proactive course correction, and accountability for results. Having a realistic, up-to-date estimate of projected costs—one that is continually revised as the program matures—can be used to support key program decisions and milestone reviews. In addition, the estimate is often used to determine the program’s budget spending plan, which outlines how and at what rate the program funding will be spent over time. Because a reasonable and supportable budget is essential to a program’s efficient and timely execution, a reliable estimate is the foundation of a good budget. However, we have also found that developing reliable cost estimates has been difficult for agencies across the federal government.Too often, programs cost more than expected and deliver results that do not satisfy all requirements.

In 2006, the Office of Management and Budget (OMB) updated its Capital Programming Guide, which requires agencies to develop a disciplined cost-estimating capability to provide greater information management support, more accurate and timely cost estimates, and improved risk assessments to help increase the credibility of program cost estimates. Further, according to OMB, programs must maintain current and well- documented estimates of costs, and these estimates must encompass the full life cycle of the program. Among other things, OMB states that generating reliable cost estimates is a critical function necessary to support OMB’s capital programming process. Without this ability, programs are at risk of experiencing cost overruns, missed deadlines, and performance shortfalls.

Building on OMB’s requirements, in March 2009, we issued a guide on best practices for estimating and managing program costs that highlights the policies and practices adopted by leading organizations to implement Specifically, these best practices an effective cost-estimating capability.

OMB, Circular No. A-11, Preparation, Submission, and Execution of the Budget (Washington, D.C.: Executive Office of the President, June 2006) and Capital Programming Guide: Supplement to Circular A-11, Part 7, Planning, Budgeting, and Acquisition of Capital Assets (Washington, D.C.: Executive Office of the President, June 2006). OMB first issued the Capital Programming Guide as a supplement to the 1997 version of Circular A-11, Part 3. We refer to the 2006 version. OMB later updated this guide again in August 2011. identify the need for organizational policies that define a clear requirement for cost estimating; require compliance with cost-estimating best practices; require management review and acceptance of program cost estimates; provide for specialized training; establish a central, independent cost-estimating team; require a standard structure for defining work products; and establish a process to collect and store cost- related data. In addition, the cost-estimating guide identifies four characteristics of a reliable cost estimate that management can use for making informed program and budget decisions: a reliable cost estimate is comprehensive, well-documented, accurate, and credible. Specifically, an estimate is comprehensive when it accounts for all possible costs associated with a program, is structured in sufficient detail to ensure that costs are neither omitted nor double counted, and documents all cost- influencing assumptions; well-documented when supporting documentation explains the process, sources, and methods used to create the estimate, contains the underlying data used to develop the estimate, and is adequately reviewed and approved by management; accurate when it is not overly conservative or optimistic, is based on an assessment of the costs most likely to be incurred, and is regularly updated so that it always reflects the current status of the program; and credible when any limitations of the analysis because of uncertainty or sensitivity surrounding data or assumptions are discussed, the estimate’s results are cross-checked, and an independent cost estimate is conducted by a group outside the acquiring organization to determine whether other estimating methods produce similar results.

We have previously reported on weaknesses associated with the implementation of sound cost-estimating practices at various agencies and the impact on budget and program decisions. For example, In January 2012, we reported that the Internal Revenue Service did not have comprehensive guidance for cost estimating. Specifically, the agency’s guidance did not clearly discuss the appropriate uses of different types of cost estimates. Further, our review of the agency’s Information Reporting and Document Matching program’s cost estimate found it was unreliable. Among other things, the program’s projected budget of $115 million through fiscal year 2016 was only partly supported by the cost estimate, which included costs only through fiscal year 2014. As a result, the agency did not have a reliable basis for the program’s budget projection. We made multiple recommendations to improve the quality of the agency’s cost and budget information, including ensuring that the Information Reporting and Document Matching program’s cost estimate is reliable and that the agency’s cost-estimating guidance is consistent and clearly requires the use of current and reliable cost estimates to inform budget requests. The agency partially agreed with these recommendations and stated that they have taken steps to ensure that their cost-estimating practices and procedures follow consistent documented guidance.

In January 2010, we reported that the Department of Energy lacked comprehensive policy for cost estimating, making it difficult for the agency to oversee development of high-quality cost estimates. Specifically, the agency’s policy did not describe how estimates should be developed and did not establish a central office for cost estimating. Further, we reviewed four programs at the department, each estimated to cost approximately $900 million or more, and reported that they did not have reliable cost estimates. For example, three of the cost estimates did not include costs for the full life cycles of the programs, omitting operations and maintenance costs or portions of program scope. Additionally, three of the cost estimates did not use adequate data, one of which relied instead on professional opinion. Further, the cost estimates did not fully incorporate risk— specifically, they did not address correlated risks among project activities. As a result, these programs were more likely to exceed their estimates and require additional funding to be completed. We made multiple recommendations to improve cost estimating at the department, including updating its cost-estimating policy and guidance and ensuring cost estimates are developed in accordance with best practices. The Department of Energy generally agreed with our recommendations and stated that it had several initiatives underway to improve cost-estimating practices, including the development of a new cost-estimating policy and guidance, a historical cost database to support future estimates, and additional training courses.

Finally, we reported in December 2009 that the Department of Veterans Affairs had 18 construction projects that had experienced cost increases due, in part, to unreliable cost estimates. For example, many estimates were completed quickly, one of which was a rough-order-of-magnitude estimate that was not intended to be relied on as a budget-quality estimate of full project costs. Additionally, we found that some projects had not conducted a risk analysis to quantify the impact of risk on the total estimated costs. As a result, in some cases, projects had to change scope to meet their initial estimate and, in others, additional funds had to be requested from Congress to allow the agency to complete the project. We recommended that the department improve cost estimating at major construction projects by conducting cost risk analyses and mitigating risks that may influence projects’ costs. The Department of Veterans Affairs agreed with our recommendation and stated that it was taking steps, such as developing a multiyear construction plan to ensure that reliable projections of program costs are available for budgeting purposes, and planning to improve its risk analyses.

Selected Agencies’ Cost-Estimating Policies and Procedures Have Significant Weaknesses

According to OMB, agencies should develop a disciplined cost- estimating capability to provide greater information management support, more accurate and timely cost estimates, and improved risk assessments to help increase the credibility of program cost estimates. In addition, we have reportedpolicies and procedures that that leading organizations establish cost-estimating define a clear requirement for cost estimating; identify and require compliance with cost-estimating best practices, and validate their use; require that estimates be reviewed and approved by management; require and enforce training in cost estimating; establish a central, independent cost-estimating team; require, at a high level, a standard, product-oriented work breakdown structure; and establish a process for collecting and storing cost-related data to support future estimates.

Table 1 describes the key components of an effective cost-estimating policy.

While the eight agencies varied in the extent to which their cost- estimating policies and procedures addressed best practices, most did not address several key components of an effective policy. Specifically, only the Department of Defense’s (DOD) policy was fully consistent with all seven components. While the Department of Homeland Security addressed most components of an effective cost-estimating policy, other agencies’ policies had significant weaknesses, particularly in cost- estimating training and in establishing a process to collect and store cost- related data.

Table 2 provides a detailed assessment of each agency’s policies against the components of an effective cost-estimating policy. In addition, a discussion of each policy component follows the table.

Clear requirement for cost estimating: Six of the eight agencies fully addressed this policy component by establishing a clear requirement for all programs to perform life-cycle cost estimates, and in certain cases specified more stringent requirements for programs designated as major investments. Among these, four agencies—the Department of Agriculture, the Environmental Protection Agency (EPA), the Department of Labor, and the Department of Justice—established this requirement as part of their policies for programs to perform a cost- benefit analysis. For example, Labor required a life-cycle cost estimate as part of a cost-benefit analysis for both major and nonmajor investments, with less detail required for nonmajor investments. The other two agencies—DOD and Homeland Security—defined a separate requirement for programs to develop life-cycle cost estimates. For the two agencies that did not fully establish a clear requirement for cost estimating, the Department of Veterans Affairs partially addressed this component because its policy only requires cost estimates to be prepared for project increments, rather than the full program life cycle. In addition, the Department of Commerce partially addressed this component because its policies only require cost estimates to be prepared for contracts, rather than for the full program life cycle (including government and contractor costs). Officials at both agencies stated that the responsibility for establishing requirements for cost estimating had been delegated to their component agencies. Further, officials at these two agencies described steps planned to address this and other weaknesses. For example, Veterans Affairs officials stated that the agency’s recently established Office of Corporate Analysis and Evaluation (part of the Office of Planning and Policy) is planning to establish a centralized cost-estimating policy that includes clear criteria for cost estimating, which it expects to complete in fiscal year 2012. Further, Commerce officials stated that the agency is currently in the process of updating its policy and guidance to address this and other weaknesses, which it plans to complete by October 2012. If the updated policies and guidance address the weaknesses we identified, decision makers should have an improved view of their programs’ life-cycle costs.

Compliance with cost-estimating best practices: Three of the eight agencies (DOD, Homeland Security, and Labor) fully addressed this policy component by identifying and requiring the use of cost- estimating best practices by their programs, and defining a process to validate their use. For example, Homeland Security draws on the GAO cost guide to identify cost-estimating best practices, and also provides agency-specific cost-estimating requirements for implementing the practices, such as identifying the cost-estimate documentation required. The agency’s policy also requires that estimates for key programs be validated. For the three agencies that partially addressed this policy component—Agriculture, EPA, and Justice—all provided guidance to their programs specific to conducting a cost-benefit analysis; however, this guidance did not fully address important cost-estimating practices, such as conducting a risk and uncertainty analysis, updating the estimate, or comparing the estimate to an independent estimate. Their guidance also did not identify a mechanism for validating estimates. Lastly, two agencies— Commerce and Veterans Affairs—had not addressed this policy component, which corresponds to our finding that these agencies did not have requirements for programs to prepare cost estimates. Among the five agencies that did not fully address this policy component, officials commonly stated that the responsibility for requiring compliance with best practices had been delegated to their component agencies or that addressing cost-estimating shortcomings had not been a priority. Without fully complying with best practices for developing cost estimates, programs are less likely to prepare reliable cost estimates, hindering agency decision making.

Management review and approval: Three of the eight agencies (DOD, Homeland Security, and Labor) fully addressed this policy component by requiring that program cost estimates be reviewed and approved by management, including defining the information to be presented and requiring that approval be documented. For example, Labor’s policy requires that senior management at both the component agency responsible for the program and the Office of the Chief Information Officer approve the estimate, based on a briefing that includes information about the estimate such as the largest cost drivers, major risks, and the findings of the integrated baseline review, and that this approval is documented. For the three agencies that partially addressed this policy component (Agriculture, EPA, and Veterans Affairs), all required that estimated costs be presented to management, but none fully defined the information to be presented, such as the confidence level associated with the estimate. Lastly, neither Justice nor Commerce had departmental requirements for management review and approval of the cost estimate. Officials at both agencies stated that this responsibility had been delegated to their component agencies. However, without requiring management review and approval of program cost estimates at the department level, agencies have reduced ability to enforce cost- estimating policies and ensure that cost estimates meet management’s needs for reliable information about programs’ estimated costs.

Training requirements: Only one agency—DOD—fully addressed this policy component by requiring cost-estimating training and enforcing this requirement. For example, DOD requires training in cost estimating via its Defense Acquisition Workforce Improvement Act certifications, among other things, for at least one staff member for each major program, as well as for personnel with investment oversight responsibility. While the two agencies that partially addressed this policy component (Homeland Security and Labor) provided cost-estimating training and had a mechanism to track participation, their policies did not address providing training to personnel with investment oversight responsibility, such as officials from Homeland Security who are responsible for reviewing and approving programs at key milestones in their life cycles. Among the five agencies whose policies did not address requiring and enforcing training in cost estimating (Agriculture, Commerce, EPA, Justice, and Veterans Affairs), four of these agencies referred to OMB’s Federal Acquisition Certification for Program and Project Managers as providing for such training. However, this certification program does not require classes on cost estimating, and furthermore, is not intended for nor provided to individuals with investment oversight responsibility. Additionally, officials at two of the five agencies— Commerce and Veterans Affairs—stated that training in cost estimating had not been viewed as a priority. Without requiring and enforcing training in cost estimating, agencies cannot effectively ensure that staff have the skills and knowledge necessary to prepare and use cost estimates to make reliable budget and program decisions.

Central, independent cost-estimating team: Three of the eight agencies (DOD, Homeland Security, and Veterans Affairs) fully addressed this policy component by establishing central, independent cost-estimating teams, all of which have responsibility for, among other things, developing cost-estimating guidance and validating that program cost estimates are developed in accordance with best practices. In addition, among these three agencies, the teams established at DOD and Veterans Affairs are also charged with improving cost-estimating training. The remaining five agencies had not established a central, independent cost-estimating team. Among these, officials commonly cited the lack of a priority at the department or agency level for cost-estimating initiatives, although in one case a component agency at Agriculture—the Food Safety and Inspection Service—established its own centralized cost-estimating team. While this will likely enhance cost estimating at the component agency, not centralizing the cost-estimating function in the department could result in ad hoc processes and a lack of commonality in the estimating tools and training across the department. Additionally, officials from Labor stated they believe the department’s IT budget is too small to cost- effectively centralize the cost-estimating function; however, doing so would likely, among other things, facilitate a better sharing of resources and could be accomplished in a manner commensurate with agency size. Agencies that do not establish a central and independent cost-estimating team may lack the ability to improve the implementation of cost-estimating policies, support cost-estimating training, and validate the reliability of program cost estimates at the department or agency level.

Standard structure for defining work products: DOD was the only agency to fully address this policy component by developing and requiring the use of standard, product-oriented work breakdown structures. Specifically, the agency provided multiple standard work breakdown structures, along with detailed guidance, for different types of programs (e.g., automated information systems, space systems, aircraft systems), and required their use. Three agencies—Homeland Security, Justice, and Veterans Affairs—partially addressed this policy component in that they provided one or more product-oriented work breakdown structures in their policies, but did not require programs to use them for cost estimating. Among these, Justice officials stated that a standard work breakdown structure was only required for their earned value management processes. Further, both Veterans Affairs and Homeland Security stated that they intend to require the use of a standard work breakdown structure in the future, but had not yet determined a time frame for establishing this requirement. Lastly, four of the selected agencies—Agriculture, Commerce, EPA, and Labor—had not established a standard structure. Among these, officials from Agriculture, EPA, and Labor stated that they believe it is difficult to standardize how programs define work products, in part, because their programs conduct different types of work and have different needs. While this presents a challenge, agencies could adopt an approach similar to DOD’s and develop various standard work structures based on the kinds of work being performed. Commerce officials stated that they plan to establish a standard structure for defining work products in the future, but have not yet determined a time frame for completing this. Without establishing a standard structure for defining work products, agencies will not be positioned to ensure that they can effectively compare programs and collect and share data among programs.

Process to collect and store cost-related data: Only one agency— DOD—fully addressed this policy component by establishing a process to collect and store cost-related data. Specifically, the agency has a central repository for collecting actual costs, software data, and related business data, which serves as a resource to support cost estimating across the agency. Among the seven agencies that have not established a process for collecting and storing cost-related data, Homeland Security’s policy assigns responsibility for doing so to the central cost-estimating team; however, the team has not yet implemented the process. Additionally, Veterans Affairs officials stated that collecting such data would depend on the use of a standard structure for defining work products, which they have not yet put in place. Agriculture and Commerce officials stated that cost- estimating initiatives have not been a priority, although in one case a component agency at Commerce—the United States Patent and Trademark Office—took the initiative to establish a process to collect and store cost-related data from past estimates. While this should improve cost estimating at the component agency, without establishing an agencywide process to collect and store cost-related data, agencies will find it difficult to improve the data available to all programs and to increase the efficiency of developing cost estimates.

Until the selected agencies address the identified weaknesses in their cost-estimating policies, it will be difficult for them to make effective use of program cost estimates for informed decision making, realistic budget formation, and meaningful progress measurement.

Selected Agencies’ Program Cost Estimates Do Not Provide a Fully Reliable Basis for Program and Budget Decisions

A reliable cost estimate is critical to the success of any government acquisition program, as it provides the basis for informed investment decision making, realistic budget formulation and program resourcing, and meaningful progress measurement. According to OMB, programs must maintain current and well-documented cost estimates, and these estimates must encompass the full life cycle of the programs. In addition, our research has identified a number of best practices that provide a basis for effective program cost estimating and should result in reliable cost estimates that management can use for making informed decisions. These practices can be organized into four characteristics— comprehensive, well-documented, accurate, and credible. These four characteristics of a reliable cost estimate are explained in table 3.

While all 16 major acquisition programs we reviewed had developed cost estimates and were using them to inform decision making, all but one of the estimates were not fully reliable and did not provide a sound basis for informed program and budget decisions. The 16 acquisition programs had developed cost estimates and were using their estimates, in part, to support program and budget decisions. For example, most programs used their cost estimate as the basis for key program decisions, such as approval to proceed to full production of a system. In addition, most programs were using their estimates as an input to their annual budget request process.

However, nearly all of these programs had estimates that did not fully reflect important cost-estimating practices. Specifically, of the 16 case study programs, only 1 fully met all four characteristics of a reliable cost estimate, while the remaining 15 programs varied in the extent to which they met the four characteristics. Table 4 identifies the 16 case study programs and summarizes our results for these programs. Following the table is a summary of the programs’ implementation of cost-estimating practices. Additional details on the 16 case studies are provided in appendix II.

Most programs partially implemented key practices needed to develop a comprehensive cost estimate. Specifically, of the 16 programs, 1 fully implemented the practices for establishing a comprehensive cost estimate, 12 partially implemented the practices, and 3 did not implement them.

DOD’s Consolidated Afloat Networks and Enterprise Services program fully implemented key practices for developing a comprehensive cost estimate. Specifically, the program’s cost estimate included both the government and contractor costs for the program over its full life cycle, from inception through design, development, deployment, operation and maintenance, and retirement of the program. Further, the cost estimate reflected the current program and technical parameters, such as the acquisition strategy and physical characteristics of the system. In addition, the estimate clearly described how the various cost subelements were summed to produce the amounts for each cost category, thereby ensuring that all pertinent costs were included, and no costs were double counted. Lastly, cost-influencing ground rules and assumptions, such as the program’s schedule, labor rates, and inflation indexes, were documented.

Twelve programs partially implemented key practices for developing a comprehensive cost estimate. Most of these programs fully identified cost-influencing ground rules and assumptions and included government and contractor costs for portions of the program life cycle. However, 10 of the 12 programs did not include the full costs for all life-cycle phases and other important aspects of the program, such as costs expected to be incurred by organizations outside of the acquiring program (e.g., by other agency subcomponents), all costs for operating and maintaining the system, and costs for the retirement of the system. Without fully accounting for all past, present, and future costs for every aspect of the program, regardless of funding source, the programs’ estimated costs are likely understated and thereby subject to underfunding and cost overruns.

In addition, 10 of the 12 programs did not provide evidence that their cost estimates completely defined the program or reflected the current program schedule by documenting a technical baseline description to provide a common definition of the current program, including detailed technical, program, and schedule descriptions of the system. For example, in 2008, Homeland Security’s Rescue 21 program documented the system’s technical characteristics, along with a high- level schedule for the program. Since 2008, however, certain technical characteristics of the program had changed, such as additional deployment sites needed to address communication service gaps identified by local commanders at previously deployed locations. In addition, the planned deployment dates for several locations of the system had been delayed. As a result, the program’s cost estimate did not fully reflect the current scope and schedule of the program.

Understanding the program—including the acquisition strategy, technical definition, characteristics, system design features, and technologies to be included—is critical to developing a reliable cost estimate. Without these data, programs will not be able to identify the technical and program parameters that bind the estimate.

Three programs did not implement key practices for developing a comprehensive cost estimate in that their estimates did not adequately (1) include all costs over the program’s full life cycle; (2) completely define the program or the current schedule; (3) include a detailed, product-oriented work breakdown structure; and (4) document cost-influencing ground rules and assumptions. For example, the cost estimate for Veterans Affairs’ Health Data Repository program did not include sufficient detail to show that it accounted for all phases of the program’s life cycle (e.g., design, development, and deployment). Further, the estimate did not include important technical baseline information, including the technical, program, and schedule aspects of the system being estimated. Lastly, the estimate only used high-level budget codes rather than a detailed, product-oriented cost element structure to decompose the work, and ground rules and assumptions (e.g., labor rates and base-year dollars) were not documented. Without implementing key practices for developing comprehensive cost estimates, management and oversight organizations cannot be assured that a program’s estimate is complete and accounts for all possible costs, thus increasing the likelihood that the estimate is understated.

The majority of programs partially implemented key practices needed to develop a well-documented cost estimate. Specifically, of the 16 programs, 1 fully implemented the practices for establishing a well- documented cost estimate, 10 partially implemented the practices, and 5 did not implement them.

DOD’s Consolidated Afloat Networks and Enterprise Services program fully implemented key practices for developing a well- documented cost estimate. Specifically, the program’s cost estimate captured in writing the source data used (e.g., historical data and program documentation), the calculations performed and their results, and the estimating methodology used to derive each cost element. In addition, the program documented a technical baseline description that included, among other things, the relationships with other systems and planned performance parameters. Lastly, the cost estimate was reviewed both by the Naval Center for Cost Analysis and the Assistant Secretary of the Navy for Research, Development, and Acquisition, which helped ensure a level of confidence in the estimating process and the estimate produced.

Ten programs partially implemented key practices for developing a well-documented cost estimate. Most of these programs included a limited description of source data and methodologies used for estimating costs, and documented management approval of the cost estimate. However, 9 of the 10 programs did not include complete documentation capturing source data used, the calculations performed and their results, and the estimating methodology used to derive each cost element. Among other things, the 9 programs had weaknesses in one or more of the following areas: relying on expert opinion but lacking historical data or other documentation to back up the opinions; not documenting their estimate in a way that a cost analyst unfamiliar with the program could understand what was done and replicate it; and lacking supporting data that could be easily updated to reflect actual costs or program changes. Without adequate documentation to support the cost estimate, questions about the approach or data used cannot be answered and the estimate may not be useful for updates or information sharing.

In addition, 8 of the 10 programs did not provide management with sufficient information about how the estimate was developed in order to make an informed approval decision. For example, while the EPA’s Financial System Modernization Project’s cost estimate was approved, management was not provided information specific to how the estimate was developed, including enough detail to show whether it was accurate, complete, and high in quality. Because cost estimates should be reviewed and accepted by management on the basis of confidence in the estimating process and the estimate produced by the process, it is imperative that management understand how the estimate was developed, including the risks associated with the underlying data and methods, in making a decision to approve a cost estimate.

Five programs did not implement key practices for developing a well- documented cost estimate in that their estimates did not adequately (1) include detailed documentation that described how the estimate was derived, (2) capture the estimating process in such a way that the estimate can be easily replicated and updated, (3) discuss the technical baseline description, and (4) provide evidence that the estimate was fully reviewed and accepted by management. In particular, three of the five programs relied on their budget submission documentation, known as the OMB Exhibit 300, as their life-cycle cost estimate. The cost estimate information included in these programs’ Exhibit 300 budget submissions was limited to the final estimates in certain phases of the program’s life cycle, such as planning, development, and operations and maintenance. Because a well-documented estimate includes detailed documentation of the source data, calculations and results, and explanations of why particular methods and references were chosen, the programs that relied on their Exhibit 300 budget submissions as their cost estimates lacked the level of rigor and supporting documentation necessary for a well-documented cost estimate. Without a well-documented estimate, a program’s credibility may suffer because the documentation cannot explain the rationale of the methodology or the calculations, a convincing argument of the estimate’s validity cannot be presented, and decision makers’ questions cannot be effectively answered.

Most programs partially implemented or did not implement key practices needed to develop an accurate cost estimate. Specifically, of the 16 programs, 2 fully implemented the practices for establishing an accurate cost estimate, 8 partially implemented the practices, and 6 did not implement them.

DOD’s Consolidated Afloat Networks and Enterprise Services and Homeland Security’s Integrated Public Alert and Warning System programs fully implemented key practices for developing an accurate cost estimate. Specifically, the programs’ estimates were based on an assessment of most likely costs, in part because a risk and uncertainty analysis was conducted to determine where the programs’ estimates fell against the range of all possible costs. In addition, the programs’ estimates were grounded in a historical record of cost estimating and actual experiences from comparable programs. For example, the cost estimate for the Integrated Public Alert and Warning System program relied, in part, on actual costs already incurred by the program as well as data from three comparable programs, including a legacy disaster management system. Moreover, the programs’ cost estimates were adjusted for inflation and updated regularly to reflect material changes in the programs, such as when the schedule changed.

Eight programs partially implemented key practices for developing an accurate cost estimate. Most of these programs accounted for inflation when projecting future costs. However, four of the eight programs did not rely, or could not provide evidence of relying, on historical costs and actual experiences from comparable programs. For example, officials from Pension Benefit Guaranty Corporation’s Benefit Administration program stated that they relied on historical data along with expert opinion in projecting costs, but the officials did not provide evidence of the data sources or how the historical data were used. Because historical data can provide estimators with insight into actual costs on similar programs—including any cost growth that occurred in the original estimates—without documenting these data, these programs lacked an effective means to challenge optimistic assumptions and bring more realism to their estimates.

In addition, six of the eight programs did not provide evidence that they had regularly updated their estimates to reflect material changes in the programs so that they accurately reflected the current status. For example, Justice’s Unified Financial Management System program developed a cost estimate in 2009; however, according to program documentation, program scope and projected costs have since changed and, as a result, the 2009 estimate no longer reflects the current program. Cost estimates that are not regularly updated with current information can make it more difficult to analyze changes in program costs, impede the collection of cost and technical data to support future estimates, and may not provide decision makers with accurate information for assessing alternative decisions.

Six programs did not implement key practices for developing an accurate cost estimate in that their estimates were not adequately (1) based on an assessment of most likely costs, (2) grounded in historical data and actual experiences from comparable programs, (3) adjusted for inflation, and (4) updated to ensure that they always reflect the current status of the program. For example, the cost estimate for Agriculture’s Public Health Information System was not based on an assessment of most likely costs because a risk and uncertainty analysis was not conducted to determine where the estimate fell against the range of all possible costs. In addition, the estimate was based primarily on the program team’s expertise, but was not grounded in historical costs or actual experiences from comparable programs. Lastly, the estimate was not adjusted for inflation and lacked adequate detail to determine whether the program’s latest updates to the cost estimate, completed in 2011, accurately reflected the current status of the program. Without implementing key practices for developing an accurate cost estimate, a program’s estimate is more likely to be biased by optimism and subject to cost overruns, and may not provide management and oversight organizations with accurate information for making well- informed decisions.

The majority of programs did not implement all key practices needed to develop a credible cost estimate. Specifically, of the 16 programs, 1 fully implemented the practices for establishing a credible cost estimate, 5 partially implemented the practices, and 10 did not implement them.

DOD’s Consolidated Afloat Networks and Enterprise Services program fully implemented key practices for developing a credible cost estimate. Specifically, the program performed a complete uncertainty analysis (i.e., both a sensitivity analysis and Monte Carlo simulation) on the estimate. For example, in performing the sensitivity analysis, the program identified a range of possible costs based on varying key parameters, such as the technology refresh cycle and procurement costs. In addition, the program performed cross checks (using different estimating methods) on key cost drivers, such as system installation costs. Lastly, an independent cost estimate was conducted by the Naval Center for Cost Analysis and the results were reconciled with the program’s cost estimate, which increased the confidence in the credibility of the resulting estimate.

Five programs partially implemented key practices for developing a credible cost estimate. Specifically, three of the five programs performed aspects of a sensitivity analysis, such as varying one or two assumptions to assess the impact on the estimate; however, these programs did not perform other important components, such as documenting the rationale for the changes to the assumptions or assessing the full impact of the changes to the assumptions by determining a range of possible costs. For example, the Pension Benefit Guaranty Corporation’s Benefits Administration program performed a sensitivity analysis by varying three program assumptions, one of which was the contractor’s hourly rate, to assess the impact on the cost estimate. However, the program did not provide evidence to support why the adjusted hourly labor rate was used nor apply a range of increases and decreases to the hourly labor rate to determine the level of sensitivity of this assumption on the cost estimate. A comprehensive sensitivity analysis that is well documented and traceable can provide programs with a better understanding of the variables that most affect the cost estimate and assist in identifying the cost elements that represent the highest risk.

In addition, three of the five programs adjusted the cost estimate to account for risk and uncertainty, but did not provide evidence to support how costs were risk adjusted or determine the level of confidence associated with the cost estimate.Homeland Security’s Integrated Public Alert and Warning System program’s cost estimate did not include information on the risks considered in its risk and uncertainty analysis or consider the relationship between multiple cost elements when accounting for risks. Without conducting an adequate risk and uncertainty analysis, the cost estimate may be unrealistic because it does not fully reflect the aggregate variability from such effects as schedule slippage, mission changes, and proposed solutions not meeting users’ needs.

Ten programs did not implement key practices for developing a credible cost estimate in that the programs did not adequately (1) assess the uncertainty or bias surrounding data and assumptions by conducting a sensitivity analysis, (2) determine the level of risk associated with the estimate by performing a risk and uncertainty analysis, (3) cross-check the estimates for key cost drivers, and (4) commission an independent cost estimate to be conducted by a group outside the acquiring organization to determine whether other estimating methods would produce similar results. For example, Agriculture’s Web-Based Supply Chain Management program did not conduct a sensitivity analysis to better understand which variables most affected the cost estimate, nor did the program conduct a risk and uncertainty analysis to quantify the impact of risks on the estimate. Further, cost drivers were not cross-checked to see if different estimating methodologies produced similar results, and an independent cost estimate was not conducted to independently validate the results of the program’s estimate. Without implementing key practices for developing a credible cost estimate, a program may lack an understanding of the limitations associated with the cost estimate and be unprepared to deal with unexpected contingencies.

Inadequate Implementation Was Largely Due to Weaknesses in Policy

The lack of reliable cost estimates across the investments exists in part because of the weaknesses previously identified in the eight agencies’ cost-estimating policies. More specifically, program officials at five agencies—Agriculture, Commerce, EPA, Justice, and Veterans Affairs— attributed weaknesses in their programs’ cost estimates, in part, to the fact that agency policies did not require cost-estimating best practices— deficiencies which we also identified in these agencies’ policies. For example, officials at Commerce’s Comprehensive Large Array-data Stewardship System program stated that, when the program developed its cost estimate, no agency guidance existed regarding the process to follow in developing the estimate. In addition, officials at Veterans Affairs’ Veteran’s Benefits Management System program stated that they did not perform a risk analysis on their cost estimate because agency guidance on how such an analysis should be performed did not exist. In certain cases, officials stated that program cost estimates were initially developed prior to 2007, when a comprehensive federal resource for cost-estimating best practices, such as GAO’s cost guide, did not exist. However, all 16 programs included in our review have either developed new estimates or updated previous estimates since 2007; nonetheless, as previously mentioned, most of the selected agencies’ policies did not fully address compliance with cost-estimating best practices, including the five agencies mentioned above. If these agencies had updated their policies, programs would have been more likely to follow a standard, high-quality process in developing or updating their cost estimates.

Until important cost-estimating practices are fully implemented, the likelihood that these programs will have to revise their current cost estimates upward is increased. Collectively, 13 of the 16 programs have already revised their original life-cycle cost estimates upward by almost $5 billion due, in part, to weaknesses in program cost-estimating practices (see app. III for details on changes in the programs’ cost estimates over time). For example, in many cases, cost estimates had to be revised upwards to reflect the incorporation of full costs for all life-cycle phases (e.g., development or operations and maintenance), which had not originally been included. This resulted, in some cases, in significant increases to estimated life-cycle costs. Other reasons that programs cited for revising their life-cycle cost estimates upward included changes to program or system requirements, schedule delays, technology upgrades, and system defects, among other things. Further, as previously mentioned, 13 of the 16 case study programs still have cost estimates that do not include the full costs for all life-cycle phases, which significantly increases the risk that these programs’ cost estimates will continue to be revised upward in the future.

Without reliable cost estimates, the 15 programs that did not fully meet best practices will not have a sound basis for informed program decision making, realistic budget formulation and program resourcing, and meaningful progress measurement. Consequently, nearly all of these programs’ cost estimates may continue to be understated and subject to underfunding and cost overruns.

Conclusions

Given the enormous size of the federal government’s investment in IT, it is critical that such investments are based on reliable estimates of program costs. While all of the selected agencies have established policies that at least partially addressed a requirement for programs to develop full life-cycle cost estimates, most of the agencies’ policies have significant weaknesses. With the exception of DOD, these policies omit or lack sufficient guidance on several key components of a comprehensive policy including, for example, management review and acceptance of program cost estimates, the type of work structure needed to effectively estimate costs, and training requirements for all relevant personnel. Without comprehensive policies, agencies may not have a sound basis for making decisions on how to most effectively manage their portfolios of projects.

Most programs’ estimates at least partially reflected cost-estimating best practices, such as documenting cost-influencing ground rules and assumptions; however, with the exception of DOD’s Consolidated Afloat Networks and Enterprise Services program, the programs we reviewed had not established fully reliable cost estimates, increasing the likelihood that the estimates are incomplete and do not account for all possible costs. For example, without including costs for all phases of a program’s life cycle and performing a comprehensive risk and uncertainty analysis, a program’s estimated costs could be understated and subject to underfunding and cost overruns, putting it at risk of being reduced in scope or requiring additional funding to meet its objectives. Many of the weaknesses found in these programs can be traced back to inadequate agency cost-estimating policies. Without better estimates of acquisition life-cycle costs, neither the programs nor the agencies have reliable information for supporting program and budget decisions. Consequently, the likelihood of cost overruns, missed deadlines, and performance shortfalls is significantly increased.

Recommendations for Executive Action

To address weaknesses identified in agencies’ policies and practices for cost estimating, we are making the following recommendations: We recommend that the Secretaries of Agriculture, Commerce, Homeland Security, Labor, and Veterans Affairs, the Attorney General, and the Administrator of the Environmental Protection Agency direct responsible officials to modify policies governing cost estimating to ensure that they address the weaknesses that we identified.

We also recommend that the Secretaries of Agriculture, Commerce, Homeland Security, Labor, and Veterans Affairs, the Attorney General, the Administrator of the Environmental Protection Agency, and the Director of the Pension Benefit Guaranty Corporation direct responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified.

Lastly, although DOD fully addressed the components of an effective cost-estimating policy, in order to address the weaknesses we identified with a key system acquisition discussed in this report, we recommend that the Secretary of Defense direct responsible officials to update future life-cycle cost estimates of the Tactical Mission Command program using cost-estimating practices that address the detailed weaknesses that we identified.

Agency Comments and Our Evaluation

We provided the selected eight agencies and the Pension Benefit Guaranty Corporation with a draft of our report for review and comment. A management analyst in the Department of Justice’s Internal Review and Evaluation Office, Justice Management Division, responded orally that the department had no comments. Six of the agencies and the Pension Benefit Guaranty Corporation provided written comments, and the Department of Labor provided oral and written comments. These agencies generally agreed with our results and recommendations, although EPA disagreed with our assessment of the cost-estimating practices used for one of its programs. These agencies also provided technical comments, which we incorporated in the report as appropriate.

The comments of the agencies and the corporation are summarized below:

The U.S. Department of Agriculture’s Acting Chief Information Officer stated that the department concurred with the content of the report. Agriculture’s comments are reprinted in appendix IV.

The Acting Secretary of Commerce stated that the department fully concurred with our findings and recommendations. Among other things, the Acting Secretary described a number of ongoing actions to address the weaknesses we identified, such as modifying departmental policies governing cost estimating to include an additional cost-estimating training course and cost-estimating training requirements. In addition, the department stated that forthcoming policy and guidance are intended to ensure that the cost estimates for high-profile programs are comprehensive, accurate, credible, and well-documented. Commerce’s comments are reprinted in appendix V.

DOD’s Director of Cost Assessment and Program Evaluation stated that the department partially concurred with our recommendation but agreed with the criteria, methodology, and assessment of the DOD programs. The director added, however, that there is no plan to formally update the Tactical Mission Command life-cycle cost estimate, as the program is in the system deployment phase of its acquisition lifecycle. We recognize that the programs included in our study are at varying stages of their acquisition life cycles and that updates to their cost estimates may not be justified. Accordingly, our recommendation to DOD is specific to only future life-cycle cost estimates. In this regard, if any significant changes occur in the program during deployment of the system that warrant an update to the cost estimate, it will be important that the program uses best practices that address the weaknesses we identified. DOD’s comments are reprinted in appendix VI.

EPA’s Assistant Administrator of the Office of Solid Waste and Emergency Response and its Assistant Administrator and Chief Information Officer of the Office of Environmental Information stated, in regard to our assessment of cost-estimating policies, that EPA recognized that its policies did not require cost-estimating best practices and that the agency will update its Systems Life Cycle Management procedures accordingly. The officials acknowledged that sound fiscal management practices should be followed in all aspects of the agency’s information technology operations, including cost estimating for the development of new systems.

In regard to our assessment of cost-estimating practices for two system acquisition programs, EPA stated that it did not have any comments on our assessment of the Financial System Modernization Project; however, it did not believe our assessment accurately reflected the cost-estimating practices employed for the development of the Superfund Enterprise Management System. In particular, the Office of Solid Waste and Emergency Response stated in its written response and in technical comments that it believed it had met the spirit and intent of the cost-estimating best practices in GAO’s cost guide, even though the program may have used different processes or documentation in order to do so. We recognize and agree that organizations should tailor the use of the cost-estimating best practices as appropriate based on, for example, the development approach being used, and we took this factor into consideration during our review of the 16 acquisition programs. However, we stand by our assessment of the Superfund Enterprise Management System program’s cost estimate on the basis of the weaknesses described in appendix II of this report. In particular, as we discuss, the program’s cost estimate lacked key supporting documentation, including costs not documented at a sufficient level of detail; the lack of documented source data, calculations, and methodologies used to develop the estimate; and a lack of documentation on the source of and rationale for the inflation factor used. In addition, the lack of detailed cost- estimate information precluded us from making the linkage between the cost estimate and other important program documents, such as the system’s technical baseline and schedule, in order to determine whether the estimate reflects the current program and status. Because rigorous documentation is essential for justifying how an estimate was developed and for presenting a convincing argument for an estimate’s validity, weaknesses in this area contributed significantly to weaknesses across multiple best practices areas, including the estimate’s comprehensiveness and accuracy. Further, regarding the Office of Solid Waste and Emergency Response’s comment that our cost-estimating guide was not published until 3 years after development of the Superfund Enterprise Management System commenced, we disagree that this would preclude the program from satisfying cost-estimating best practices. Specifically, the program updated its cost estimate in 2011, 2 years after the issuance of the GAO cost guide. At that time, the program could have revised its cost estimate using available best practice guidance.

Lastly, we disagree that the draft report erroneously concluded that the Superfund Enterprise Management System cost estimate increased from $39.3 million to $62.0 million in just 2 years. In its written response, the Office of Solid Waste and Emergency Response stated that the revised cost estimate was a direct result of an increase in the duration of operations and maintenance from fiscal year 2013 (in the $39.3 million estimate) to fiscal year 2017 (in the $62.0 million estimate). However, according to documentation provided by the Superfund Enterprise Management System program, the $39.3 million estimate, which was completed in 2009, was based on a 10-year life cycle (from fiscal year 2007 to fiscal year 2017) and included costs for operations and maintenance through fiscal year 2017. Subsequently, in 2011, the program revised its estimate to approximately $62.0 million, which was also based on a 10-year life cycle (from fiscal year 2007 to fiscal year 2017) and included operations and maintenance costs through 2017. The revised estimate is an increase of about $22.7 million over the initial estimate. According to program documentation, this change in the cost estimate was primarily due to the inclusion of additional operations and maintenance costs for data and content storage and hosting for the fully integrated system between fiscal year 2014 and fiscal year 2017, which were erroneously omitted from the 2009 estimate. Based on these factors, we maintain that our report reflects this information appropriately. EPA’s comments are reprinted in appendix VII.

The Department of Homeland Security’s Director of the Departmental GAO-Office of the Inspector General Liaison Office stated that the department concurred with our recommendations. Among other things, the department stated that its Office of Program Accountability and Risk Management intends to develop a revised cost-estimating policy that will further incorporate cost-estimating best practices, as well as work to provide cost-estimating training to personnel on major programs throughout the department. Homeland Security’s comments are reprinted in appendix VIII.

In oral comments, the Administrative Officer in the Department of Labor’s Office of the Assistant Secretary for Administration and Management stated that the department generally agreed with our recommendations. Further, in written comments, the Assistant Secretary for Administration and Management stated that the department, through several initiatives, such as its Post Implementation Review process and training to IT managers, will continue to improve upon its IT cost estimation. The department also commented on certain findings in our draft report. In particular, the Assistant Secretary stated that, given the department’s relatively small IT portfolio, establishing a central, independent office dedicated to cost estimating is not justified. We recognize that agency IT portfolios vary in size; however, as noted in our report, agencies should establish a central cost-estimating team commensurate with the size of their agency, which could consist of a few resident experts instead of a full independent office. Regarding our second recommendation, according to the Assistant Secretary, the Occupational Safety and Health Administration (OSHA) stated that it believes our assessment of the credibility of the OSHA Information System program’s 2010 cost estimate was too low and did not reflect additional information provided in support of the program’s 2008 cost estimate. In our assessment of the program’s 2010 estimate we acknowledge evidence provided from the 2008 estimate; however, this evidence did not adequately show that important practices for ensuring an estimate’s credibility, including making adjustments to account for risk and conducting a sensitivity analysis, were performed on the 2010 cost estimate. In addition, OSHA stated that an independent estimate was conducted at the outset of the program by an industry-leading IT consulting firm as recommended by the Department of Labor Office of the Inspector General. While we acknowledge that this was done in 2005, the resulting estimate was the only one developed at the time and thus was not used as a means of independent validation—i.e., to determine whether multiple estimating methods produced similar results. Therefore, the independent estimate conducted in 2005 would not increase the credibility of the program’s current cost estimate. Labor’s comments are reprinted in appendix IX.

The Director of the Pension Benefit Guaranty Corporation stated that the corporation was pleased that its selected IT investment met at least half, or a large portion, of our quality indicators for cost estimating. Further, the Director stated that the corporation will evaluate and improve future life-cycle cost estimates for the Benefit Administration investment. The Pension Benefit Guaranty Corporation’s comments are reprinted in appendix X.

The Chief of Staff for the Department of Veterans Affairs stated that the department concurred with our recommendations and has efforts under way to improve its cost-estimating capabilities. Among other things, the Chief of Staff stated that the department plans to complete, by the end of the first quarter of fiscal year 2013, an evaluation of the utility of establishing an organizational function focused solely on multiyear cost estimation. In addition, to improve cost-estimating practices on its IT efforts, the department stated that it has additional training planned in early fiscal year 2013. Veterans Affairs’ comments are reprinted in appendix XI.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretaries of Agriculture, Commerce, Defense, Homeland Security, Labor, and Veterans Affairs; the Attorney General; the Administrator of the Environmental Protection Agency; the Director of the Pension Benefit Guaranty Corporation; and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-6304 or by e-mail at melvinv@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs are on the last page of this report. Key contributors to this report are listed in appendix XII.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) assess the extent to which selected departments and agencies have appropriately implemented cost- estimating policies and procedures, and (2) evaluate whether selected information technology (IT) investments at these departments and agencies have reliable cost estimates to support budget and program decisions. For this review, we assessed eight federal agencies and 16 investments.

To select these agencies and investments, we relied on the Office of Management and Budget’s Fiscal Year 2010 Exhibit 53 which, at the time we made our selections, contained the most current and complete data on 28 agencies’ planned IT spending. To ensure that we selected agencies with varying levels of spending on IT, we sorted them into three ranges based on their planned spending in fiscal year 2010: greater than or equal to $10 billion; greater than or equal to $1 billion but less than $10 billion; and greater than $0, but less than $1 billion.

The number of agencies selected from each range was based on the relative number of IT investments within each range, and the specific agencies selected were those with the highest amount of planned IT spending in fiscal year 2010. Specifically, we selected one agency with greater than $10 billion in planned IT spending,between $1 billion and $10 billion in planned spending, and two agencies with less than $1 billion in planned spending. In doing so, we limited our selections to those agencies at which we could identify two investments that met our selection criteria for investments (see the following paragraph for a discussion of our investment selection methodology). These agencies were the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Labor, and Veterans Affairs, and the Environmental Protection Agency. We excluded the Departments of Education, Health and Human Services, and the Treasury, and the General Services Administration from our selection, even though they initially met our agency selection criteria, because we could not identify two investments at these agencies that met our investment selection criteria.

The Office of Management and Budget defines a major IT investment as a system or an acquisition requiring special management attention because it has significant importance to the mission or function of the agency, a component of the agency, or another organization; is for financial management and obligates more than $500,000 annually; has significant program or policy implications; has high executive visibility; has high development, operating, or maintenance costs; is funded through other than direct appropriations; or is defined as major by the agency’s capital planning and investment control process. primarily an infrastructure investment, had a high percentage of steady- state spending versus development spending, had less than $5 million in planned spending for fiscal year 2010, or were the subjects of recent or ongoing GAO audit work.

To assess the extent to which selected agencies had appropriately implemented cost-estimating policies and procedures, we analyzed agency policies and guidance for cost estimating. Specifically, we compared these policies and guidance documents to best practices recognized within the federal government and private industry for cost estimating. These best practices are contained in the GAO Cost Guide and include, for example, establishing a clear requirement for cost estimating, requiring management review and approval of cost estimates, and requiring and enforcing training in cost estimating. For each policy component, we assessed it as either being not met—the agency did not provide evidence that it addressed the policy component or provided evidence that it minimally addressed the policy component; partially met—the agency provided evidence that it addressed about half or a large portion of the policy component; or fully met—the agency provided evidence that it fully addressed the policy component. We also interviewed key agency officials to obtain information on their ongoing and future cost-estimating plans.

GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009). implemented the practices; partially met—the program provided evidence that it implemented about half or a large portion of the practices; or fully met—the program provided evidence that it fully implemented the practices. We then summarized these assessments by characteristic. We also interviewed program officials to obtain clarification on how cost- estimating practices are implemented and how the cost estimates are used to support budget and program decisions.

We conducted this performance audit from July 2011 to July 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Case Studies of Selected Programs’ Cost-Estimating Practices

We conducted case studies of 16 major system acquisition programs (listed in table 5). For each of these programs, the remaining sections of this appendix provide the following: a brief description of the program and its life-cycle cost estimate, and an assessment of the program’s cost estimate against the four characteristics of a reliable cost estimate— comprehensive, well-documented, accurate, and credible.

The key below defines “fully met,” “partially met,” and “not met” as assessments of programs’ implementation of cost-estimating best practices.

The program provided evidence that it fully implemented the cost- estimating practices.

The program provided evidence that it implemented about half or a large portion of the cost-estimating practices.

The program did not provide evidence that it implemented the practices or provided evidence that it only minimally implemented the cost-estimating practices.

Public Health Information System

The Public Health Information System (PHIS) program is designed to modernize the Food Safety and Inspection Service’s systems for ensuring the safety of meat, poultry, and egg products. According to the agency, the current systems environment includes multiple, disparate legacy systems that do not effectively support agency operations. PHIS is intended to replace these legacy systems with a single, web-based system that addresses major business areas such as domestic inspection, import inspection, and export inspection. The program intends to implement functionality to support domestic inspection and import inspection in 2012, and export inspection in 2013.

In 2007, PHIS was a development contract within the larger Public Health Information Consolidation Projects investment. In 2011, after PHIS was separated out as its own major investment and the program was rebaselined, the PHIS program developed its own cost estimate of $82.3 million. This includes $71.4 million for development and $10.9 million for operations and maintenance over a 12-year life cycle.

The PHIS program’s current cost estimate does not exhibit all of the qualities of a reliable cost estimate. Specifically, while the estimate partially reflects key practices for developing a comprehensive estimate, it does not reflect key practices for developing a well-documented, accurate, or credible estimate. Table 6 provides details on our assessment of the PHIS program’s cost estimate.

Web-Based Supply Chain Management

The Web-Based Supply Chain Management (WBSCM) program is designed to modernize the U.S. Department of Agriculture’s commodity management operations, including the purchasing and distribution of approximately $2.5 billion in food products for distribution to needy recipients through domestic and foreign food programs. To accomplish this, the program is replacing a legacy system with a web-based commercial-off-the-shelf solution. In 2010, the program achieved full operational capability. Ongoing efforts are focused on addressing a significant number of system defects identified since deployment.

In 2003, WBSCM developed an initial cost estimate of $142.9 million. This included $105.5 million for development and $37.4 million for operations and maintenance over a 7-year life cycle. Subsequently, after revising the estimate each year as part the program’s Office of Management and Budget Exhibit 300 submission, in 2011, WBSCM revised its cost estimate to $378.4 million, an increase of about $235.5 million over its initial cost estimate. This includes $104.9 million for development and $273.5 million for operations and maintenance over a 18-year life cycle. These changes are due to, among other things, incorporating additional years of operations and maintenance costs, a recently planned system upgrade, and additional costs associated with addressing system defects.

The WBSCM program’s current cost estimate does not exhibit any of the qualities of a reliable cost estimate. Specifically, the estimate did not reflect key practices for developing a comprehensive, well-documented, accurate, or credible estimate. Table 7 provides details on our assessment of the WBSCM program’s cost estimate.

Comprehensive Large Array-data Stewardship System

The Comprehensive Large Array-data Stewardship System (CLASS) is designed to provide environmental data archiving and access. The National Atmospheric and Oceanic Administration has been acquiring these data for more than 30 years, from a variety of observing systems throughout the agency and from a number of its partners. Currently, large portions of the nation’s environmental data are stored and maintained in disparate systems, with nonstandard archive and access capabilities. With significant increases expected in both the data volume and the number and sophistication of users over the next 15 years, CLASS is intended to provide a standard, integrated solution for environmental data archiving and access managed at the enterprise level. CLASS is currently developing satellite data archiving and access capabilities for several satellite programs, including the next generation of geostationary satellites—known as the Geostationary Operational Environmental Satellites-R Series, which are planned for launch beginning in 2015.

In 2006, the National Oceanic and Atmospheric Administration developed the initial CLASS cost estimate of approximately $195.5 million. This included $118.3 million for development and $77.2 million for operations and maintenance over a 9-year life cycle. Subsequently, after revising the cost estimate three times, in 2011, CLASS established its current cost estimate of approximately $240.0 million, an increase of about $44.5 million over its initial cost estimate. This includes $176.0 million for development and $64.0 million for operations and maintenance over a 17- year life cycle. CLASS program officials stated that the increase in the estimate was due, in part, to additional data archiving requirements and external program delays.

The CLASS program’s current cost estimate does not exhibit all qualities of a reliable cost estimate. Specifically, while the estimate partially reflects key practices for developing a comprehensive estimate, it does not reflect key practices for developing a well-documented, accurate, or credible estimate. Table 8 provides details on our assessment of CLASS program’s cost estimate.

Patents End-to-End: Software Engineering

The Patents End-to-End: Software Engineering (PE2E-SE) program is designed to provide a fully electronic patent application process. According to the U.S. Patent and Trademark Office, the agency’s current enterprise architecture is unable to meet current demands, and it has relied on inefficient and outdated automated legacy systems that inhibit the timely examination of patent applications. PE2E-SE intends to provide an electronic filing and processing application that enables examiners to meet current needs for the timely examination of patents. To accomplish this, PE2E-SE is following an Agile development approach and intends to implement a system using a text-based eXtensible Markup Language standard that is flexible, scalable, and leverages modern technologies with open standards. In fiscal year 2012, the program plans to build new functionality, such as new text search tools, and deploy the system to a limited set of examiners.

In 2010, PE2E-SE developed an initial cost estimate of $130.2 million. This estimate only included costs for development, over a 3-year life cycle. Subsequently, in 2012 and after multiple revisions, PE2E-SE revised its cost estimate to $188.2 million, an increase of $58.0 million. This includes $122.8 million for development and $65.4 million for operations and maintenance over a 7-year life cycle. According to program officials, these changes are primarily due to incorporating costs for operations and maintenance into the estimate.

The PE2E-SE program’s current cost estimate does not exhibit all of the qualities of a reliable cost estimate. Specifically, while the estimate partially reflects key practices for developing a comprehensive, well- documented, and accurate estimate, it does not reflect key practices for developing a credible estimate. Table 9 provides details on our assessment of the PE2E-SE program’s cost estimate.

Consolidated Afloat Networks and Enterprise Services

The Consolidated Afloat Networks and Enterprise Services (CANES) program is designed to consolidate and standardize the Department of the Navy’s existing network infrastructures and services. According to the department, the current network infrastructure is highly segmented and includes several legacy environments that have created inefficiencies in the management and support of shipboard networks. The CANES program is intended to, among other things, reduce and eliminate existing standalone afloat networks, provide a technology platform that can rapidly adjust to changing warfighting requirements, and reduce the shipboard hardware footprint. To accomplish this, the program will rely primarily on commercial off-the-shelf software integrated with network infrastructure hardware components. The CANES program is currently planning to procure and conduct preinstallation activities of four limited fielding units by the end of fiscal year 2012, and achieve full operational capability in 2023.

In 2010, the Navy’s Space and Naval Warfare Systems Command Cost Analysis Division developed a program life-cycle cost estimate for the CANES program, and the Naval Center for Cost Analysis developed an independent cost estimate. Subsequently, these organizations worked collaboratively to develop the program’s life-cycle cost estimate of approximately $12.7 billion. This included approximately $4.0 billion for development and approximately $8.8 billion for operations and maintenance over a 23-year life cycle.

The CANES program’s cost estimate exhibits all of the qualities of a reliable cost estimate. Specifically, the estimate reflects key practices for developing a comprehensive, well-documented, accurate, and credible estimate. Table 10 provides details on our assessment of the CANES program’s cost estimate.

Tactical Mission Command

The Tactical Mission Command (TMC)battle command system for commanders and staffs from battalions through the Army Service Component Commands. TMC is intended to provide commanders and staff with improved battle command capabilities, including increasing the speed and quality of command decisions. In the near term, TMC is to address gaps in the Army’s tactical battle command capability by delivering enhanced collaborative tools and enterprise services, and, in the long term, TMC is to address rapid improvements in technological capabilities through technology refresh. A key component—known as the Command Post of the Future—is intended to provide commanders and key staff with an executive-level decision support capability enhanced with real-time collaborative tools. These capabilities are expected to enhance situational awareness and support an execution-focused battle command process. Currently, the program is working to complete development of Command Post of the Future 7.0, which the program plans to complete by the end of fiscal year 2012. is designed to be the tactical In 2008, the TMC program developed an initial cost estimate of approximately $2.0 billion. This included approximately $1.9 billion for development and $116.5 million for maintenance over a 14-year life cycle. According to program officials, each subsequent year, in preparation for the annual Weapons System Review, the program updated its life-cycle cost estimate. In 2011 the TMC program established its current cost estimate of approximately $2.7 billion, an increase of approximately $723 million over its initial cost estimate. This included approximately $2.0 billion for development and $650.7 million for operations and maintenance over a 23-year life cycle. Program officials stated that the increase in the estimate was due, in part, to changes in the life-cycle time frames, fielding schedules, number of units planned for deployment, and other software development changes.

The TMC program’s current cost estimate does not exhibit all qualities of a reliable cost estimate. Specifically, while the estimate partially reflects key practices for developing a comprehensive, well-documented, and accurate estimate, it does not reflect key practices for developing a credible estimate. Table 11 provides details on our assessment of TMC program’s cost estimate.

Financial System Modernization Project

The Financial System Modernization Project (FSMP) replaced the Environmental Protection Agency’s legacy core financial system. The system is intended to address agency-identified shortcomings in its previous financial systems, such as inconsistent data, limited system interoperability, low system usability, and costly maintenance. FSMP includes key functionality for performing cost and project management, general ledger, payment management, and receivables management. According to the agency, the system is intended to, among other things, eliminate repetitive data entry, integrate legacy systems, and enable agency staff to manage workflow among the Office of the Chief Financial Officer and between other business lines (e.g., acquisitions and grants management). The system was deployed in October 2011.

In 2005, the FSMP program developed an initial cost estimate of approximately $163.2 million. This included $42.8 million for development and $120.4 million for operations and maintenance over a 25-year life cycle. After revising the cost estimate three times, in 2010 the program established its current cost estimate of approximately $169.3 million, an increase of approximately $6 million over its initial cost estimate. This includes $103.7 million for development and $65.7 million for operations and maintenance over a 15-year life cycle. Program officials stated that the changes to the program’s life-cycle cost estimate are due, in part, to changes in the Environmental Protection Agency’s policies and guidance, such as using a 15-year program life cycle instead of the 25-year life cycle used in the program’s original estimate. In addition, officials stated that the FSMP program has undergone significant schedule and scope changes, including delaying the system’s deployment date from 2008 to 2011 and reducing in the planned system components (e.g., budget formulation)—all of which have impacted the program’s life-cycle cost estimate.

The FSMP program’s current cost estimate does not exhibit all qualities of a reliable cost estimate. Specifically, while the estimate partially reflects key practices for developing a comprehensive, well-documented, and accurate estimate, it does not reflect key practices for developing a credible estimate. Table 12 provides details on our assessment of FSMP program’s cost estimate.

Superfund Enterprise Management System

The Superfund Enterprise Management System (SEMS) is to replace three legacy systems and multiple applications used to comply with the Comprehensive Environmental Response, Compensation, and Liability Act of 1980—commonly known as Superfund, which provides federal authority to respond directly to releases or threatened releases of hazardous substances that may endanger public health or the environment. In addition, SEMS is designed to implement innovative software tools that will allow for more efficient operation of the Superfund program. Of the three legacy systems expected to be replaced by SEMS, two have already been integrated, and the one remaining system is expected to be fully integrated in 2013, at which time SEMS is planned to achieve full operational capability.

In 2009, the SEMS program developed an initial cost estimate of approximately $39.3 million. This included $20.8 million for development, $14.7 million for operations and maintenance, and $3.8 million for government personnel costs over a 10-year life cycle. Subsequently, in 2011, the program revised its estimate to approximately $62.0 million, an increase of about $22.7 million over its initial cost estimate. This includes $22.8 million for development and $39.2 million for operations and maintenance over a 10-year life cycle. Program officials stated that the increase in the estimate was primarily due to incorporating additional operations and maintenance costs that were erroneously omitted from the initial estimate.

The SEMS program’s current cost estimate does not exhibit all qualities of a reliable cost estimate. Specifically, while the estimate partially reflects key practices for developing a credible estimate, it does not reflect key practices for developing a comprehensive, well-documented, or accurate estimate. Table 13 provides details on our assessment of SEMS program’s cost estimate.

Integrated Public Alert and Warning System

The Integrated Public Alert and Warning System (IPAWS) is designed to provide a reliable, integrated, and comprehensive system to alert and warn the American people before, during, and after disasters. To accomplish this, the program is developing the capability to disseminate national alerts to cellular phones and expanding the existing Emergency Alert System to cover 90 percent of the American public. In 2011, IPAWS established standards for alert messages, began cellular carrier testing, and conducted a nationwide test of the expanded Emergency Alert System capabilities. The program intends to deploy the cellular alerting capability nationwide in 2012 and complete its expansion of the Emergency Alert System in 2017.

In 2009, IPAWS developed its initial estimate of $259 million, which included $252.1 million for development and $6.9 million for government personnel costs, but did not include operations and maintenance costs. In 2011, the program revised its estimate to $311.4 million, an increase of about $52.3 million. This includes $268.9 million for development and $42.5 million for operations and maintenance over an 11-year life cycle. According to program officials, the increase in the cost estimate is primarily due to the inclusion of costs to operate and maintain the system during development.

The IPAWS program’s current cost estimate does not exhibit all qualities of a reliable cost estimate. Specifically, while the estimate fully reflects key practices for developing an accurate estimate, it only partially reflects key practices for developing a comprehensive, well-documented, and credible estimate. Table 14 provides details on our assessment of IPAWS program’s cost estimate.

Rescue 21

Rescue 21 is designed to modernize the U.S. Coast Guard’s maritime search and rescue capability. According to the agency, the current system—the National Distress and Response System, does not meet the demands of the 21st century in that it does not provide complete coverage of the continental United States, cannot receive distress calls during certain transmissions, lacks interoperability with other government agencies, and is supported by outdated equipment. Rescue 21 is intended to provide a modernized maritime distress and response communications system, with increased maritime homeland security capabilities that encompass coastlines, navigable rivers, and waterways in the continental United States, in addition to Hawaii, Guam, and Puerto Rico. Rescue 21 is currently undergoing regional deployment, which is planned to be completed in fiscal year 2017.

In 1999, the Rescue 21 program developed an initial cost estimate of $250 million for acquisition of the system, but this estimate did not include any costs for operations and maintenance of the system. Following three rebaselines, in 2006 the Rescue 21 program revised the estimate to $1.44 billion, an increase of approximately $1.19 billion over the initial estimate. This included $730 million in development and $707 million in operations and maintenance over a 16-year life cycle. According to program documentation, these increases were due, in part, to incorporating costs for the operation and maintenance of the system. Subsequently, in 2008, the Rescue 21 program revised its cost estimate again to $2.66 billion, an increase of approximately $1.22 billion over the previous estimate, and approximately $2.41 billion over the initial cost estimate. This includes $1.07 billion in development and $1.59 billion in operations and maintenance over a 16-year life cycle. Program officials stated that the most recent increase in the cost estimate was primarily due to schedule delays, an extension of the program’s life cycle by 6 years based on an expected increase in the system’s useful life, and to reflect more realistic estimates of future costs for ongoing system technology refreshment.

The Rescue 21 program’s current cost estimate does not exhibit all qualities of a reliable cost estimate. Specifically, the estimate partially reflects key practices for developing a comprehensive, well-documented, accurate, and credible estimate. Table 15 provides details on our assessment of the Rescue 21 program’s cost estimate.

Next Generation Combined DNA Index System

Since 1998, the Combined DNA Index System (CODIS) has supported the Federal Bureau of Investigation’s mission by assisting criminal investigation and surveillance through DNA collection and examination capabilities. CODIS is an automated DNA information processing and telecommunications system that generates potential investigative leads in cases where biological evidence is recovered. Among other things, CODIS links crime scene evidence to other crimes and/or offenders, which can identify serial offenders and/or potential suspects. CODIS serves over 190 participating laboratories and 73 international laboratories representing 38 countries. According to the Federal Bureau of Investigation, the reliability and expandability of CODIS are critical to the agency’s ability to effectively aid law enforcement investigations through the use of biometrics, prompting the decision in 2006 to initiate a modernization effort, referred to as Next Generation CODIS (NGCODIS). In 2011, the program achieved full operational capability for CODIS 7.0, a software release of NGCODIS, which included functionality for, among other things, implementing a software solution to comply with European Union legislation for DNA data exchange and maintaining DNA records of arrested persons. Additional functionality is expected in the future; however, all program development has been put on hold until the necessary funding is approved.

In 2006, the CODIS program developed an initial cost estimate for NGCODIS of $128.4 million. This included approximately $69.6 million for development and $58.8 million for operations and maintenance over an 11-year life cycle. In 2009, the CODIS program developed an additional cost estimate of $58.6 million to account for operations costs associated with certain versions of NGCODIS. According to program officials, even though the program estimated additional operations costs of $58.6 million, the program’s original cost estimate has increased by only $8.6 million because originally planned development work related to incorporating advancements in DNA technology was delayed and the costs associated with this work were removed from the cost estimate.

The CODIS program’s current cost estimate for NGCODIS does not exhibit all qualities of a reliable cost estimate. Specifically, the estimate partially reflects key practices for developing a comprehensive, well- documented, accurate, and credible estimate. Table 16 provides details on our assessment of the NGCODIS cost estimate.

Unified Financial Management System

The Unified Financial Management System (UFMS) is to modernize the Department of Justice’s financial management and procurement operations. To accomplish this, UFMS is to replace four legacy core accounting systems and multiple procurement systems with a commercial off-the-shelf product. Ultimately, the system is expected to streamline and standardize financial management and procurement processes and procedures across the department’s component agencies. UFMS was deployed to two component agencies—the Drug Enforcement Administration and the Bureau of Alcohol, Tobacco, Firearms, and Explosives—in fiscal years 2009 and 2011, respectively. The system is planned to be deployed at other component agencies, including the U.S. Marshals Service and the Federal Bureau of Investigation, between fiscal years 2013 and 2014, and is expected to achieve full operational capability in fiscal year 2014.

In 2002, the UFMS program developed an initial cost estimate of $357.2 million. This included approximately $196.4 million for development and $160.8 million for maintenance over a 10-year life cycle. In 2009, the UFMS program revised the estimate to $1.05 billion, an increase of approximately $692.8 million. This included $469.5 million for development and $581.6 million for operations and maintenance over a 20-year life cycle. Program officials stated that the increase in the estimate was due to extending the program’s life cycle to include additional years of development work and operations and maintenance of the system. Subsequently, in 2011, the program revised its cost estimate to $851.1 million, a decrease of approximately $198.9 million. This estimate includes $419.5 million for development and $431.6 million for operations and maintenance over a 20-year life cycle. Program officials stated that the decrease in the cost estimate was due to a reduction in the number of component agencies that planned to implement UFMS. Specifically, UFMS removed the Federal Bureau of Prisons; Offices, Boards and Divisions; and Office of Justice Programs from the system’s deployment schedule in order to reduce the overall cost of the system.

The UFMS program’s current cost estimate does not exhibit all qualities of a reliable cost estimate. Specifically, while the estimate partially reflects key practices for developing a comprehensive, well-documented, and accurate estimate, it does not reflect key practices for developing a credible estimate. Table 17 provides details on our assessment of UFMS program’s cost estimate.

OSHA Information System

The OSHA Information System (OIS) is a management tool consisting of a suite of applications to reduce workplace fatalities, injuries, and illnesses through enforcement, compliance assistance, and consultation. According to the agency, OIS is intended to close performance gaps with existing legacy systems resulting from irreplaceable legacy hardware and software, the inability of legacy systems to fully support the agency’s mission, and the absence of an application that supports key business process areas, such as compliance assistance. Ultimately, OIS is expected to provide a centralized web-based solution to be used by more than 5,900 users at the federal and state level, including approximately 4,200 enforcement officers and 500 safety and health consultants. The program completed development in 2011, and is working to complete deployment of the system while addressing operations and maintenance of the system, which the program plans to complete by the end of fiscal year 2016.

In 2006, the OIS program developed an initial cost of $72.3 million. This included $42.0 million for development and $30.3 million for operations and maintenance over a 12-year life cycle. Subsequently, in 2010, the OIS program revised its cost estimate to $91.3 million, an increase of $19.0 million. This includes $63.3 million for development and approximately $28.0 million for operations and maintenance over a 12- year life cycle. The OIS Program Manager stated that the increase in the estimate was due, in part, to unanticipated changes to the OIS program’s scope to better align with the Department of Labor’s strategic goals, including securing safe and healthy workplaces, particularly in high-risk industries. For example, according to this official, the agency’s methodology for penalty calculations for violators of occupational safety and health rules and regulations was modified, which required a redesign of OIS in order to capture and accurately calculate these changes.

The OIS program’s current cost estimate does not exhibit all qualities of a reliable cost estimate. Specifically, while the estimate partially reflects key practices for developing a comprehensive, well-documented, and accurate estimate, it does not reflect key practices for developing a credible estimate. Table 19 provides details on our assessment of the OIS program’s cost estimate.

PBGC Benefit Administration

The Pension Benefit Guaranty Corporation’s (PBGC) Benefit Administration (BA) is a collection of IT systems and applications that allows PBGC to administer and service the approximately 1.5 million participants in over 4,300 plans that have been terminated and trusteed as part of PBGC’s insurance program for single-employer pensions. The BA program is intended to modernize and consolidate applications, retire legacy systems, and address performance gaps. To do this, the BA program is grouped into four projects—Customer Care, Document Management, Case Management, and Benefit Management—in support of paying accurate and timely payments and providing customer service to participants. The BA program is expected to offer multiple self-service channels to participants, reengineer benefit payment processes to increase efficiency and productivity, and implement enhanced reporting and document management systems. According to the agency, this modernization effort is ultimately expected to increase customer satisfaction, reduce operational costs, and improve data quality. Currently, the program is scheduled to complete modernization and decommission the remaining legacy applications in fiscal year 2015.

In 2007, the BA program developed an initial cost estimate of $186.9 million. This included $39.4 million for development and $147.5 million for operations and maintenance over a 5-year life cycle. Subsequently, in 2010, BA revised its cost estimate to $155.9 million, a decrease of $31.0 million. This revised estimate includes $80.7 million for development and approximately $75.2 million for operations and maintenance over a 10- year life cycle. Program officials stated that the decrease in the estimate was due to changes to the program’s schedule milestones and changes to the system’s architecture.

The BA program’s current cost estimate does not exhibit all qualities of a reliable cost estimate. Specifically, the estimate partially reflects key practices for developing a comprehensive, well-documented, accurate, and credible estimate. Table 18 provides details on our assessment of the BA program’s cost estimate.

Health Data Repository

The Health Data Repository (HDR) is intended to support the integration of clinical data across the Department of Veterans Affairs and with external healthcare systems such as that of the Department of Defense. Specifically, the system is designed to provide a nationally accessible repository of clinical data by accessing and making available data from existing healthcare systems to support clinical and nonclinical decision- making for the care of the department’s patients. The system is being developed using an Agile software development approach and, currently, the program is working on software releases to improve the ability to access data in VA’s legacy healthcare information system, and intends to achieve full operating capability in 2017.

In 2001, the HDR program developed an initial cost estimate of $126.7 million. This included $105.9 million for development and $20.8 million for operations and maintenance over a 7-year life cycle. According to officials, the program revised its estimate each year during the budget cycle; in 2011, HDR revised its cost estimate to $491.5 million, an increase of approximately $364.8 million over its initial cost estimate. This includes $281.9 million for development and $209.6 million for operations and maintenance over a 17-year life cycle. Program officials stated that the increase in the cost estimate was primarily due to the unplanned deployment and operation of a prototype system for 5 years, and the delay of the planned date for full operational capability from 2006 to 2017, in part, because of changes in the program’s scope and technology refreshes (i.e., equipment and storage capacity).

The HDR program’s current cost estimate does not exhibit any of the qualities of a reliable cost estimate. Specifically, the estimate does not reflect key practices for developing a comprehensive, well-documented, accurate, and credible estimate. Table 20 provides details on our assessment of HDR program’s cost estimate.

Veterans Benefits Management System

The Veterans Benefits Management System (VBMS) is intended to provide a paperless claims processing system to support processing a growing volume of claims—for example, the number of compensation and pension claims submitted in a year passed 1 million for the first time in 2009. According to the department, due to the reliance on paper-based processing, the current system is inefficient and costly, and carries risks to veterans’ sensitive information. To address this, VBMS is designed to provide veterans a secure and accessible means to obtain benefits, reduce the claims backlog, implement standardized business practices, and support the integration with other veteran-facing systems. The program is currently developing functionality for compensation and pension claims processing, and plans to add additional lines of business in future years.

In 2008, the VBMS program developed an initial, high-level cost estimate of $560.0 million for system development over a 5-year life cycle, which did not include costs for operations and maintenance. Subsequently, after revising the estimate each year as part the program’s Office of Management and Budget Exhibit 300 submission, in 2011 VBMS revised its cost estimate to $934.8 million, an increase of approximately $374.8 million over its initial estimate. This includes $433.7 million for development and $501.1 million for operations and maintenance over an 11-year life cycle. Program officials stated that the increase in the estimate was primarily due to incorporating costs associated with operations and maintenance and effort spent on changing to an Agile development approach.

The VBMS program’s current cost estimate does not exhibit all of the qualities of a reliable cost estimate. Specifically, while the estimate partially reflects key practices for developing a comprehensive and well- documented estimate, it does not reflect key practices for developing an accurate and credible estimate. Table 21 provides details on our assessment of the VBMS program’s cost estimate.

Appendix III: Original and Current Life-Cycle Cost Estimates for Case Study Programs

Collectively, 13 of the 16 case study programs have revised their cost estimates upward by almost $5 billion. More specifically, the 13 programs have experienced cost increases ranging from about $6 million to over $2 billion. For example, in many cases, cost estimates had to be revised upwards to reflect the incorporation of full costs for all life-cycle phases (e.g. development or operations and maintenance), which had not originally been included. Other reasons that programs cited for revising their life-cycle cost estimates upward included changes to program or system requirements, schedule delays, technology upgrades, and system defects, among other things. Among the remaining 3 programs, 1 program’s cost estimate had decreased, 1 had not changed, and 1 was not applicable because the program only had a current cost estimate (see table 22).

Appendix IV: Comments from the Department of Agriculture

Appendix V: Comments from the Department of Commerce

Appendix VI: Comments from the Department of Defense

Appendix VII: Comments from the Environmental Protection Agency

Appendix VIII: Comments from the Department of Homeland Security

Appendix IX: Comments from the Department of Labor

Appendix X: Comments from the Pension Benefit Guaranty Corporation

Appendix XI: Comments from the Department of Veterans Affairs

Appendix XII: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact name above, individuals making contributions to this report included Eric Winter (Assistant Director), Mathew Bader, Carol Cha, Jennifer Echard, J. Christopher Martin, Lee McCracken, Constantine Papanastasiou, Karen Richey, Matthew Snyder, and Jonathan Ticehurst.