Information Technology Cost Estimation: Agencies Need to Address Significant Weaknesses in Policies and Practices
Published: 2012-07-11T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The federal government plans to spend at least $75 billion on information technology (IT) investments in fiscal year 2012. The size of this investment highlights the importance of reliably estimating the costs of IT acquisitions. A reliable cost estimate is critical to the success of any IT program, providing the basis for informed decision making and realistic budget formation. Without the ability to generate such estimates, programs risk missing their cost, schedule, and performance targets.
GAO was asked to (1) assess selected federal agencies implementation of cost-estimating policies and procedures, and (2) evaluate whether selected IT investments at these agencies have reliable cost estimates to support budget and program decisions. To do so, GAO compared policies and procedures to best practices at eight agencies. GAO also reviewed documentation supporting cost estimates for 16 major investments at these eight agenciesrepresenting about $51.5 billion of the planned IT spending for fiscal year 2012.
section_title: What GAO Found
paragraphs: While the eight agencies GAO reviewedthe Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Labor, and Veterans Affairs, and the Environmental Protection Agencyvaried in the extent to which their cost-estimating policies and procedures addressed best practices, most had significant weaknesses. For example, six of the eight agencies had established a clear requirement for programs to develop life-cycle cost estimates. However, most of the eight agencies policies lacked requirements for cost-estimating training, a standard structure for defining work products, and a central, independent cost-estimating team, among other things. The weaknesses in agencies policies were due, in part, to the lack of a priority for establishing or enhancing department or agency-level cost-estimating functions. Until agencies address weaknesses in their policies, it will be difficult for them to make effective use of program cost estimates for informed decision making, realistic budget formation, and meaningful progress measurement.
The 16 major acquisition programs had developed cost estimates and were using them, in part, to support program and budget decisions. However, all but 1 of the estimates were not fully reliablemeaning that they did not fully reflect all four characteristics of a reliable cost estimate identified in the GAO cost-estimating guide: comprehensive, well-documented, accurate, and credible. For example, the estimates for many of these investments did not include all life-cycle costs, such as costs for operating and maintaining the system; did not adequately document the source data and methodologies used to develop the estimate; were not regularly updated so that they accurately reflected current status; and lacked credibility because they were not properly adjusted to account for risks and uncertainty. The inadequate implementation of cost-estimating best practices was largely due to weaknesses in agencies policies. Until cost-estimating best practices are fully implemented, these programs face an increased risk that managers will not be able to effectively use their cost estimates as a sound basis for informed program and budget decision making.
section_title: What GAO Recommends
paragraphs: GAO is recommending that the selected agencies modify cost-estimating policies to be consistent with best practices and update future cost estimates of the selected acquisition programs to address identified weaknesses. The seven agencies that commented on a draft of this report generally agreed with GAOs results and recommendations, although the Environmental Protection Agency disagreed with the assessment of one of its investments. However, GAO stands by its assessment.

section_title: Letter
paragraphs: In fiscal year 2012, the federal government plans to spend at least $75  billion on information technology (IT) investments, many of which involve  systems and technologies to modernize legacy systems, increase  communication and networking capabilities, and transition to new  systems designed to significantly improve the government’s ability to  carry out critical mission functions in the 21st century. Given the size of  this investment, it is important that IT acquisitions are based on reliable  estimates of costs over their full acquisition life cycles. The ability to  generate a reliable cost estimate is critical to the success of any IT  program, as it provides the basis for informed decision making, realistic  budget formulation, and meaningful progress measurement. Without this  ability, programs are at risk of experiencing cost overruns, missed  deadlines, and performance shortfalls.
This report responds to your request that we evaluate the implementation  of cost-estimating processes at selected federal government departments  and agencies. Specifically, our objectives were to (1) assess the extent to  which selected departments and agencies have appropriately  implemented cost-estimating policies and procedures, and (2) evaluate  whether selected IT investments at these departments and agencies have  reliable cost estimates to support budget and program decisions.
To assess the extent to which selected departments and agencies have  appropriately implemented cost-estimating policies and procedures, we  reviewed cost-estimating policies and procedures from eight agencies.  The eight agencies were selected from across different ranges of planned  IT spending in fiscal year 2010. The number of agencies selected from  each range was based on the relative number of IT investments within  each range, and the specific agencies selected were those with the  highest amount of planned IT spending in fiscal year 2010. Specifically,  we chose one agency with greater than $10 billion in planned IT  spending, five agencies with between $1 billion and $10 billion in  planned spending, and two agencies with less than $1 billion in planned  spending. We compared the agencies’ policies and procedures with the  best practices identified in GAO’s cost-estimating guide to determine the  comprehensiveness of each agency’s established policies for cost  estimating. For each policy component, we assessed it as either being  not met—the agency did not provide evidence that it addressed the policy  component or provided evidence that it minimally addressed the policy  component; partially met—the agency provided evidence that it  addressed about half or a large portion of the policy component; or fully  met—the agency provided evidence that it fully addressed the policy  component. In addition, we interviewed relevant agency officials,  including officials responsible for developing cost-estimating policies.
To evaluate whether selected IT investments at these departments and  agencies have reliable cost estimates to support budget and program  decisions, we reviewed individual programs’ relevant cost-estimating  documentation, including, for example, the current life-cycle cost estimate  and schedule and technical baseline information, from 16 major  investments at the eight agencies. The 16 programs selected for case  study (2 per agency) were among the largest in terms of planned  spending; considered major IT investments; and had a higher  percentage of development versus steady-state spending, among other  things. We compared the programs’ life-cycle cost estimates and  underlying support with the best practices identified in GAO’s cost- estimating guidereliable and are being used to support budget and program decisions.  Specifically, we assessed program practices against the four  characteristics of a reliable estimate—comprehensive, well-documented,  accurate, and credible. For each characteristic, we assessed multiple  practices as being not met—the program did not provide evidence that it  implemented the practices or provided evidence that it only minimally  implemented the practices; partially met—the program provided evidence  that it implemented about half or a large portion of the practices; or fully  met—the program provided evidence that it fully implemented the  practices. We then summarized these assessments by characteristic. In  addition, we interviewed relevant agency officials, including key personnel  on the programs that we selected for case study.   to determine the extent to which the estimates are  We conducted this performance audit from July 2011 through July 2012,  in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Appendix I contains further  details about our objectives, scope, and methodology.
subsections: section_title: Background
paragraphs: Given the size and significance of the government’s investment in IT, it is  important that projects be managed effectively to ensure that public  resources are wisely invested. Effectively managing projects entails,  among other things, developing reliable and high-quality cost estimates  that project realistic life-cycle costs. A life-cycle cost estimate provides an  exhaustive and structured accounting of all resources and associated  cost elements required to develop, produce, deploy, and sustain a  particular program. In essence, life cycle can be thought of as a “cradle to  grave” approach to managing a program throughout its useful life.  Because a life-cycle cost estimate encompasses all past (or sunk),  present, and future costs for every aspect of the program, regardless of  funding source, it provides a wealth of information about how much  programs are expected to cost over time.
We have previously reported that a reliable cost estimate is critical to  the success of any government acquisition program, as it provides the  basis for informed investment decision making, realistic budget  formulation and program resourcing, meaningful progress measurement,  proactive course correction, and accountability for results. Having a  realistic, up-to-date estimate of projected costs—one that is continually  revised as the program matures—can be used to support key program  decisions and milestone reviews. In addition, the estimate is often used to  determine the program’s budget spending plan, which outlines how and at  what rate the program funding will be spent over time. Because a  reasonable and supportable budget is essential to a program’s efficient  and timely execution, a reliable estimate is the foundation of a good  budget. However, we have also found that developing reliable cost  estimates has been difficult for agencies across the federal government.Too often, programs cost more than expected and deliver results that do  not satisfy all requirements.
In 2006, the Office of Management and Budget (OMB) updated its Capital  Programming Guide, which requires agencies to develop a disciplined  cost-estimating capability to provide greater information management  support, more accurate and timely cost estimates, and improved risk  assessments to help increase the credibility of program cost estimates. Further, according to OMB, programs must maintain current and well- documented estimates of costs, and these estimates must encompass  the full life cycle of the program. Among other things, OMB states that  generating reliable cost estimates is a critical function necessary to  support OMB’s capital programming process. Without this ability,  programs are at risk of experiencing cost overruns, missed deadlines,  and performance shortfalls.
Building on OMB’s requirements, in March 2009, we issued a guide on  best practices for estimating and managing program costs that highlights  the policies and practices adopted by leading organizations to implement   Specifically, these best practices  an effective cost-estimating capability.
OMB, Circular No. A-11, Preparation, Submission, and Execution of the Budget  (Washington, D.C.: Executive Office of the President, June 2006) and Capital  Programming Guide: Supplement to Circular A-11, Part 7, Planning, Budgeting, and  Acquisition of Capital Assets (Washington, D.C.: Executive Office of the President, June  2006). OMB first issued the Capital Programming Guide as a supplement to the 1997  version of Circular A-11, Part 3. We refer to the 2006 version. OMB later updated this  guide again in August 2011. identify the need for organizational policies that define a clear  requirement for cost estimating; require compliance with cost-estimating  best practices; require management review and acceptance of program  cost estimates; provide for specialized training; establish a central,  independent cost-estimating team; require a standard structure for  defining work products; and establish a process to collect and store cost- related data. In addition, the cost-estimating guide identifies four  characteristics of a reliable cost estimate that management can use for  making informed program and budget decisions: a reliable cost estimate  is comprehensive, well-documented, accurate, and credible. Specifically,  an estimate is  comprehensive when it accounts for all possible costs associated with  a program, is structured in sufficient detail to ensure that costs are  neither omitted nor double counted, and documents all cost- influencing assumptions; well-documented when supporting documentation explains the  process, sources, and methods used to create the estimate, contains  the underlying data used to develop the estimate, and is adequately  reviewed and approved by management; accurate when it is not overly conservative or optimistic, is based on  an assessment of the costs most likely to be incurred, and is regularly  updated so that it always reflects the current status of the program;  and credible when any limitations of the analysis because of uncertainty or  sensitivity surrounding data or assumptions are discussed, the  estimate’s results are cross-checked, and an independent cost  estimate is conducted by a group outside the acquiring organization to  determine whether other estimating methods produce similar results.
We have previously reported on weaknesses associated with the  implementation of sound cost-estimating practices at various agencies  and the impact on budget and program decisions. For example,  In January 2012, we reported that the Internal Revenue Service did  not have comprehensive guidance for cost estimating. Specifically,  the agency’s guidance did not clearly discuss the appropriate uses of  different types of cost estimates. Further, our review of the agency’s  Information Reporting and Document Matching program’s cost  estimate found it was unreliable. Among other things, the program’s  projected budget of $115 million through fiscal year 2016 was only  partly supported by the cost estimate, which included costs only  through fiscal year 2014. As a result, the agency did not have a  reliable basis for the program’s budget projection. We made multiple  recommendations to improve the quality of the agency’s cost and  budget information, including ensuring that the Information Reporting  and Document Matching program’s cost estimate is reliable and that  the agency’s cost-estimating guidance is consistent and clearly  requires the use of current and reliable cost estimates to inform  budget requests. The agency partially agreed with these  recommendations and stated that they have taken steps to ensure  that their cost-estimating practices and procedures follow consistent  documented guidance.
In January 2010, we reported that the Department of Energy lacked  comprehensive policy for cost estimating, making it difficult for the  agency to oversee development of high-quality cost estimates.  Specifically, the agency’s policy did not describe how estimates  should be developed and did not establish a central office for cost  estimating. Further, we reviewed four programs at the department,  each estimated to cost approximately $900 million or more, and  reported that they did not have reliable cost estimates. For example,  three of the cost estimates did not include costs for the full life cycles  of the programs, omitting operations and maintenance costs or  portions of program scope. Additionally, three of the cost estimates  did not use adequate data, one of which relied instead on professional  opinion. Further, the cost estimates did not fully incorporate risk— specifically, they did not address correlated risks among project  activities. As a result, these programs were more likely to exceed their  estimates and require additional funding to be completed. We made  multiple recommendations to improve cost estimating at the  department, including updating its cost-estimating policy and guidance  and ensuring cost estimates are developed in accordance with best  practices. The Department of Energy generally agreed with our  recommendations and stated that it had several initiatives underway  to improve cost-estimating practices, including the development of a  new cost-estimating policy and guidance, a historical cost database to  support future estimates, and additional training courses.
Finally, we reported in December 2009 that the Department of  Veterans Affairs had 18 construction projects that had experienced  cost increases due, in part, to unreliable cost estimates. For  example, many estimates were completed quickly, one of which was a  rough-order-of-magnitude estimate that was not intended to be relied  on as a budget-quality estimate of full project costs. Additionally, we  found that some projects had not conducted a risk analysis to quantify  the impact of risk on the total estimated costs. As a result, in some  cases, projects had to change scope to meet their initial estimate and,  in others, additional funds had to be requested from Congress to allow  the agency to complete the project. We recommended that the  department improve cost estimating at major construction projects by  conducting cost risk analyses and mitigating risks that may influence  projects’ costs. The Department of Veterans Affairs agreed with our  recommendation and stated that it was taking steps, such as  developing a multiyear construction plan to ensure that reliable  projections of program costs are available for budgeting purposes,  and planning to improve its risk analyses.
section_title: Selected Agencies’ Cost-Estimating Policies and Procedures Have Significant Weaknesses
paragraphs: According to OMB, agencies should develop a disciplined cost- estimating capability to provide greater information management support,  more accurate and timely cost estimates, and improved risk assessments  to help increase the credibility of program cost estimates. In addition, we  have reportedpolicies and procedures that   that leading organizations establish cost-estimating define a clear requirement for cost estimating;  identify and require compliance with cost-estimating best practices,  and validate their use; require that estimates be reviewed and approved by management; require and enforce training in cost estimating; establish a central, independent cost-estimating team;  require, at a high level, a standard, product-oriented work breakdown  structure; and establish a process for collecting and storing cost-related data to  support future estimates.
Table 1 describes the key components of an effective cost-estimating  policy.
While the eight agencies varied in the extent to which their cost- estimating policies and procedures addressed best practices, most did  not address several key components of an effective policy. Specifically,  only the Department of Defense’s (DOD) policy was fully consistent with  all seven components. While the Department of Homeland Security  addressed most components of an effective cost-estimating policy, other  agencies’ policies had significant weaknesses, particularly in cost- estimating training and in establishing a process to collect and store cost- related data.
Table 2 provides a detailed assessment of each agency’s policies against  the components of an effective cost-estimating policy. In addition, a  discussion of each policy component follows the table.
Clear requirement for cost estimating: Six of the eight agencies fully  addressed this policy component by establishing a clear requirement  for all programs to perform life-cycle cost estimates, and in certain  cases specified more stringent requirements for programs designated  as major investments. Among these, four agencies—the Department  of Agriculture, the Environmental Protection Agency (EPA), the  Department of Labor, and the Department of Justice—established this  requirement as part of their policies for programs to perform a cost- benefit analysis. For example, Labor required a life-cycle cost  estimate as part of a cost-benefit analysis for both major and  nonmajor investments, with less detail required for nonmajor  investments. The other two agencies—DOD and Homeland  Security—defined a separate requirement for programs to develop  life-cycle cost estimates. For the two agencies that did not fully  establish a clear requirement for cost estimating, the Department of  Veterans Affairs partially addressed this component because its policy  only requires cost estimates to be prepared for project increments,  rather than the full program life cycle. In addition, the Department of  Commerce partially addressed this component because its policies  only require cost estimates to be prepared for contracts, rather than  for the full program life cycle (including government and contractor  costs). Officials at both agencies stated that the responsibility for  establishing requirements for cost estimating had been delegated to  their component agencies. Further, officials at these two agencies  described steps planned to address this and other weaknesses. For  example, Veterans Affairs officials stated that the agency’s recently  established Office of Corporate Analysis and Evaluation (part of the  Office of Planning and Policy) is planning to establish a centralized  cost-estimating policy that includes clear criteria for cost estimating,  which it expects to complete in fiscal year 2012. Further, Commerce  officials stated that the agency is currently in the process of updating  its policy and guidance to address this and other weaknesses, which it  plans to complete by October 2012. If the updated policies and  guidance address the weaknesses we identified, decision makers  should have an improved view of their programs’ life-cycle costs.
Compliance with cost-estimating best practices: Three of the eight  agencies (DOD, Homeland Security, and Labor) fully addressed this  policy component by identifying and requiring the use of cost- estimating best practices by their programs, and defining a process to  validate their use. For example, Homeland Security draws on the  GAO cost guide to identify cost-estimating best practices, and also  provides agency-specific cost-estimating requirements for  implementing the practices, such as identifying the cost-estimate  documentation required. The agency’s policy also requires that  estimates for key programs be validated. For the three agencies that  partially addressed this policy component—Agriculture, EPA, and  Justice—all provided guidance to their programs specific to  conducting a cost-benefit analysis; however, this guidance did not  fully address important cost-estimating practices, such as conducting  a risk and uncertainty analysis, updating the estimate, or comparing  the estimate to an independent estimate. Their guidance also did not  identify a mechanism for validating estimates. Lastly, two agencies— Commerce and Veterans Affairs—had not addressed this policy  component, which corresponds to our finding that these agencies did  not have requirements for programs to prepare cost estimates.  Among the five agencies that did not fully address this policy  component, officials commonly stated that the responsibility for  requiring compliance with best practices had been delegated to their  component agencies or that addressing cost-estimating shortcomings  had not been a priority. Without fully complying with best practices for  developing cost estimates, programs are less likely to prepare reliable  cost estimates, hindering agency decision making.
Management review and approval: Three of the eight agencies (DOD,  Homeland Security, and Labor) fully addressed this policy component  by requiring that program cost estimates be reviewed and approved  by management, including defining the information to be presented  and requiring that approval be documented. For example, Labor’s  policy requires that senior management at both the component  agency responsible for the program and the Office of the Chief  Information Officer approve the estimate, based on a briefing that  includes information about the estimate such as the largest cost  drivers, major risks, and the findings of the integrated baseline  review, and that this approval is documented. For the three  agencies that partially addressed this policy component (Agriculture,  EPA, and Veterans Affairs), all required that estimated costs be  presented to management, but none fully defined the information to  be presented, such as the confidence level associated with the  estimate. Lastly, neither Justice nor Commerce had departmental  requirements for management review and approval of the cost  estimate. Officials at both agencies stated that this responsibility had  been delegated to their component agencies. However, without  requiring management review and approval of program cost estimates  at the department level, agencies have reduced ability to enforce cost- estimating policies and ensure that cost estimates meet  management’s needs for reliable information about programs’  estimated costs.
Training requirements: Only one agency—DOD—fully addressed this  policy component by requiring cost-estimating training and enforcing  this requirement. For example, DOD requires training in cost  estimating via its Defense Acquisition Workforce Improvement Act  certifications, among other things, for at least one staff member for  each major program, as well as for personnel with investment  oversight responsibility. While the two agencies that partially  addressed this policy component (Homeland Security and Labor)  provided cost-estimating training and had a mechanism to track  participation, their policies did not address providing training to  personnel with investment oversight responsibility, such as officials  from Homeland Security who are responsible for reviewing and  approving programs at key milestones in their life cycles. Among the  five agencies whose policies did not address requiring and enforcing  training in cost estimating (Agriculture, Commerce, EPA, Justice, and  Veterans Affairs), four of these agencies referred to OMB’s Federal  Acquisition Certification for Program and Project Managers as  providing for such training. However, this certification program does  not require classes on cost estimating, and furthermore, is not  intended for nor provided to individuals with investment oversight  responsibility. Additionally, officials at two of the five agencies— Commerce and Veterans Affairs—stated that training in cost  estimating had not been viewed as a priority. Without requiring and  enforcing training in cost estimating, agencies cannot effectively  ensure that staff have the skills and knowledge necessary to prepare  and use cost estimates to make reliable budget and program  decisions.
Central, independent cost-estimating team: Three of the eight  agencies (DOD, Homeland Security, and Veterans Affairs) fully  addressed this policy component by establishing central, independent  cost-estimating teams, all of which have responsibility for, among  other things, developing cost-estimating guidance and validating that  program cost estimates are developed in accordance with best  practices. In addition, among these three agencies, the teams established at DOD and Veterans Affairs are also charged with  improving cost-estimating training. The remaining five agencies had  not established a central, independent cost-estimating team. Among  these, officials commonly cited the lack of a priority at the department  or agency level for cost-estimating initiatives, although in one case a  component agency at Agriculture—the Food Safety and Inspection  Service—established its own centralized cost-estimating team. While  this will likely enhance cost estimating at the component agency, not  centralizing the cost-estimating function in the department could result  in ad hoc processes and a lack of commonality in the estimating tools  and training across the department. Additionally, officials from Labor  stated they believe the department’s IT budget is too small to cost- effectively centralize the cost-estimating function; however, doing so  would likely, among other things, facilitate a better sharing of  resources and could be accomplished in a manner commensurate  with agency size. Agencies that do not establish a central and  independent cost-estimating team may lack the ability to improve the  implementation of cost-estimating policies, support cost-estimating  training, and validate the reliability of program cost estimates at the  department or agency level.
Standard structure for defining work products: DOD was the only  agency to fully address this policy component by developing and  requiring the use of standard, product-oriented work breakdown  structures. Specifically, the agency provided multiple standard work  breakdown structures, along with detailed guidance, for different types  of programs (e.g., automated information systems, space systems,  aircraft systems), and required their use. Three agencies—Homeland  Security, Justice, and Veterans Affairs—partially addressed this policy  component in that they provided one or more product-oriented work  breakdown structures in their policies, but did not require programs to  use them for cost estimating. Among these, Justice officials stated  that a standard work breakdown structure was only required for their  earned value management processes. Further, both Veterans  Affairs and Homeland Security stated that they intend to require the  use of a standard work breakdown structure in the future, but had not  yet determined a time frame for establishing this requirement. Lastly,  four of the selected agencies—Agriculture, Commerce, EPA, and  Labor—had not established a standard structure. Among these,  officials from Agriculture, EPA, and Labor stated that they believe it is  difficult to standardize how programs define work products, in part,  because their programs conduct different types of work and have  different needs. While this presents a challenge, agencies could adopt  an approach similar to DOD’s and develop various standard work  structures based on the kinds of work being performed. Commerce  officials stated that they plan to establish a standard structure for  defining work products in the future, but have not yet determined a  time frame for completing this. Without establishing a standard  structure for defining work products, agencies will not be positioned to  ensure that they can effectively compare programs and collect and  share data among programs.
Process to collect and store cost-related data: Only one agency— DOD—fully addressed this policy component by establishing a  process to collect and store cost-related data. Specifically, the agency  has a central repository for collecting actual costs, software data, and  related business data, which serves as a resource to support cost  estimating across the agency. Among the seven agencies that have  not established a process for collecting and storing cost-related data,  Homeland Security’s policy assigns responsibility for doing so to the  central cost-estimating team; however, the team has not yet  implemented the process. Additionally, Veterans Affairs officials  stated that collecting such data would depend on the use of a  standard structure for defining work products, which they have not yet  put in place. Agriculture and Commerce officials stated that cost- estimating initiatives have not been a priority, although in one case a  component agency at Commerce—the United States Patent and  Trademark Office—took the initiative to establish a process to collect  and store cost-related data from past estimates. While this should  improve cost estimating at the component agency, without  establishing an agencywide process to collect and store cost-related  data, agencies will find it difficult to improve the data available to all  programs and to increase the efficiency of developing cost estimates.
Until the selected agencies address the identified weaknesses in their  cost-estimating policies, it will be difficult for them to make effective use of  program cost estimates for informed decision making, realistic budget  formation, and meaningful progress measurement.
section_title: Selected Agencies’ Program Cost Estimates Do Not Provide a Fully Reliable Basis for Program and Budget Decisions
paragraphs: A reliable cost estimate is critical to the success of any government  acquisition program, as it provides the basis for informed investment  decision making, realistic budget formulation and program resourcing,  and meaningful progress measurement. According to OMB, programs  must maintain current and well-documented cost estimates, and these  estimates must encompass the full life cycle of the programs. In addition,  our research has identified a number of best practices that provide a  basis for effective program cost estimating and should result in reliable  cost estimates that management can use for making informed decisions.  These practices can be organized into four characteristics— comprehensive, well-documented, accurate, and credible. These four  characteristics of a reliable cost estimate are explained in table 3.
While all 16 major acquisition programs we reviewed had developed cost  estimates and were using them to inform decision making, all but one of  the estimates were not fully reliable and did not provide a sound basis for  informed program and budget decisions. The 16 acquisition programs  had developed cost estimates and were using their estimates, in part, to  support program and budget decisions. For example, most programs  used their cost estimate as the basis for key program decisions, such as  approval to proceed to full production of a system. In addition, most  programs were using their estimates as an input to their annual budget  request process.
However, nearly all of these programs had estimates that did not fully  reflect important cost-estimating practices. Specifically, of the 16 case  study programs, only 1 fully met all four characteristics of a reliable cost  estimate, while the remaining 15 programs varied in the extent to which  they met the four characteristics. Table 4 identifies the 16 case study  programs and summarizes our results for these programs. Following the  table is a summary of the programs’ implementation of cost-estimating  practices. Additional details on the 16 case studies are provided in  appendix II.
Most programs partially implemented key practices needed to develop a  comprehensive cost estimate. Specifically, of the 16 programs, 1 fully  implemented the practices for establishing a comprehensive cost  estimate, 12 partially implemented the practices, and 3 did not implement  them.
DOD’s Consolidated Afloat Networks and Enterprise Services  program fully implemented key practices for developing a  comprehensive cost estimate. Specifically, the program’s cost  estimate included both the government and contractor costs for the  program over its full life cycle, from inception through design,  development, deployment, operation and maintenance, and  retirement of the program. Further, the cost estimate reflected the  current program and technical parameters, such as the acquisition  strategy and physical characteristics of the system. In addition, the  estimate clearly described how the various cost subelements were  summed to produce the amounts for each cost category, thereby  ensuring that all pertinent costs were included, and no costs were  double counted. Lastly, cost-influencing ground rules and  assumptions, such as the program’s schedule, labor rates, and  inflation indexes, were documented.
Twelve programs partially implemented key practices for developing a  comprehensive cost estimate. Most of these programs fully identified  cost-influencing ground rules and assumptions and included  government and contractor costs for portions of the program life cycle.  However, 10 of the 12 programs did not include the full costs for all  life-cycle phases and other important aspects of the program, such as  costs expected to be incurred by organizations outside of the  acquiring program (e.g., by other agency subcomponents), all costs  for operating and maintaining the system, and costs for the retirement  of the system. Without fully accounting for all past, present, and future  costs for every aspect of the program, regardless of funding source,  the programs’ estimated costs are likely understated and thereby  subject to underfunding and cost overruns.
In addition, 10 of the 12 programs did not provide evidence that their  cost estimates completely defined the program or reflected the current  program schedule by documenting a technical baseline description to  provide a common definition of the current program, including detailed  technical, program, and schedule descriptions of the system. For  example, in 2008, Homeland Security’s Rescue 21 program  documented the system’s technical characteristics, along with a high- level schedule for the program. Since 2008, however, certain  technical characteristics of the program had changed, such as  additional deployment sites needed to address communication service  gaps identified by local commanders at previously deployed locations.  In addition, the planned deployment dates for several locations of the  system had been delayed. As a result, the program’s cost estimate  did not fully reflect the current scope and schedule of the program.
Understanding the program—including the acquisition strategy,  technical definition, characteristics, system design features, and  technologies to be included—is critical to developing a reliable cost  estimate. Without these data, programs will not be able to identify the  technical and program parameters that bind the estimate.
Three programs did not implement key practices for developing a  comprehensive cost estimate in that their estimates did not  adequately (1) include all costs over the program’s full life cycle; (2)  completely define the program or the current schedule; (3) include a  detailed, product-oriented work breakdown structure; and (4)  document cost-influencing ground rules and assumptions. For  example, the cost estimate for Veterans Affairs’ Health Data  Repository program did not include sufficient detail to show that it  accounted for all phases of the program’s life cycle (e.g., design,  development, and deployment). Further, the estimate did not include  important technical baseline information, including the technical,  program, and schedule aspects of the system being estimated. Lastly,  the estimate only used high-level budget codes rather than a detailed,  product-oriented cost element structure to decompose the work, and  ground rules and assumptions (e.g., labor rates and base-year  dollars) were not documented. Without implementing key practices for  developing comprehensive cost estimates, management and  oversight organizations cannot be assured that a program’s estimate  is complete and accounts for all possible costs, thus increasing the  likelihood that the estimate is understated.
The majority of programs partially implemented key practices needed to  develop a well-documented cost estimate. Specifically, of the 16  programs, 1 fully implemented the practices for establishing a well- documented cost estimate, 10 partially implemented the practices, and 5  did not implement them.
DOD’s Consolidated Afloat Networks and Enterprise Services  program fully implemented key practices for developing a well- documented cost estimate. Specifically, the program’s cost estimate  captured in writing the source data used (e.g., historical data and  program documentation), the calculations performed and their results,  and the estimating methodology used to derive each cost element. In  addition, the program documented a technical baseline description  that included, among other things, the relationships with other  systems and planned performance parameters. Lastly, the cost  estimate was reviewed both by the Naval Center for Cost Analysis  and the Assistant Secretary of the Navy for Research, Development,  and Acquisition, which helped ensure a level of confidence in the  estimating process and the estimate produced.
Ten programs partially implemented key practices for developing a  well-documented cost estimate. Most of these programs included a  limited description of source data and methodologies used for  estimating costs, and documented management approval of the cost  estimate. However, 9 of the 10 programs did not include complete  documentation capturing source data used, the calculations  performed and their results, and the estimating methodology used to  derive each cost element. Among other things, the 9 programs had  weaknesses in one or more of the following areas: relying on expert  opinion but lacking historical data or other documentation to back up  the opinions; not documenting their estimate in a way that a cost  analyst unfamiliar with the program could understand what was done  and replicate it; and lacking supporting data that could be easily  updated to reflect actual costs or program changes. Without adequate  documentation to support the cost estimate, questions about the  approach or data used cannot be answered and the estimate may not  be useful for updates or information sharing.
In addition, 8 of the 10 programs did not provide management with  sufficient information about how the estimate was developed in order  to make an informed approval decision. For example, while the EPA’s  Financial System Modernization Project’s cost estimate was  approved, management was not provided information specific to how  the estimate was developed, including enough detail to show whether  it was accurate, complete, and high in quality. Because cost estimates  should be reviewed and accepted by management on the basis of  confidence in the estimating process and the estimate produced by  the process, it is imperative that management understand how the  estimate was developed, including the risks associated with the  underlying data and methods, in making a decision to approve a cost  estimate.
Five programs did not implement key practices for developing a well- documented cost estimate in that their estimates did not adequately  (1) include detailed documentation that described how the estimate  was derived, (2) capture the estimating process in such a way that the  estimate can be easily replicated and updated, (3) discuss the  technical baseline description, and (4) provide evidence that the  estimate was fully reviewed and accepted by management. In  particular, three of the five programs relied on their budget submission  documentation, known as the OMB Exhibit 300, as their life-cycle  cost estimate. The cost estimate information included in these  programs’ Exhibit 300 budget submissions was limited to the final  estimates in certain phases of the program’s life cycle, such as  planning, development, and operations and maintenance. Because a  well-documented estimate includes detailed documentation of the  source data, calculations and results, and explanations of why  particular methods and references were chosen, the programs that  relied on their Exhibit 300 budget submissions as their cost estimates  lacked the level of rigor and supporting documentation necessary for  a well-documented cost estimate. Without a well-documented  estimate, a program’s credibility may suffer because the  documentation cannot explain the rationale of the methodology or the  calculations, a convincing argument of the estimate’s validity cannot  be presented, and decision makers’ questions cannot be effectively  answered.
Most programs partially implemented or did not implement key practices  needed to develop an accurate cost estimate. Specifically, of the 16  programs, 2 fully implemented the practices for establishing an accurate  cost estimate, 8 partially implemented the practices, and 6 did not  implement them.
DOD’s Consolidated Afloat Networks and Enterprise Services and  Homeland Security’s Integrated Public Alert and Warning System  programs fully implemented key practices for developing an accurate  cost estimate. Specifically, the programs’ estimates were based on an  assessment of most likely costs, in part because a risk and  uncertainty analysis was conducted to determine where the programs’  estimates fell against the range of all possible costs. In addition, the  programs’ estimates were grounded in a historical record of cost  estimating and actual experiences from comparable programs. For  example, the cost estimate for the Integrated Public Alert and Warning  System program relied, in part, on actual costs already incurred by the  program as well as data from three comparable programs, including a  legacy disaster management system. Moreover, the programs’ cost  estimates were adjusted for inflation and updated regularly to reflect  material changes in the programs, such as when the schedule  changed.
Eight programs partially implemented key practices for developing an  accurate cost estimate. Most of these programs accounted for  inflation when projecting future costs. However, four of the eight  programs did not rely, or could not provide evidence of relying, on  historical costs and actual experiences from comparable programs.  For example, officials from Pension Benefit Guaranty Corporation’s  Benefit Administration program stated that they relied on historical  data along with expert opinion in projecting costs, but the officials did  not provide evidence of the data sources or how the historical data  were used. Because historical data can provide estimators with insight  into actual costs on similar programs—including any cost growth that  occurred in the original estimates—without documenting these data,  these programs lacked an effective means to challenge optimistic  assumptions and bring more realism to their estimates.
In addition, six of the eight programs did not provide evidence that  they had regularly updated their estimates to reflect material changes  in the programs so that they accurately reflected the current status.  For example, Justice’s Unified Financial Management System  program developed a cost estimate in 2009; however, according to  program documentation, program scope and projected costs have  since changed and, as a result, the 2009 estimate no longer reflects  the current program. Cost estimates that are not regularly updated  with current information can make it more difficult to analyze changes  in program costs, impede the collection of cost and technical data to  support future estimates, and may not provide decision makers with  accurate information for assessing alternative decisions.
Six programs did not implement key practices for developing an  accurate cost estimate in that their estimates were not adequately   (1) based on an assessment of most likely costs, (2) grounded in  historical data and actual experiences from comparable programs,   (3) adjusted for inflation, and (4) updated to ensure that they always  reflect the current status of the program. For example, the cost  estimate for Agriculture’s Public Health Information System was not  based on an assessment of most likely costs because a risk and  uncertainty analysis was not conducted to determine where the  estimate fell against the range of all possible costs. In addition, the  estimate was based primarily on the program team’s expertise, but  was not grounded in historical costs or actual experiences from  comparable programs. Lastly, the estimate was not adjusted for  inflation and lacked adequate detail to determine whether the  program’s latest updates to the cost estimate, completed in 2011,  accurately reflected the current status of the program. Without  implementing key practices for developing an accurate cost estimate,  a program’s estimate is more likely to be biased by optimism and  subject to cost overruns, and may not provide management and  oversight organizations with accurate information for making well- informed decisions.
The majority of programs did not implement all key practices needed to  develop a credible cost estimate. Specifically, of the 16 programs, 1 fully  implemented the practices for establishing a credible cost estimate, 5  partially implemented the practices, and 10 did not implement them.
DOD’s Consolidated Afloat Networks and Enterprise Services  program fully implemented key practices for developing a credible  cost estimate. Specifically, the program performed a complete  uncertainty analysis (i.e., both a sensitivity analysis and Monte Carlo  simulation) on the estimate. For example, in performing the  sensitivity analysis, the program identified a range of possible costs  based on varying key parameters, such as the technology refresh  cycle and procurement costs. In addition, the program performed  cross checks (using different estimating methods) on key cost drivers,  such as system installation costs. Lastly, an independent cost  estimate was conducted by the Naval Center for Cost Analysis and  the results were reconciled with the program’s cost estimate, which  increased the confidence in the credibility of the resulting estimate.
Five programs partially implemented key practices for developing a  credible cost estimate. Specifically, three of the five programs  performed aspects of a sensitivity analysis, such as varying one or  two assumptions to assess the impact on the estimate; however,  these programs did not perform other important components, such as  documenting the rationale for the changes to the assumptions or  assessing the full impact of the changes to the assumptions by  determining a range of possible costs. For example, the Pension  Benefit Guaranty Corporation’s Benefits Administration program  performed a sensitivity analysis by varying three program  assumptions, one of which was the contractor’s hourly rate, to assess  the impact on the cost estimate. However, the program did not  provide evidence to support why the adjusted hourly labor rate was  used nor apply a range of increases and decreases to the hourly labor  rate to determine the level of sensitivity of this assumption on the cost  estimate. A comprehensive sensitivity analysis that is well  documented and traceable can provide programs with a better  understanding of the variables that most affect the cost estimate and  assist in identifying the cost elements that represent the highest risk.
In addition, three of the five programs adjusted the cost estimate to  account for risk and uncertainty, but did not provide evidence to  support how costs were risk adjusted or determine the level of  confidence associated with the cost estimate.Homeland Security’s Integrated Public Alert and Warning System  program’s cost estimate did not include information on the risks  considered in its risk and uncertainty analysis or consider the  relationship between multiple cost elements when accounting for  risks. Without conducting an adequate risk and uncertainty analysis,  the cost estimate may be unrealistic because it does not fully reflect  the aggregate variability from such effects as schedule slippage,  mission changes, and proposed solutions not meeting users’ needs.
Ten programs did not implement key practices for developing a  credible cost estimate in that the programs did not adequately (1)  assess the uncertainty or bias surrounding data and assumptions by  conducting a sensitivity analysis, (2) determine the level of risk  associated with the estimate by performing a risk and uncertainty  analysis, (3) cross-check the estimates for key cost drivers, and (4)  commission an independent cost estimate to be conducted by a group  outside the acquiring organization to determine whether other  estimating methods would produce similar results. For example,  Agriculture’s Web-Based Supply Chain Management program did not  conduct a sensitivity analysis to better understand which variables  most affected the cost estimate, nor did the program conduct a risk  and uncertainty analysis to quantify the impact of risks on the  estimate. Further, cost drivers were not cross-checked to see if  different estimating methodologies produced similar results, and an  independent cost estimate was not conducted to independently  validate the results of the program’s estimate. Without implementing  key practices for developing a credible cost estimate, a program may  lack an understanding of the limitations associated with the cost  estimate and be unprepared to deal with unexpected contingencies.
subsections: section_title: Inadequate Implementation Was Largely Due to Weaknesses in Policy
paragraphs: The lack of reliable cost estimates across the investments exists in part  because of the weaknesses previously identified in the eight agencies’  cost-estimating policies. More specifically, program officials at five  agencies—Agriculture, Commerce, EPA, Justice, and Veterans Affairs— attributed weaknesses in their programs’ cost estimates, in part, to the  fact that agency policies did not require cost-estimating best practices— deficiencies which we also identified in these agencies’ policies. For  example, officials at Commerce’s Comprehensive Large Array-data  Stewardship System program stated that, when the program developed  its cost estimate, no agency guidance existed regarding the process to  follow in developing the estimate. In addition, officials at Veterans Affairs’  Veteran’s Benefits Management System program stated that they did not  perform a risk analysis on their cost estimate because agency guidance  on how such an analysis should be performed did not exist. In certain  cases, officials stated that program cost estimates were initially  developed prior to 2007, when a comprehensive federal resource for  cost-estimating best practices, such as GAO’s cost guide, did not exist.  However, all 16 programs included in our review have either developed  new estimates or updated previous estimates since 2007; nonetheless,  as previously mentioned, most of the selected agencies’ policies did not  fully address compliance with cost-estimating best practices, including the  five agencies mentioned above. If these agencies had updated their  policies, programs would have been more likely to follow a standard,  high-quality process in developing or updating their cost estimates.
Until important cost-estimating practices are fully implemented, the  likelihood that these programs will have to revise their current cost  estimates upward is increased. Collectively, 13 of the 16 programs have  already revised their original life-cycle cost estimates upward by almost  $5 billion due, in part, to weaknesses in program cost-estimating  practices (see app. III for details on changes in the programs’ cost  estimates over time). For example, in many cases, cost estimates had to  be revised upwards to reflect the incorporation of full costs for all life-cycle  phases (e.g., development or operations and maintenance), which had  not originally been included. This resulted, in some cases, in significant  increases to estimated life-cycle costs. Other reasons that programs cited  for revising their life-cycle cost estimates upward included changes to  program or system requirements, schedule delays, technology upgrades,  and system defects, among other things. Further, as previously  mentioned, 13 of the 16 case study programs still have cost estimates  that do not include the full costs for all life-cycle phases, which  significantly increases the risk that these programs’ cost estimates will  continue to be revised upward in the future.
Without reliable cost estimates, the 15 programs that did not fully meet  best practices will not have a sound basis for informed program decision  making, realistic budget formulation and program resourcing, and  meaningful progress measurement. Consequently, nearly all of these  programs’ cost estimates may continue to be understated and subject to  underfunding and cost overruns.
section_title: Conclusions
paragraphs: Given the enormous size of the federal government’s investment in IT, it  is critical that such investments are based on reliable estimates of  program costs. While all of the selected agencies have established  policies that at least partially addressed a requirement for programs to  develop full life-cycle cost estimates, most of the agencies’ policies have  significant weaknesses. With the exception of DOD, these policies omit or  lack sufficient guidance on several key components of a comprehensive  policy including, for example, management review and acceptance of  program cost estimates, the type of work structure needed to effectively  estimate costs, and training requirements for all relevant personnel.  Without comprehensive policies, agencies may not have a sound basis  for making decisions on how to most effectively manage their portfolios of  projects.
Most programs’ estimates at least partially reflected cost-estimating best  practices, such as documenting cost-influencing ground rules and  assumptions; however, with the exception of DOD’s Consolidated Afloat  Networks and Enterprise Services program, the programs we reviewed  had not established fully reliable cost estimates, increasing the likelihood  that the estimates are incomplete and do not account for all possible  costs. For example, without including costs for all phases of a program’s  life cycle and performing a comprehensive risk and uncertainty analysis, a  program’s estimated costs could be understated and subject to  underfunding and cost overruns, putting it at risk of being reduced in  scope or requiring additional funding to meet its objectives. Many of the  weaknesses found in these programs can be traced back to inadequate  agency cost-estimating policies. Without better estimates of acquisition  life-cycle costs, neither the programs nor the agencies have reliable  information for supporting program and budget decisions. Consequently,  the likelihood of cost overruns, missed deadlines, and performance  shortfalls is significantly increased.
section_title: Recommendations for Executive Action
paragraphs: To address weaknesses identified in agencies’ policies and practices for  cost estimating, we are making the following recommendations:  We recommend that the Secretaries of Agriculture, Commerce,  Homeland Security, Labor, and Veterans Affairs, the Attorney General,  and the Administrator of the Environmental Protection Agency direct  responsible officials to modify policies governing cost estimating to  ensure that they address the weaknesses that we identified.
We also recommend that the Secretaries of Agriculture, Commerce,  Homeland Security, Labor, and Veterans Affairs, the Attorney General,  the Administrator of the Environmental Protection Agency, and the  Director of the Pension Benefit Guaranty Corporation direct responsible  officials to update future life-cycle cost estimates of the system acquisition  programs discussed in this report using cost-estimating practices that  address the detailed weaknesses that we identified.
Lastly, although DOD fully addressed the components of an effective  cost-estimating policy, in order to address the weaknesses we identified  with a key system acquisition discussed in this report, we recommend  that the Secretary of Defense direct responsible officials to update future  life-cycle cost estimates of the Tactical Mission Command program using  cost-estimating practices that address the detailed weaknesses that we  identified.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided the selected eight agencies and the Pension Benefit  Guaranty Corporation with a draft of our report for review and comment. A  management analyst in the Department of Justice’s Internal Review and  Evaluation Office, Justice Management Division, responded orally that the  department had no comments. Six of the agencies and the Pension  Benefit Guaranty Corporation provided written comments, and the  Department of Labor provided oral and written comments. These  agencies generally agreed with our results and recommendations,  although EPA disagreed with our assessment of the cost-estimating  practices used for one of its programs. These agencies also provided  technical comments, which we incorporated in the report as appropriate.
The comments of the agencies and the corporation are summarized  below:
The U.S. Department of Agriculture’s Acting Chief Information Officer  stated that the department concurred with the content of the report.  Agriculture’s comments are reprinted in appendix IV.
The Acting Secretary of Commerce stated that the department fully  concurred with our findings and recommendations. Among other  things, the Acting Secretary described a number of ongoing actions to  address the weaknesses we identified, such as modifying  departmental policies governing cost estimating to include an  additional cost-estimating training course and cost-estimating training  requirements. In addition, the department stated that forthcoming  policy and guidance are intended to ensure that the cost estimates for  high-profile programs are comprehensive, accurate, credible, and  well-documented. Commerce’s comments are reprinted in appendix  V.
DOD’s Director of Cost Assessment and Program Evaluation stated  that the department partially concurred with our recommendation but  agreed with the criteria, methodology, and assessment of the DOD  programs. The director added, however, that there is no plan to  formally update the Tactical Mission Command life-cycle cost  estimate, as the program is in the system deployment phase of its  acquisition lifecycle. We recognize that the programs included in our  study are at varying stages of their acquisition life cycles and that  updates to their cost estimates may not be justified. Accordingly, our  recommendation to DOD is specific to only future life-cycle cost  estimates. In this regard, if any significant changes occur in the  program during deployment of the system that warrant an update to  the cost estimate, it will be important that the program uses best  practices that address the weaknesses we identified. DOD’s  comments are reprinted in appendix VI.
EPA’s Assistant Administrator of the Office of Solid Waste and  Emergency Response and its Assistant Administrator and Chief  Information Officer of the Office of Environmental Information stated,  in regard to our assessment of cost-estimating policies, that EPA  recognized that its policies did not require cost-estimating best  practices and that the agency will update its Systems Life Cycle  Management procedures accordingly. The officials acknowledged that  sound fiscal management practices should be followed in all aspects  of the agency’s information technology operations, including cost  estimating for the development of new systems.
In regard to our assessment of cost-estimating practices for two  system acquisition programs, EPA stated that it did not have any  comments on our assessment of the Financial System Modernization  Project; however, it did not believe our assessment accurately  reflected the cost-estimating practices employed for the development  of the Superfund Enterprise Management System. In particular, the  Office of Solid Waste and Emergency Response stated in its written  response and in technical comments that it believed it had met the  spirit and intent of the cost-estimating best practices in GAO’s cost  guide, even though the program may have used different processes  or documentation in order to do so. We recognize and agree that  organizations should tailor the use of the cost-estimating best  practices as appropriate based on, for example, the development  approach being used, and we took this factor into consideration during  our review of the 16 acquisition programs. However, we stand by our  assessment of the Superfund Enterprise Management System  program’s cost estimate on the basis of the weaknesses described in  appendix II of this report. In particular, as we discuss, the program’s  cost estimate lacked key supporting documentation, including costs  not documented at a sufficient level of detail; the lack of documented  source data, calculations, and methodologies used to develop the  estimate; and a lack of documentation on the source of and rationale  for the inflation factor used. In addition, the lack of detailed cost- estimate information precluded us from making the linkage between  the cost estimate and other important program documents, such as  the system’s technical baseline and schedule, in order to determine  whether the estimate reflects the current program and status.  Because rigorous documentation is essential for justifying how an  estimate was developed and for presenting a convincing argument for  an estimate’s validity, weaknesses in this area contributed  significantly to weaknesses across multiple best practices areas,  including the estimate’s comprehensiveness and accuracy. Further,  regarding the Office of Solid Waste and Emergency Response’s  comment that our cost-estimating guide was not published until 3  years after development of the Superfund Enterprise Management  System commenced, we disagree that this would preclude the  program from satisfying cost-estimating best practices. Specifically,  the program updated its cost estimate in 2011, 2 years after the  issuance of the GAO cost guide. At that time, the program could have  revised its cost estimate using available best practice guidance.
Lastly, we disagree that the draft report erroneously concluded that  the Superfund Enterprise Management System cost estimate  increased from $39.3 million to $62.0 million in just 2 years. In its  written response, the Office of Solid Waste and Emergency Response  stated that the revised cost estimate was a direct result of an increase  in the duration of operations and maintenance from fiscal year 2013  (in the $39.3 million estimate) to fiscal year 2017 (in the $62.0 million  estimate). However, according to documentation provided by the  Superfund Enterprise Management System program, the $39.3 million  estimate, which was completed in 2009, was based on a 10-year life  cycle (from fiscal year 2007 to fiscal year 2017) and included costs for  operations and maintenance through fiscal year 2017. Subsequently,  in 2011, the program revised its estimate to approximately $62.0  million, which was also based on a 10-year life cycle (from fiscal year  2007 to fiscal year 2017) and included operations and maintenance  costs through 2017. The revised estimate is an increase of about  $22.7 million over the initial estimate. According to program  documentation, this change in the cost estimate was primarily due to  the inclusion of additional operations and maintenance costs for data  and content storage and hosting for the fully integrated system  between fiscal year 2014 and fiscal year 2017, which were  erroneously omitted from the 2009 estimate. Based on these factors,  we maintain that our report reflects this information appropriately.  EPA’s comments are reprinted in appendix VII.
The Department of Homeland Security’s Director of the Departmental  GAO-Office of the Inspector General Liaison Office stated that the  department concurred with our recommendations. Among other  things, the department stated that its Office of Program Accountability  and Risk Management intends to develop a revised cost-estimating  policy that will further incorporate cost-estimating best practices, as  well as work to provide cost-estimating training to personnel on major  programs throughout the department. Homeland Security’s comments  are reprinted in appendix VIII.
In oral comments, the Administrative Officer in the Department of  Labor’s Office of the Assistant Secretary for Administration and  Management stated that the department generally agreed with our  recommendations. Further, in written comments, the Assistant  Secretary for Administration and Management stated that the  department, through several initiatives, such as its Post  Implementation Review process and training to IT managers, will  continue to improve upon its IT cost estimation. The department also  commented on certain findings in our draft report. In particular, the  Assistant Secretary stated that, given the department’s relatively small  IT portfolio, establishing a central, independent office dedicated to  cost estimating is not justified. We recognize that agency IT portfolios  vary in size; however, as noted in our report, agencies should  establish a central cost-estimating team commensurate with the size  of their agency, which could consist of a few resident experts instead  of a full independent office. Regarding our second recommendation,  according to the Assistant Secretary, the Occupational Safety and  Health Administration (OSHA) stated that it believes our assessment  of the credibility of the OSHA Information System program’s 2010 cost  estimate was too low and did not reflect additional information  provided in support of the program’s 2008 cost estimate. In our  assessment of the program’s 2010 estimate we acknowledge  evidence provided from the 2008 estimate; however, this evidence did  not adequately show that important practices for ensuring an  estimate’s credibility, including making adjustments to account for risk  and conducting a sensitivity analysis, were performed on the 2010  cost estimate. In addition, OSHA stated that an independent estimate  was conducted at the outset of the program by an industry-leading IT  consulting firm as recommended by the Department of Labor Office of  the Inspector General. While we acknowledge that this was done in  2005, the resulting estimate was the only one developed at the time  and thus was not used as a means of independent validation—i.e., to  determine whether multiple estimating methods produced similar  results. Therefore, the independent estimate conducted in 2005 would  not increase the credibility of the program’s current cost estimate.  Labor’s comments are reprinted in appendix IX.
The Director of the Pension Benefit Guaranty Corporation stated that  the corporation was pleased that its selected IT investment met at  least half, or a large portion, of our quality indicators for cost  estimating. Further, the Director stated that the corporation will  evaluate and improve future life-cycle cost estimates for the Benefit  Administration investment. The Pension Benefit Guaranty  Corporation’s comments are reprinted in appendix X.
The Chief of Staff for the Department of Veterans Affairs stated that  the department concurred with our recommendations and has efforts  under way to improve its cost-estimating capabilities. Among other  things, the Chief of Staff stated that the department plans to complete,  by the end of the first quarter of fiscal year 2013, an evaluation of the  utility of establishing an organizational function focused solely on  multiyear cost estimation. In addition, to improve cost-estimating  practices on its IT efforts, the department stated that it has additional  training planned in early fiscal year 2013. Veterans Affairs’ comments  are reprinted in appendix XI.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  Secretaries of Agriculture, Commerce, Defense, Homeland Security,  Labor, and Veterans Affairs; the Attorney General; the Administrator of  the Environmental Protection Agency; the Director of the Pension Benefit  Guaranty Corporation; and other interested parties. In addition, the report  will be available at no charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-6304 or by e-mail at melvinv@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs are on  the last page of this report. Key contributors to this report are listed in  appendix XII.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) assess the extent to which selected  departments and agencies have appropriately implemented cost- estimating policies and procedures, and (2) evaluate whether selected  information technology (IT) investments at these departments and  agencies have reliable cost estimates to support budget and program  decisions. For this review, we assessed eight federal agencies and 16  investments.
To select these agencies and investments, we relied on the Office of  Management and Budget’s Fiscal Year 2010 Exhibit 53 which, at the  time we made our selections, contained the most current and complete  data on 28 agencies’ planned IT spending. To ensure that we selected  agencies with varying levels of spending on IT, we sorted them into three  ranges based on their planned spending in fiscal year 2010: greater than or equal to $10 billion; greater than or equal to $1 billion but less than $10 billion; and greater than $0, but less than $1 billion.
The number of agencies selected from each range was based on the  relative number of IT investments within each range, and the specific  agencies selected were those with the highest amount of planned IT  spending in fiscal year 2010. Specifically, we selected one agency with  greater than $10 billion in planned IT spending,between $1 billion and $10 billion in planned spending, and two agencies  with less than $1 billion in planned spending. In doing so, we limited our  selections to those agencies at which we could identify two investments  that met our selection criteria for investments (see the following  paragraph for a discussion of our investment selection methodology).  These agencies were the Departments of Agriculture, Commerce,  Defense, Homeland Security, Justice, Labor, and Veterans Affairs, and  the Environmental Protection Agency. We excluded the Departments of  Education, Health and Human Services, and the Treasury, and the  General Services Administration from our selection, even though they  initially met our agency selection criteria, because we could not identify  two investments at these agencies that met our investment selection  criteria.
The Office of Management and Budget defines a major IT investment as a system or an  acquisition requiring special management attention because it has significant importance  to the mission or function of the agency, a component of the agency, or another  organization; is for financial management and obligates more than $500,000 annually; has  significant program or policy implications; has high executive visibility; has high  development, operating, or maintenance costs; is funded through other than direct  appropriations; or is defined as major by the agency’s capital planning and investment  control process. primarily an infrastructure investment, had a high percentage of steady- state spending versus development spending, had less than $5 million in  planned spending for fiscal year 2010, or were the subjects of recent or  ongoing GAO audit work.
To assess the extent to which selected agencies had appropriately  implemented cost-estimating policies and procedures, we analyzed  agency policies and guidance for cost estimating. Specifically, we  compared these policies and guidance documents to best practices  recognized within the federal government and private industry for cost  estimating. These best practices are contained in the GAO Cost Guide  and include, for example, establishing a clear requirement for cost  estimating, requiring management review and approval of cost estimates,  and requiring and enforcing training in cost estimating.  For each policy  component, we assessed it as either being not met—the agency did not  provide evidence that it addressed the policy component or provided  evidence that it minimally addressed the policy component; partially  met—the agency provided evidence that it addressed about half or a  large portion of the policy component; or fully met—the agency provided  evidence that it fully addressed the policy component. We also  interviewed key agency officials to obtain information on their ongoing  and future cost-estimating plans.
GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and  Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009). implemented the practices; partially met—the program provided evidence  that it implemented about half or a large portion of the practices; or fully  met—the program provided evidence that it fully implemented the  practices. We then summarized these assessments by characteristic. We  also interviewed program officials to obtain clarification on how cost- estimating practices are implemented and how the cost estimates are  used to support budget and program decisions.
We conducted this performance audit from July 2011 to July 2012 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Case Studies of Selected Programs’ Cost-Estimating Practices
paragraphs: We conducted case studies of 16 major system acquisition programs  (listed in table 5). For each of these programs, the remaining sections of  this appendix provide the following: a brief description of the program and  its life-cycle cost estimate, and an assessment of the program’s cost  estimate against the four characteristics of a reliable cost estimate— comprehensive, well-documented, accurate, and credible.
The key below defines “fully met,” “partially met,” and “not met” as  assessments of programs’ implementation of cost-estimating best  practices.
The program provided evidence that it fully implemented the cost- estimating practices.
The program provided evidence that it implemented about half or a  large portion of the cost-estimating practices.
The program did not provide evidence that it implemented the  practices or provided evidence that it only minimally implemented  the cost-estimating practices.
subsections: section_title: Public Health Information System
paragraphs: The Public Health Information System (PHIS) program is designed to  modernize the Food Safety and Inspection Service’s systems for ensuring  the safety of meat, poultry, and egg products. According to the agency,  the current systems environment includes multiple, disparate legacy  systems that do not effectively support agency operations. PHIS is  intended to replace these legacy systems with a single, web-based  system that addresses major business areas such as domestic  inspection, import inspection, and export inspection. The program intends  to implement functionality to support domestic inspection and import  inspection in 2012, and export inspection in 2013.
In 2007, PHIS was a development contract within the larger Public Health  Information Consolidation Projects investment. In 2011, after PHIS was  separated out as its own major investment and the program was  rebaselined, the PHIS program developed its own cost estimate of $82.3  million. This includes $71.4 million for development and $10.9 million for  operations and maintenance over a 12-year life cycle.
The PHIS program’s current cost estimate does not exhibit all of the  qualities of a reliable cost estimate. Specifically, while the estimate  partially reflects key practices for developing a comprehensive estimate, it  does not reflect key practices for developing a well-documented,  accurate, or credible estimate. Table 6 provides details on our  assessment of the PHIS program’s cost estimate.
section_title: Web-Based Supply Chain Management
paragraphs: The Web-Based Supply Chain Management (WBSCM) program is  designed to modernize the U.S. Department of Agriculture’s commodity  management operations, including the purchasing and distribution of  approximately $2.5 billion in food products for distribution to needy  recipients through domestic and foreign food programs. To accomplish  this, the program is replacing a legacy system with a web-based  commercial-off-the-shelf solution. In 2010, the program achieved full  operational capability. Ongoing efforts are focused on addressing a  significant number of system defects identified since deployment.
In 2003, WBSCM developed an initial cost estimate of $142.9 million.  This included $105.5 million for development and $37.4 million for  operations and maintenance over a 7-year life cycle. Subsequently, after  revising the estimate each year as part the program’s Office of  Management and Budget Exhibit 300 submission, in 2011, WBSCM  revised its cost estimate to $378.4 million, an increase of about $235.5  million over its initial cost estimate. This includes $104.9 million for  development and $273.5 million for operations and maintenance over a  18-year life cycle. These changes are due to, among other things,  incorporating additional years of operations and maintenance costs, a  recently planned system upgrade, and additional costs associated with  addressing system defects.
The WBSCM program’s current cost estimate does not exhibit any of the  qualities of a reliable cost estimate. Specifically, the estimate did not  reflect key practices for developing a comprehensive, well-documented,  accurate, or credible estimate. Table 7 provides details on our  assessment of the WBSCM program’s cost estimate.
section_title: Comprehensive Large Array-data Stewardship System
paragraphs: The Comprehensive Large Array-data Stewardship System (CLASS) is  designed to provide environmental data archiving and access. The  National Atmospheric and Oceanic Administration has been acquiring  these data for more than 30 years, from a variety of observing systems  throughout the agency and from a number of its partners. Currently, large  portions of the nation’s environmental data are stored and maintained in  disparate systems, with nonstandard archive and access capabilities.  With significant increases expected in both the data volume and the  number and sophistication of users over the next 15 years, CLASS is  intended to provide a standard, integrated solution for environmental data  archiving and access managed at the enterprise level. CLASS is currently  developing satellite data archiving and access capabilities for several  satellite programs, including the next generation of geostationary  satellites—known as the Geostationary Operational Environmental  Satellites-R Series, which are planned for launch beginning in 2015.
In 2006, the National Oceanic and Atmospheric Administration developed  the initial CLASS cost estimate of approximately $195.5 million. This  included $118.3 million for development and $77.2 million for operations  and maintenance over a 9-year life cycle. Subsequently, after revising the  cost estimate three times, in 2011, CLASS established its current cost  estimate of approximately $240.0 million, an increase of about $44.5  million over its initial cost estimate. This includes $176.0 million for  development and $64.0 million for operations and maintenance over a 17- year life cycle. CLASS program officials stated that the increase in the  estimate was due, in part, to additional data archiving requirements and  external program delays.
The CLASS program’s current cost estimate does not exhibit all qualities  of a reliable cost estimate. Specifically, while the estimate partially reflects  key practices for developing a comprehensive estimate, it does not reflect  key practices for developing a well-documented, accurate, or credible  estimate. Table 8 provides details on our assessment of CLASS  program’s cost estimate.
section_title: Patents End-to-End: Software Engineering
paragraphs: The Patents End-to-End: Software Engineering (PE2E-SE) program is  designed to provide a fully electronic patent application process.  According to the U.S. Patent and Trademark Office, the agency’s current  enterprise architecture is unable to meet current demands, and it has  relied on inefficient and outdated automated legacy systems that inhibit  the timely examination of patent applications. PE2E-SE intends to provide  an electronic filing and processing application that enables examiners to  meet current needs for the timely examination of patents. To accomplish  this, PE2E-SE is following an Agile development approach and intends  to implement a system using a text-based eXtensible Markup Language  standard that is flexible, scalable, and leverages modern technologies  with open standards. In fiscal year 2012, the program plans to build new  functionality, such as new text search tools, and deploy the system to a  limited set of examiners.
In 2010, PE2E-SE developed an initial cost estimate of $130.2 million.  This estimate only included costs for development, over a 3-year life  cycle. Subsequently, in 2012 and after multiple revisions, PE2E-SE  revised its cost estimate to $188.2 million, an increase of $58.0 million.  This includes $122.8 million for development and $65.4 million for  operations and maintenance over a 7-year life cycle. According to  program officials, these changes are primarily due to incorporating costs  for operations and maintenance into the estimate.
The PE2E-SE program’s current cost estimate does not exhibit all of the  qualities of a reliable cost estimate. Specifically, while the estimate  partially reflects key practices for developing a comprehensive, well- documented, and accurate estimate, it does not reflect key practices for  developing a credible estimate. Table 9 provides details on our  assessment of the PE2E-SE program’s cost estimate.
section_title: Consolidated Afloat Networks and Enterprise Services
paragraphs: The Consolidated Afloat Networks and Enterprise Services (CANES)  program is designed to consolidate and standardize the Department of  the Navy’s existing network infrastructures and services. According to the  department, the current network infrastructure is highly segmented and  includes several legacy environments that have created inefficiencies in  the management and support of shipboard networks. The CANES  program is intended to, among other things, reduce and eliminate existing  standalone afloat networks, provide a technology platform that can rapidly  adjust to changing warfighting requirements, and reduce the shipboard  hardware footprint. To accomplish this, the program will rely primarily on  commercial off-the-shelf software integrated with network infrastructure  hardware components. The CANES program is currently planning to  procure and conduct preinstallation activities of four limited fielding units  by the end of fiscal year 2012, and achieve full operational capability in  2023.
In 2010, the Navy’s Space and Naval Warfare Systems Command Cost  Analysis Division developed a program life-cycle cost estimate for the  CANES program, and the Naval Center for Cost Analysis developed an  independent cost estimate. Subsequently, these organizations worked  collaboratively to develop the program’s life-cycle cost estimate of  approximately $12.7 billion. This included approximately $4.0 billion for  development and approximately $8.8 billion for operations and  maintenance over a 23-year life cycle.
The CANES program’s cost estimate exhibits all of the qualities of a  reliable cost estimate. Specifically, the estimate reflects key practices for  developing a comprehensive, well-documented, accurate, and credible  estimate. Table 10 provides details on our assessment of the CANES  program’s cost estimate.
section_title: Tactical Mission Command
paragraphs: The Tactical Mission Command (TMC)battle command system for commanders and staffs from battalions  through the Army Service Component Commands. TMC is intended to  provide commanders and staff with improved battle command  capabilities, including increasing the speed and quality of command  decisions. In the near term, TMC is to address gaps in the Army’s tactical  battle command capability by delivering enhanced collaborative tools and  enterprise services, and, in the long term, TMC is to address rapid  improvements in technological capabilities through technology refresh. A  key component—known as the Command Post of the Future—is intended  to provide commanders and key staff with an executive-level decision  support capability enhanced with real-time collaborative tools. These  capabilities are expected to enhance situational awareness and support  an execution-focused battle command process. Currently, the program is  working to complete development of Command Post of the Future 7.0,  which the program plans to complete by the end of fiscal year 2012.   is designed to be the tactical  In 2008, the TMC program developed an initial cost estimate of  approximately $2.0 billion. This included approximately $1.9 billion for  development and $116.5 million for maintenance over a 14-year life cycle.  According to program officials, each subsequent year, in preparation for  the annual Weapons System Review, the program updated its life-cycle  cost estimate. In 2011 the TMC program established its current cost  estimate of approximately $2.7 billion, an increase of approximately $723  million over its initial cost estimate. This included approximately $2.0  billion for development and $650.7 million for operations and  maintenance over a 23-year life cycle. Program officials stated that the  increase in the estimate was due, in part, to changes in the life-cycle time  frames, fielding schedules, number of units planned for deployment, and  other software development changes.
The TMC program’s current cost estimate does not exhibit all qualities of  a reliable cost estimate. Specifically, while the estimate partially reflects  key practices for developing a comprehensive, well-documented, and  accurate estimate, it does not reflect key practices for developing a  credible estimate. Table 11 provides details on our assessment of TMC  program’s cost estimate.
section_title: Financial System Modernization Project
paragraphs: The Financial System Modernization Project (FSMP) replaced the  Environmental Protection Agency’s legacy core financial system. The  system is intended to address agency-identified shortcomings in its  previous financial systems, such as inconsistent data, limited system  interoperability, low system usability, and costly maintenance. FSMP  includes key functionality for performing cost and project management,  general ledger, payment management, and receivables management.  According to the agency, the system is intended to, among other things,  eliminate repetitive data entry, integrate legacy systems, and enable  agency staff to manage workflow among the Office of the Chief Financial  Officer and between other business lines (e.g., acquisitions and grants  management). The system was deployed in October 2011.
In 2005, the FSMP program developed an initial cost estimate of  approximately $163.2 million. This included $42.8 million for development  and $120.4 million for operations and maintenance over a 25-year life  cycle. After revising the cost estimate three times, in 2010 the program  established its current cost estimate of approximately $169.3 million, an  increase of approximately $6 million over its initial cost estimate. This  includes $103.7 million for development and $65.7 million for operations  and maintenance over a 15-year life cycle. Program officials stated that  the changes to the program’s life-cycle cost estimate are due, in part, to  changes in the Environmental Protection Agency’s policies and guidance,  such as using a 15-year program life cycle instead of the 25-year life  cycle used in the program’s original estimate. In addition, officials stated  that the FSMP program has undergone significant schedule and scope  changes, including delaying the system’s deployment date from 2008 to  2011 and reducing in the planned system components (e.g., budget  formulation)—all of which have impacted the program’s life-cycle cost  estimate.
The FSMP program’s current cost estimate does not exhibit all qualities of  a reliable cost estimate. Specifically, while the estimate partially reflects  key practices for developing a comprehensive, well-documented, and  accurate estimate, it does not reflect key practices for developing a  credible estimate. Table 12 provides details on our assessment of FSMP  program’s cost estimate.
section_title: Superfund Enterprise Management System
paragraphs: The Superfund Enterprise Management System (SEMS) is to replace  three legacy systems and multiple applications used to comply with the  Comprehensive Environmental Response, Compensation, and Liability  Act of 1980—commonly known as Superfund, which provides federal  authority to respond directly to releases or threatened releases of  hazardous substances that may endanger public health or the  environment. In addition, SEMS is designed to implement innovative  software tools that will allow for more efficient operation of the Superfund  program. Of the three legacy systems expected to be replaced by SEMS,  two have already been integrated, and the one remaining system is  expected to be fully integrated in 2013, at which time SEMS is planned to  achieve full operational capability.
In 2009, the SEMS program developed an initial cost estimate of  approximately $39.3 million. This included $20.8 million for development,  $14.7 million for operations and maintenance, and $3.8 million for  government personnel costs over a 10-year life cycle. Subsequently, in  2011, the program revised its estimate to approximately $62.0 million, an  increase of about $22.7 million over its initial cost estimate. This includes  $22.8 million for development and $39.2 million for operations and  maintenance over a 10-year life cycle. Program officials stated that the  increase in the estimate was primarily due to incorporating additional  operations and maintenance costs that were erroneously omitted from the  initial estimate.
The SEMS program’s current cost estimate does not exhibit all qualities  of a reliable cost estimate. Specifically, while the estimate partially reflects  key practices for developing a credible estimate, it does not reflect key  practices for developing a comprehensive, well-documented, or accurate  estimate. Table 13 provides details on our assessment of SEMS  program’s cost estimate.
section_title: Integrated Public Alert and Warning System
paragraphs: The Integrated Public Alert and Warning System (IPAWS) is designed to  provide a reliable, integrated, and comprehensive system to alert and  warn the American people before, during, and after disasters. To  accomplish this, the program is developing the capability to disseminate  national alerts to cellular phones and expanding the existing Emergency  Alert System to cover 90 percent of the American public. In 2011, IPAWS  established standards for alert messages, began cellular carrier testing,  and conducted a nationwide test of the expanded Emergency Alert  System capabilities. The program intends to deploy the cellular alerting  capability nationwide in 2012 and complete its expansion of the  Emergency Alert System in 2017.
In 2009, IPAWS developed its initial estimate of $259 million, which  included $252.1 million for development and $6.9 million for government  personnel costs, but did not include operations and maintenance costs. In  2011, the program revised its estimate to $311.4 million, an increase of  about $52.3 million. This includes $268.9 million for development and  $42.5 million for operations and maintenance over an 11-year life cycle.  According to program officials, the increase in the cost estimate is  primarily due to the inclusion of costs to operate and maintain the system  during development.
The IPAWS program’s current cost estimate does not exhibit all qualities  of a reliable cost estimate. Specifically, while the estimate fully reflects  key practices for developing an accurate estimate, it only partially reflects  key practices for developing a comprehensive, well-documented, and  credible estimate. Table 14 provides details on our assessment of IPAWS  program’s cost estimate.
section_title: Rescue 21
paragraphs: Rescue 21 is designed to modernize the U.S. Coast Guard’s maritime  search and rescue capability. According to the agency, the current  system—the National Distress and Response System, does not meet the  demands of the 21st century in that it does not provide complete  coverage of the continental United States, cannot receive distress calls  during certain transmissions, lacks interoperability with other government  agencies, and is supported by outdated equipment. Rescue 21 is  intended to provide a modernized maritime distress and response  communications system, with increased maritime homeland security  capabilities that encompass coastlines, navigable rivers, and waterways  in the continental United States, in addition to Hawaii, Guam, and Puerto  Rico. Rescue 21 is currently undergoing regional deployment, which is  planned to be completed in fiscal year 2017.
In 1999, the Rescue 21 program developed an initial cost estimate of  $250 million for acquisition of the system, but this estimate did not include  any costs for operations and maintenance of the system. Following three  rebaselines, in 2006 the Rescue 21 program revised the estimate to  $1.44 billion, an increase of approximately $1.19 billion over the initial  estimate. This included $730 million in development and $707 million in  operations and maintenance over a 16-year life cycle. According to  program documentation, these increases were due, in part, to  incorporating costs for the operation and maintenance of the system.  Subsequently, in 2008, the Rescue 21 program revised its cost estimate  again to $2.66 billion, an increase of approximately $1.22 billion over the  previous estimate, and approximately $2.41 billion over the initial cost  estimate. This includes $1.07 billion in development and $1.59 billion in  operations and maintenance over a 16-year life cycle. Program officials  stated that the most recent increase in the cost estimate was primarily  due to schedule delays, an extension of the program’s life cycle by 6  years based on an expected increase in the system’s useful life, and to  reflect more realistic estimates of future costs for ongoing system  technology refreshment.
The Rescue 21 program’s current cost estimate does not exhibit all  qualities of a reliable cost estimate. Specifically, the estimate partially  reflects key practices for developing a comprehensive, well-documented,  accurate, and credible estimate. Table 15 provides details on our  assessment of the Rescue 21 program’s cost estimate.
section_title: Next Generation Combined DNA Index System
paragraphs: Since 1998, the Combined DNA Index System (CODIS) has supported  the Federal Bureau of Investigation’s mission by assisting criminal  investigation and surveillance through DNA collection and examination  capabilities. CODIS is an automated DNA information processing and  telecommunications system that generates potential investigative leads in  cases where biological evidence is recovered. Among other things,  CODIS links crime scene evidence to other crimes and/or offenders,  which can identify serial offenders and/or potential suspects. CODIS  serves over 190 participating laboratories and 73 international  laboratories representing 38 countries. According to the Federal Bureau  of Investigation, the reliability and expandability of CODIS are critical to  the agency’s ability to effectively aid law enforcement investigations  through the use of biometrics, prompting the decision in 2006 to initiate a  modernization effort, referred to as Next Generation CODIS (NGCODIS).  In 2011, the program achieved full operational capability for CODIS 7.0, a  software release of NGCODIS, which included functionality for, among  other things, implementing a software solution to comply with European  Union legislation for DNA data exchange and maintaining DNA records of  arrested persons. Additional functionality is expected in the future;  however, all program development has been put on hold until the  necessary funding is approved.
In 2006, the CODIS program developed an initial cost estimate for  NGCODIS of $128.4 million. This included approximately $69.6 million for  development and $58.8 million for operations and maintenance over an  11-year life cycle. In 2009, the CODIS program developed an additional  cost estimate of $58.6 million to account for operations costs associated  with certain versions of NGCODIS. According to program officials, even  though the program estimated additional operations costs of $58.6  million, the program’s original cost estimate has increased by only $8.6  million because originally planned development work related to  incorporating advancements in DNA technology was delayed and the  costs associated with this work were removed from the cost estimate.
The CODIS program’s current cost estimate for NGCODIS does not  exhibit all qualities of a reliable cost estimate. Specifically, the estimate  partially reflects key practices for developing a comprehensive, well- documented, accurate, and credible estimate. Table 16 provides details  on our assessment of the NGCODIS cost estimate.
section_title: Unified Financial Management System
paragraphs: The Unified Financial Management System (UFMS) is to modernize the  Department of Justice’s financial management and procurement  operations. To accomplish this, UFMS is to replace four legacy core  accounting systems and multiple procurement systems with a commercial  off-the-shelf product. Ultimately, the system is expected to streamline and  standardize financial management and procurement processes and  procedures across the department’s component agencies. UFMS was  deployed to two component agencies—the Drug Enforcement  Administration and the Bureau of Alcohol, Tobacco, Firearms, and  Explosives—in fiscal years 2009 and 2011, respectively. The system is  planned to be deployed at other component agencies, including the U.S.  Marshals Service and the Federal Bureau of Investigation, between fiscal  years 2013 and 2014, and is expected to achieve full operational  capability in fiscal year 2014.
In 2002, the UFMS program developed an initial cost estimate of $357.2  million. This included approximately $196.4 million for development and  $160.8 million for maintenance over a 10-year life cycle. In 2009, the  UFMS program revised the estimate to $1.05 billion, an increase of  approximately $692.8 million. This included $469.5 million for  development and $581.6 million for operations and maintenance over a  20-year life cycle. Program officials stated that the increase in the  estimate was due to extending the program’s life cycle to include  additional years of development work and operations and maintenance of  the system. Subsequently, in 2011, the program revised its cost estimate  to $851.1 million, a decrease of approximately $198.9 million. This  estimate includes $419.5 million for development and $431.6 million for  operations and maintenance over a 20-year life cycle. Program officials  stated that the decrease in the cost estimate was due to a reduction in the  number of component agencies that planned to implement UFMS.  Specifically, UFMS removed the Federal Bureau of Prisons; Offices,  Boards and Divisions; and Office of Justice Programs from the system’s  deployment schedule in order to reduce the overall cost of the system.
The UFMS program’s current cost estimate does not exhibit all qualities  of a reliable cost estimate. Specifically, while the estimate partially reflects  key practices for developing a comprehensive, well-documented, and  accurate estimate, it does not reflect key practices for developing a  credible estimate. Table 17 provides details on our assessment of UFMS  program’s cost estimate.
section_title: OSHA Information System
paragraphs: The OSHA Information System (OIS) is a management tool consisting of  a suite of applications to reduce workplace fatalities, injuries, and  illnesses through enforcement, compliance assistance, and consultation.  According to the agency, OIS is intended to close performance gaps with  existing legacy systems resulting from irreplaceable legacy hardware and  software, the inability of legacy systems to fully support the agency’s  mission, and the absence of an application that supports key business  process areas, such as compliance assistance. Ultimately, OIS is  expected to provide a centralized web-based solution to be used by more  than 5,900 users at the federal and state level, including approximately  4,200 enforcement officers and 500 safety and health consultants. The  program completed development in 2011, and is working to complete  deployment of the system while addressing operations and maintenance  of the system, which the program plans to complete by the end of fiscal  year 2016.
In 2006, the OIS program developed an initial cost of $72.3 million. This  included $42.0 million for development and $30.3 million for operations  and maintenance over a 12-year life cycle. Subsequently, in 2010, the  OIS program revised its cost estimate to $91.3 million, an increase of  $19.0 million. This includes $63.3 million for development and  approximately $28.0 million for operations and maintenance over a 12- year life cycle. The OIS Program Manager stated that the increase in the  estimate was due, in part, to unanticipated changes to the OIS program’s  scope to better align with the Department of Labor’s strategic goals,  including securing safe and healthy workplaces, particularly in high-risk  industries. For example, according to this official, the agency’s  methodology for penalty calculations for violators of occupational safety  and health rules and regulations was modified, which required a redesign  of OIS in order to capture and accurately calculate these changes.
The OIS program’s current cost estimate does not exhibit all qualities of a  reliable cost estimate. Specifically, while the estimate partially reflects key  practices for developing a comprehensive, well-documented, and  accurate estimate, it does not reflect key practices for developing a  credible estimate. Table 19 provides details on our assessment of the  OIS program’s cost estimate.
section_title: PBGC Benefit Administration
paragraphs: The Pension Benefit Guaranty Corporation’s (PBGC) Benefit  Administration (BA) is a collection of IT systems and applications that  allows PBGC to administer and service the approximately 1.5 million  participants in over 4,300 plans that have been terminated and trusteed  as part of PBGC’s insurance program for single-employer pensions. The  BA program is intended to modernize and consolidate applications, retire  legacy systems, and address performance gaps. To do this, the BA  program is grouped into four projects—Customer Care, Document  Management, Case Management, and Benefit Management—in support  of paying accurate and timely payments and providing customer service  to participants. The BA program is expected to offer multiple self-service  channels to participants, reengineer benefit payment processes to  increase efficiency and productivity, and implement enhanced reporting  and document management systems. According to the agency, this  modernization effort is ultimately expected to increase customer  satisfaction, reduce operational costs, and improve data quality.  Currently, the program is scheduled to complete modernization and  decommission the remaining legacy applications in fiscal year 2015.
In 2007, the BA program developed an initial cost estimate of $186.9  million. This included $39.4 million for development and $147.5 million for  operations and maintenance over a 5-year life cycle. Subsequently, in  2010, BA revised its cost estimate to $155.9 million, a decrease of $31.0  million. This revised estimate includes $80.7 million for development and  approximately $75.2 million for operations and maintenance over a 10- year life cycle. Program officials stated that the decrease in the estimate  was due to changes to the program’s schedule milestones and changes  to the system’s architecture.
The BA program’s current cost estimate does not exhibit all qualities of a  reliable cost estimate. Specifically, the estimate partially reflects key  practices for developing a comprehensive, well-documented, accurate,  and credible estimate. Table 18 provides details on our assessment of the  BA program’s cost estimate.
section_title: Health Data Repository
paragraphs: The Health Data Repository (HDR) is intended to support the integration  of clinical data across the Department of Veterans Affairs and with  external healthcare systems such as that of the Department of Defense.  Specifically, the system is designed to provide a nationally accessible  repository of clinical data by accessing and making available data from  existing healthcare systems to support clinical and nonclinical decision- making for the care of the department’s patients. The system is being  developed using an Agile software development approach and, currently,  the program is working on software releases to improve the ability to  access data in VA’s legacy healthcare information system, and intends to  achieve full operating capability in 2017.
In 2001, the HDR program developed an initial cost estimate of $126.7  million. This included $105.9 million for development and $20.8 million for  operations and maintenance over a 7-year life cycle. According to  officials, the program revised its estimate each year during the budget  cycle; in 2011, HDR revised its cost estimate to $491.5 million, an  increase of approximately $364.8 million over its initial cost estimate. This  includes $281.9 million for development and $209.6 million for operations  and maintenance over a 17-year life cycle. Program officials stated that  the increase in the cost estimate was primarily due to the unplanned  deployment and operation of a prototype system for 5 years, and the  delay of the planned date for full operational capability from 2006 to 2017,  in part, because of changes in the program’s scope and technology  refreshes (i.e., equipment and storage capacity).
The HDR program’s current cost estimate does not exhibit any of the  qualities of a reliable cost estimate. Specifically, the estimate does not  reflect key practices for developing a comprehensive, well-documented,  accurate, and credible estimate. Table 20 provides details on our  assessment of HDR program’s cost estimate.
section_title: Veterans Benefits Management System
paragraphs: The Veterans Benefits Management System (VBMS) is intended to  provide a paperless claims processing system to support processing a  growing volume of claims—for example, the number of compensation and  pension claims submitted in a year passed 1 million for the first time in  2009. According to the department, due to the reliance on paper-based  processing, the current system is inefficient and costly, and carries risks  to veterans’ sensitive information. To address this, VBMS is designed to  provide veterans a secure and accessible means to obtain benefits,  reduce the claims backlog, implement standardized business practices,  and support the integration with other veteran-facing systems. The  program is currently developing functionality for compensation and  pension claims processing, and plans to add additional lines of business  in future years.
In 2008, the VBMS program developed an initial, high-level cost estimate  of $560.0 million for system development over a 5-year life cycle, which  did not include costs for operations and maintenance. Subsequently, after  revising the estimate each year as part the program’s Office of  Management and Budget Exhibit 300 submission, in 2011 VBMS revised  its cost estimate to $934.8 million, an increase of approximately $374.8  million over its initial estimate. This includes $433.7 million for  development and $501.1 million for operations and maintenance over an  11-year life cycle. Program officials stated that the increase in the  estimate was primarily due to incorporating costs associated with  operations and maintenance and effort spent on changing to an Agile  development approach.
The VBMS program’s current cost estimate does not exhibit all of the  qualities of a reliable cost estimate. Specifically, while the estimate  partially reflects key practices for developing a comprehensive and well- documented estimate, it does not reflect key practices for developing an  accurate and credible estimate. Table 21 provides details on our  assessment of the VBMS program’s cost estimate.
section_title: Appendix III: Original and Current Life-Cycle Cost Estimates for Case Study Programs
paragraphs: Collectively, 13 of the 16 case study programs have revised their cost  estimates upward by almost $5 billion. More specifically, the 13 programs  have experienced cost increases ranging from about $6 million to over $2  billion. For example, in many cases, cost estimates had to be revised  upwards to reflect the incorporation of full costs for all life-cycle phases  (e.g. development or operations and maintenance), which had not  originally been included. Other reasons that programs cited for revising  their life-cycle cost estimates upward included changes to program or  system requirements, schedule delays, technology upgrades, and system  defects, among other things. Among the remaining 3 programs, 1  program’s cost estimate had decreased, 1 had not changed, and 1 was  not applicable because the program only had a current cost estimate (see  table 22).
section_title: Appendix IV: Comments from the Department of Agriculture
section_title: Appendix V: Comments from the Department of Commerce
section_title: Appendix VI: Comments from the Department of Defense
section_title: Appendix VII: Comments from the Environmental Protection Agency
section_title: Appendix VIII: Comments from the Department of Homeland Security
section_title: Appendix IX: Comments from the Department of Labor
section_title: Appendix X: Comments from the Pension Benefit Guaranty Corporation
section_title: Appendix XI: Comments from the Department of Veterans Affairs
section_title: Appendix XII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, individuals making contributions to  this report included Eric Winter (Assistant Director), Mathew Bader, Carol  Cha, Jennifer Echard, J. Christopher Martin, Lee McCracken, Constantine  Papanastasiou, Karen Richey, Matthew Snyder, and Jonathan Ticehurst.