2020 Census: Census Bureau Improved the Quality of Its Cost Estimation but Additional Steps Are Needed to Ensure Reliability
Released: 2018-09-17T00:00:00

Letter

Background

A high-quality, reliable cost estimate is a key tool for budgeting, planning, and managing the 2020 Census. According to OMB, programs must maintain current and well-documented estimates of program costs, and these estimates must encompass the full life-cycle of the program. Among other things, OMB states that generating reliable program cost estimates is a critical function necessary to support OMB’s capital programming process. Without this capability, agencies are at risk of experiencing program cost overruns, missed deadlines, and performance shortfalls.

A reliable cost estimate is critical to the success of any federal government program. With the information from reliable estimates, managers can: make informed investment decisions, allocate program resources, measure program progress, proactively correct course when warranted, and ensure overall accountability for results.

To be considered reliable, a cost estimate must meet the criteria for each of the four characteristics outlined in our Cost Estimating and Assessment Guide. According to our analysis, a cost estimate is considered reliable if the overall assessment ratings for each of the four characteristics are substantially or fully met. If any of the characteristics are not met, minimally met, or partially met, then the cost estimate does not fully reflect the characteristics of a high-quality estimate and cannot be considered reliable. Those characteristics are:

Well-documented: An estimate is thoroughly documented, including source data and significance, clearly detailed calculations and results, and explanations of why particular methods and references were chosen. Data can be traced to their source documents.

Accurate: An estimate is unbiased, the work is not overly conservative or overly optimistic, and is based on an assessment of most likely costs. Few, if any, mathematical mistakes are present.

Credible: Any limitations of the analysis because of uncertainty or bias surrounding data or assumptions are discussed. Major assumptions are varied, and other outcomes are recomputed, to determine how sensitive they are to changes in the assumptions. Risk and uncertainty analysis is performed to determine the level of risk associated with the estimate. The estimate’s results are cross- checked, and an independent cost estimate (ICE) is conducted to see whether other estimation methods produce similar results.

Comprehensive: An estimate has enough detail to ensure that cost elements are neither omitted nor double counted. All cost-influencing ground rules and assumptions are detailed in the estimate’s documentation.

Past GAO Work on Census Cost Estimation

Meeting best practices outlined in our Cost Estimating and Assessment Guide for a reliable cost estimate has been a long-standing challenge for the Bureau. In 2008 we reported that the 2010 Census cost estimate was not reliable because it lacked documentation and was not comprehensive, accurate, or credible. For example, in our 2008 report on the Bureau’s cost estimation process, Bureau officials were unable to provide documentation that supported the assumptions for the initial 2001 life-cycle cost estimate as well as the updates. Consequently, we recommended that the Bureau establish guidance, policies, and procedures for estimating costs that would meet best practices criteria. The Bureau agreed with the recommendation and said at the time that it already had efforts underway to improve its future cost estimation methods and systems. Moreover, weaknesses in the life-cycle cost estimate were one reason we designated the 2010 Census a GAO High- Risk Area in 2008.

In 2012 we reported that, while the Bureau was taking steps to strengthen its life-cycle cost estimates, it had not yet established guidance for developing cost estimates. We recommended that the Bureau finalize its guidance, policies, and procedures for cost estimation in accordance with best practices. The Bureau agreed with the overall theme of the report but did not comment on the recommendation. During this review we found that the Bureau took steps to address this recommendation, which is discussed later in this report. Such guidance can help to institutionalize best practices and ensure consistent processes and operations for producing reliable estimates.

In a 2016 report we found that the October 2015 version of the Bureau’s life-cycle cost estimate for the 2020 Census was not reliable. Overall, we reported that the 2020 Census life-cycle cost estimate partially met two of the characteristics of a reliable cost estimate (comprehensive and accurate) and minimally met the other two (well-documented and credible). We recommended that the Bureau take specific steps to ensure its cost estimate meets the characteristics of a high-quality estimate. The Bureau agreed with this recommendation, and took steps to improve the reliability of its cost estimate, which we focus on later in this report. Consequently, an unreliable life-cycle cost estimate is one of the reasons we designated the 2020 Census a GAO High-Risk Area in 2017.

Development of the 2020 Cost Estimate

In October 2015, the Bureau estimated the cost of the 2020 Census to be $12.3 billion. According to the Bureau, the October 2015 version was the Bureau’s first attempt to model the life-cycle cost of its planned 2020 Census, in contrast to its earlier 2011 estimate, which the Bureau said was intended to produce an approximation of potential savings and to begin developing the methodology for producing decennial life-cycle cost estimates covering all phases of the decennial life cycle. To help control costs while maintaining accuracy, the Bureau introduced significant change to how it conducts the decennial census in 2020. Its planned innovations include reengineering how it builds its address list, improving self-response by encouraging the use of the Internet and telephone, using administrative records to reduce field work, and reengineering field operations using technology to reduce manual effort and improve productivity. In contrast to the estimated $12.3 billion in 2015, the 2020 Census would cost $17.8 billion in constant 2020 dollars if the Bureau repeated the 2010 Census design and methods, according to the Bureau’s estimates.

In October 2017, Commerce announced that it had updated the October 2015 life-cycle cost estimate, projecting the life-cycle cost of the 2020 Census to be $15.6 billion, an increase of over $3 billion (27 percent) over its 2015 estimate. (See figure 1.) In developing the 2017 version of the cost estimate, Bureau cost estimators identified cost inputs, their ranges for possible outcomes, and overall cost estimating relationships (i.e., logical or mathematical formulas, or both). To identify cost inputs and the ranges of potential outcomes, the Bureau worked with subject matter experts and used historical data to support assumptions and generate inputs. The Bureau’s cost estimation team used a software tool to generate the cost estimate.

Because cost estimates predict future program costs, uncertainty is always associated with them. For example, data from the past (such as fuel prices) may not always be relevant in the future. Risk and uncertainty refer to the fact that because a cost estimate is a forecast, there is always a chance that the actual cost will differ from the estimate. One way to determine whether a program is realistically budgeted is to perform an uncertainty analysis, so that the probability associated with achieving its point estimate can be determined, usually relying on simulations such as those of Monte Carlo methods. This can be particularly useful in portraying the uncertainty implications of various cost estimates.

Consistent with cost estimation practices outlined in our Cost Estimating and Assessment Guide, the estimate was compared with two independent cost estimates (ICE), developed by Commerce’s Office of Acquisition Management (OAM) and the Bureau’s Office of Cost Estimation, Analysis, and Assessment. The offices producing the ICEs and the cost estimate team worked together to examine the process each used, an effort known as the reconciliation process. Through this reconciliation, the Bureau identified areas where discrepancies existed and elements that could require additional review and possible improvement.

According to Bureau documentation the estimate will be updated as the program meets milestones and to reflect changes in technical or program assumptions. Figure 2 details the Bureau’s cost estimation process. OAM was involved extensively in the development of the 2017 estimate, an increased involvement compared to 2015, according to Bureau officials. OAM participated in regular review meetings throughout the development of the estimate and also developed an independent cost estimate, as shown in the figure below.

End-to-end system testing activities for the 2020 Census are currently underway in Providence, Rhode Island. According to the Bureau, information collected from the test, such as overall response rates and the use of administrative records to inform census records, will inform future versions of the life-cycle cost estimate. Some updates from the test will be incorporated into the next cost estimate, which will be available in the first quarter of the coming fiscal year.

Census Bureau Has Made Progress but Has Not Taken All the Steps Needed to Ensure the Reliability of 2020 Cost Estimate

Since our June 2016 report, in which we reviewed the Bureau’s 2015 version of the cost estimate, the Bureau has made significant progress. For example, the Bureau has put into place a work breakdown structure (WBS) that defines the work, products, activities, and resources necessary to accomplish the 2020 Census and is standardized for use in budget planning, operational planning, and cost estimation. However, the Bureau’s October 2017 cost estimate for the 2020 Census does not fully reflect characteristics of a high-quality estimate as described in our Cost Estimating and Assessment Guide and cannot be considered reliable.

Our Cost Estimating and Assessment Guide describes best practices for developing reliable cost estimates. For our reporting needs, we collapsed these best practices into four characteristics for sound cost estimating— comprehensive, well-documented, accurate, and credible—and identified specific best practices for each characteristic. To be considered reliable, an organization must meet or substantially meet each characteristic. Our review found the Bureau met or substantially met three out of the four characteristics of a reliable cost estimate, while it partially met one characteristic: well-documented. When compared to the October 2015 estimate, the 2017 estimate shows considerable improvement. (See figure 3 below.)

Well-Documented

Cost estimates are considered valid if they are well-documented to the point they can be easily repeated or updated and can be traced to original sources through auditing, according to best practices. The Bureau only partially met the criteria for well-documented, as set forth in our Cost Estimating and Assessment Guide. A cost estimate that does not fully meet the criteria for well-documented cannot be used by management to make informed and effective implementation decisions.

The well-documented characteristic comprises five best practices. The Bureau substantially met two out of five best practices (as shown in figure 4). First, the estimate describes in sufficient detail the calculations performed and the estimating methodology used to derive each element’s cost, and the cost estimate had been reviewed by management. Since cost estimates can inform key decisions and budget requests, it is vital that management review and understand how the estimate was developed, including risks associated with the underlying data and methods.

The cost estimate only partially met three best practices for the characteristic of being well-documented. In general, some documentation was missing, inconsistent, or difficult to understand. First, we found that source data did not always support the information described in the basis of estimate document or could not be found in the files provided for two of the Bureau’s largest field operations: Address Canvassing and Non- Response Follow-Up (NRFU). For example, the cost estimate documentation referred to actual data from the 2010 Census and information obtained from experts as sources for address canvassing rework rates. However, the folder source documents provided as support for the basis of estimate did not include this information. Next, in several cases, we could not replicate calculations, such as for mileage costs, using the description provided. Lastly, we found that some of the cost elements did not trace clearly to supporting spreadsheets and assumption documents.

Failure to document an estimate in enough detail makes it more difficult to replicate calculations, or to detect possible errors in the estimate; reduces transparency of the estimation process; and can undermine the ability to use the information to improve future cost estimates or even to reconcile the estimate with another independent cost estimate. The Bureau told us it would continue to make improvements to ensure the estimate is well- documented. For the estimate to be considered well-documented, the Bureau will need to address these issues.

Accurate

An accurate cost estimate supports measurement of program progress by providing unbiased and correct data, which can help management ensure accountability for scheduled results. We found the Bureau’s cost estimate substantially met the criteria for accuracy. As shown in figure 5, and in line with best practices outlined in our Cost Estimating and Assessment Guide, the estimate was not overly optimistic; appeared to be free of errors; was based on historical data or input from subject matter experts; and, according to Bureau officials, is updated regularly as information becomes available.

The Bureau can enhance the accuracy of their estimate by increasing the level of detail included in the documentation, such as detail on specific inflation indices used, and by monitoring actual costs against estimates. We identified areas for improvement, which, according to Bureau officials, will be addressed as part of its ongoing efforts. For example, while the basis of estimate document describes different inflation indexes, it was not clear exactly which indexes were applied to the various cost elements in the estimate. Also, evidence of how variances between estimated costs and actual expenses would be tracked over time was not available at the time of our analysis. Tools to track variance enable management to measure progress against planned outcomes. Bureau officials stated that they already have systems in place that can be adapted for tracking estimated and actual costs.

Credible

All estimates include a certain amount of informed judgment about the future. Assumptions made at the start of a program can turn out to be inaccurate. Credible cost estimates identify limitations due to uncertainty or bias surrounding data or assumptions, and control for these uncertainties by identifying and quantifying cost elements that represent the most risk. We found that the Bureau’s cost estimate substantially met the criteria for credible, as shown in figure 6 below.

The Bureau’s cost estimate clearly identifies risks and uncertainties, and describes approaches taken to mitigate them. In line with best practices outlined in our Cost Estimating and Assessment Guide, the Bureau did the following:

Sensitivity analysis. The Bureau conducted sensitivity analysis to identify possible changes to estimated costs for the 2020 Census based on varying major assumptions, parameters, and data inputs. For example, the Bureau calculated the likely cost implications for a range of possible response rates to identify a range of projected costs and to calculate appropriate reserves for risk. Bureau officials stated that they also identified the estimate input parameters that contributed the most to estimate uncertainty.

Risk and uncertainty analysis. A cost estimate is a forecast, and as such, there is always a chance that the actual cost will differ from the estimate. Uncertainty is the indefiniteness about the outcome of a situation. Uncertainty is assessed in cost estimate models to estimate the risk (or probability) that a specific funding level will be exceeded. We found the Bureau performed an uncertainty analysis on a portion of the estimate to determine whether estimated costs were realistic and to establish the probability of achieving projections outlined in the estimate. The Bureau used a combination of modeling based on Monte Carlo analysis and allocations of funding for risks. The Monte Carlo simulation was performed on a portion of the estimate to account for uncertainty around various operational parameters for which a range of outcomes was possible, including Internet response rates and the extent to which data collection issues might be resolved using administrative records. To account for the inherent uncertainty of assumptions included within the life-cycle cost estimate, the Bureau added funding to the cost estimate totaling approximately $292 million to account for risks based on the results of the Monte Carlo analysis.

For other risks, such as acquisition lead time and the possibility of delays in information technology (IT) development, contingency funding was added to the estimate to reflect the potential cost of resolving these issues, through use of a backup system or an alternative approach. These are described as “special risks” in the Bureau’s basis of estimate, and total approximately $171 million.

Based on additional sensitivity analysis, the Bureau added approximately $965 million to the cost estimate to reflect discrete risks outlined in the risk register as well as those associated with (1) variability in self-response rates, (2) the effect of fluctuations in the size and wage rate of the temporary workforce on the cost of field operations, and (3) the potential need to reduce the enumerator-to- manager staffing ratio in case expected efficiencies in field operations are not realized.

In addition to these provisions, the Secretary of Commerce added a contingency amount of about $1.2 billion to account for what the Bureau refers to as unknown-unknowns. Bureau documentation states that conducting a decennial census is an extremely complex, high-risk operation. In order to mitigate some of the risk, contingency funding must be available to initiate ad hoc activities necessary to overcome unforeseen issues. According to Bureau documentation these include such risks as natural disasters or cyber-attacks. The Bureau provides a description of how the general risk contingency is calculated. However, this description does not clearly link calculated amounts to the risks themselves.

In our June 2016 report we reported the Bureau had not properly accounted for risk and recommended the Bureau, in part; improve control over how risk and uncertainty are accounted for. We continue to believe the prior recommendation from our June 2016 report remains valid and should be addressed: that the Bureau properly account for risk in the 2020 Census cost estimate, among other things. As such, risks need to be linked to the $1.2 billion general risk contingency fund.

Independent cost estimate. According to best practices outlined in our Cost Estimating and Assessment Guide, an independent cost estimate should be performed to determine whether alternate estimate approaches produce similar results. The Bureau compared their estimate with two independent cost estimates, developed by Commerce’s Office of Acquisition Management and the Bureau’s Office of Cost Estimation and Assessment. As part of their process for finalizing the cost estimate, Bureau officials reconciled differences between the estimates in discussions with the two offices, resulting in more conservative assumptions by the Bureau around risk and uncertainty in both cases.

In addition to implementing our recommendation to properly account for risk, going forward, while the Bureau substantially met the credibility characteristic it will be important for them to also integrate regular cross-checks of methodology into their cost estimation process. In our analysis we observed that no specific cross-checks of cost methodology were performed. According to the Bureau, cross- checks were not performed because the Bureau considered the independent cost estimates as overall cross-checks on the reliability of their methodology and did not conduct additional cross-checks. The main purpose of cross-checking is to determine whether alternative methods for specific cost elements within the cost estimate could produce similar results. An independent cost estimate, though important for the credibility of an estimate, does not fulfill the same function as a targeted cross-check of individual elements.

Comprehensive

Comprehensive estimates have enough detail to ensure that cost elements are neither omitted nor double-counted, all cost-influencing assumptions are detailed in the estimate’s documentation, and a work breakdown structure is defined. Our analysis of the 2017 cost estimate demonstrates improvement over the 2015 cost estimate when the Bureau’s cost estimate only partially met the criteria for comprehensive.

We found the Bureau met or substantially met all four best practices for the comprehensive characteristic, as shown in figure 7. For example, all life-cycle costs are included in the estimate along with a complete description of the 2020 Census program and current schedule. We also found that the Bureau substantially met criteria for documenting cost influencing ground rules and assumptions.

A standardized WBS (as detailed in table 1) with supporting dictionary outlines the major work of the program and describes the activities and deliverables at the project level where costs are tracked. In 2016, the Bureau’s WBS did not contain sufficient detail and we found significant differences in the presentation of the work between sources. In 2017, based on our review of Bureau documentation and interviews with Bureau officials, we found that the WBS is standardized and cost elements are presented in detail. The WBS is a necessary program management tool because it provides a basic framework for a variety of related tasks like estimating costs, developing schedules, identifying resources, determining where risks may occur, and providing the means for measuring program status.

Although the Bureau’s updated life-cycle cost estimate reflects three of the four characteristics of a reliable cost estimate, we are not making any new recommendations to the Bureau in this report. We continue to believe the prior recommendation, made in 2016, remains relevant: that the Secretary of Commerce ensure that the Bureau finalizes the steps needed to fully meet the characteristics of a high-quality estimate, most notably in the well-documented area. The Bureau told us it has used our best practices for cost estimation to develop their cost estimate, and will focus on those best practices that require attention moving forward. Without a reliable cost estimate, the Bureau is limited in its ability to make informed decisions about program resources and to effectively measure progress against operational objectives.

Life-Cycle Cost Estimate Is Used by Management to Inform Decisions

OMB, in its guidance for preparing and executing agency budgets, cites that credible cost estimates are vital for sound management decision making and for any program or capital project to succeed. A well- developed cost estimate serves as a tool for program development and oversight, supporting management to make informed decisions.

According to the Bureau, the 2020 Census cost estimate is used as a management tool to guide decision making. Bureau officials stated the cost estimate is used to examine the cost impact of program changes. For example, the cost estimate served as the basis for the fiscal year 2019 funding request developed by the Bureau. The Bureau also said it used the 2020 Census life-cycle cost estimate to establish cost controls during budget formulation activities and to monitor spending levels for fiscal year 2019 activities. According to the Bureau, as detailed operational and implementation plans are defined, the 2020 Census life- cycle cost estimate has been and will continue to be used to support ongoing “what-if” analyses in determining the cost impacts of design decisions. Specifically, using the cost estimate to model the impact of changes on overall cost, the Bureau adjusted the scope of the Census Enterprise Data Collection and Processing (CEDCaP) operation.

Census Bureau Guidance to Develop Cost Estimates Meets Best Practices

The processes for developing and updating estimates are designed to inform management about program progress and the use of program resources, supporting cost-driven planning efforts and well-informed decision making. Our work has identified a number of best practices for use in developing guidance related to cost estimation and analysis that are the basis of effective program cost estimating and should result in reliable and valid cost estimates that management can use for making informed decisions.

In 2012 we reported that the Bureau had not yet established guidance for developing cost estimates. We recommended that the Bureau establish guidance, policies, and procedures for developing cost estimates that would meet best practice criteria. The Bureau agreed with the theme of the report but did not specifically agree with the recommendation. Moreover, in June 2016, we also reported that the cost estimation team did not record how and why it changed assumptions that were provided to it and did not document the sources of all data it used. The documentation of these changes to assumptions did not happen because the Bureau lacked written guidance and procedures for the cost estimation team to follow. During this review we found the Bureau has since established reliable guidance, processes, and policies for developing cost estimates and managing the cost estimation process. The following documents, shown in table 2, establish roles and responsibilities for oversight and approval of cost estimation processes, provide a detailed description of the steps taken to produce a high-quality cost estimate, and clarify the process for updating the cost estimate and associated documents over the life of a project.

The Decennial Census Program’s Cost Estimate and Analysis Process, which provides a detailed description of the steps taken to produce a high-quality estimate, is reliable as it met the criteria for 8 steps and substantially met the criteria for 4 steps of the 12 best steps outlined in our Cost Estimating and Assessment Guide, as shown below in figure 8. To avoid cost overruns and to support high performance, it will be important for the Bureau to abide by their newly developed policies and guidance and continue to use the life-cycle cost estimate as a management tool.

Increased Costs Are Driven by an Assumed Decrease in Self-Response Rates and Increases in Contingency Funds and IT Cost Categories

The 2017 life-cycle cost estimate includes significantly higher costs than those included in the 2015 estimate. In 2015, the Bureau estimated that they could conduct the operation at a cost of $12.3 billion in constant 2020 dollars. The Bureau’s latest cost estimate, announced in October 2017, reflects the same design, but at an expected cost of $15.6 billion. Figure 9 below shows the change in cost by WBS category for 2015 and 2017. The largest increases occurred in the Response, Managerial Contingency, and Census/Survey Engineering categories. Increased costs of $1.3 billion in the response category (costs related to collecting, maintaining, and processing survey response data) were in part due to reduced assumptions for self-response rates, leading to increases in the amount of data collected in the field, which is more costly to the Bureau. Contingency allocations increased overall from $1.35 billion in 2015 to $2.6 billion in 2017, as the Bureau gained a greater understanding of risks facing the 2020 Census. Increases of $838 million in the Census/Survey Engineering category were due mainly to the cost of an IT contract for integrating decennial survey systems that was not included in the 2015 cost estimate. Bureau officials attribute a decrease of $551 million in estimated costs for Program Management to changes in the categorization of costs associated with risks: In the 2017 version of the estimate, estimated costs related to program risks were allocated to their corresponding WBS element.

More generally, factors that contributed to cost fluctuations between the 2015 and 2017 cost estimates include: changes in assumptions for census operations, improved ability to anticipate and quantify risk, an overall increase in IT costs, and more defined contract requirements.

Changes in Assumptions

Several assumptions for the implementation of the 2020 Census have changed since the 2015 cost estimate. Some assumptions contributing to cost changes, mainly in the Response (related to collecting and processing response data) and Frame (the mapping and collecting addresses to frame enumeration activities) categories, include the following:

Self-response rates. Changes in assumptions for expected self- response rates contributed to increases in the response category, as the assumed rate decreased from 63.5 percent in 2015 to 60.5 percent in 2017, thereby increasing the anticipated percentage and associated cost of nonresponse follow-up. When the Bureau does not receive responses by mail, phone, or Internet, census enumerators visit each nonresponding household to obtain data. Thus, reduced self-response rates lead to increases in the amount of data collected in the field, which is more costly to the Bureau. Bureau officials attributed this decrease to a forecasted reduction in Internet response due to added authentication steps at log in and the elimination of the function allowing users to save their responses and return later to complete the survey.

Productivity rates. The productivity of enumerators collecting data for NRFU is another variable in the cost estimate that was updated, contributing to cost increases in the response category. Expected productivity rates for NRFU decreased from the 2015 estimate of 4 attempts per hour to 2.9. According to Bureau documentation, this more conservative estimate is based on historical data, rather than research and test data.

In-office address canvassing rates. The Bureau will not go door-to- door to conduct in-field address canvassing across the country to update address and map information for every housing unit, as it has in prior decennial censuses. Rather, some areas would only need a review of their address and map information using computer imagery and third-party data sources—what the Bureau calls “in-office” address canvassing procedures. However, in March 2017, citing budget uncertainty the Bureau decided to discontinue one of the phases of in-office review address canvassing for the 2020 Census. The cancellation of that phase of in-office review is expected to increase the number of housing units canvassed in-field by 5 percent (from 25 to 30 percent of all canvassed housing units). In-field canvassing is more labor intensive compared to in office procedures. The 2017 version of the cost estimate reflects this increase in workload for in-field address canvassing, though overall changes in estimated costs for the Frame category, of which Address Canvassing is a part, were minimal.

Staffing. Updated analysis resulted in changes to several staffing assumptions, which resulted in decreases across WBS categories. Changes included reduced pay rates for field data collection staff based on current labor market conditions and reductions in the length of staff engagement.

Anticipation of Risk

In general, contingency allocations increased overall from $1.35 billion in 2015 to $2.6 billion in 2017. This increase in contingency can be attributed, in part, to the Bureau gaining a clearer understanding of risk and uncertainty in the 2020 Census as it approaches. The Bureau developed some of its contingency based on proven risk management techniques, including Monte Carlo analysis and allocated funding for known risk scenarios. The 2017 estimate includes close to $1.4 billion in estimated costs for these risks, almost three times the amount included in the 2015 estimate. The basis of estimate contains detail on the various risks and the process for calculating the associated contingency. The 2017 version also includes a contingency amount of $1.2 billion for general risks, or unknown-unknowns, such as natural disasters and cyber-attacks.

Contingency amounts were reallocated within the WBS to more closely reflect the nature of the risk: Bureau officials attribute a decrease from the 2015 estimate of $551 million in estimated costs for program management to changes in the categorization of costs associated with risks. Officials stated that, in 2015, discrete program risks were previously consolidated as program management costs. In 2017, these discrete costs were reallocated to associate risks with the appropriate WBS element. For example, contingency amounts related to the likelihood of achieving a certain response rate previously included in the program management work breakdown category are now a part of the “response” work breakdown category.

Increased IT Costs

Increases in IT costs, totaling $1.59 billion, represented almost 50 percent of the total cost increase from 2015 to 2017. The total share of IT costs as a percentage of total census costs increased from 28 percent in 2015 to 32 percent in 2017, or from $3.41 billion to approximately $5 billion. Increases in IT costs are spread across seven cost categories. Figure 10 shows the IT and non-IT cost by WBS for the 2017 cost estimate. IT costs in infrastructure, response data, and census/survey WBSs account for the majority of the approximately $5 billion.

The Bureau’s October 2015 cost estimate included IT costs for, among other things, system engineering, test and evaluation, and infrastructure, as well as for a portion of the Census Enterprise Data Collection and Processing (CEDCaP) program. The 2017 estimated IT cost increases were due, in large part, to the Bureau (1) updating the cost estimate for CEDCaP; (2) including an estimate for technical integration services that contributed to increases in the Census and Survey Engineering category; and (3) updating costs related to other major contracts (such as mobile device as a service, field IT services, and payroll systems).

Contract Requirements

Bureau documents described an overall improvement in the Bureau’s ability to define and specify contract requirements. This resulted in updated estimates for several contracts, including for the Census Questionnaire Assistance (CQA) contract. Assumptions regarding call volume to the CQA were increased by 5 percent to account for expected response by phone after the elimination of the option to save Internet responses and return to complete the form later. The Bureau also cited updated cost data and the results of reconciliation with independent cost estimates as factors contributing to the increased costs of other major contracts, including for the procurement of data collection devices.

Agency Comments and Our Evaluation

The Secretary of Commerce provided comments on a draft of this report on August 2, 2018. The comments are reprinted in appendix II. The Department of Commerce generally agreed with our findings regarding the improvements the Census Bureau has made in its cost estimates. However, Commerce did not agree with our assessment that the Bureau’s 2017 lifecycle cost estimate is “not reliable.” Commerce noted that it had conducted two independent cost analyses and was satisfied that the cost estimate was reliable. The Bureau also provided technical comments that we incorporated, as appropriate.

We maintain that, to be considered reliable, a cost estimate must meet or substantially meet the criteria for each of the four characteristics outlined in our Cost Estimating and Assessment Guide. These characteristics are derived from measures consistently applied by cost estimating organizations throughout the federal government and industry and are considered best practices for the development of reliable cost estimates. Without a reliable cost estimate, the Bureau is limited in its ability to make informed decisions about program resources and to effectively measure progress against operational objectives. Thus, while the Bureau has made considerable progress in all four of the characteristics, it has only partially met the criteria for the characteristic of being well-documented. Until the Bureau meets or substantially meets the criteria for this characteristic, the cost estimate cannot be considered reliable.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of the report to the appropriate congressional committees, the Secretary of Commerce, the Under Secretary of Economic Affairs, the Acting Director of the U.S. Census Bureau, and other interested parties. In addition, this report is available at no charge on the GAO website at http://gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

The purpose of our review was to evaluate the reliability of the Census Bureau’s (Bureau) life-cycle cost estimate using our Cost Estimating and Assessment Guide. We reviewed (1) the extent to which the Bureau’s life-cycle cost estimate and associated guidance met our best practices for cost estimation using documentation and information obtained in discussions with the Bureau related to the 2020 life-cycle cost estimate and (2) compared the 2015 and 2017 life-cycle cost estimates to describe key drivers of cost growth. For both objectives we reviewed documentation from the Bureau on the 2020 life-cycle cost estimate and interviewed Bureau and Department of Commerce officials.

For the first objective, we relied on our Cost Estimating and Assessment Guide as criteria. Our cost specialists assessed measures consistently applied by cost-estimating organizations throughout the federal government and industry and considered best-practices for developing reliable cost estimates. We analyzed the cost estimating practices used by the Bureau against these best practices and evaluated them in four categories: comprehensive, well-documented, accurate, and credible.

Comprehensive. The cost estimate should include both government and contractor costs of the program over its full life-cycle, from inception of the program through design, development, deployment, and operation and maintenance to retirement of the program. It should also completely define the program, reflect the current schedule, and be technically reasonable. Comprehensive cost estimates should be structured in sufficient detail to ensure that cost elements are neither omitted nor double counted. Specifically, the cost estimate should be based on a product-oriented work breakdown structure (WBS) that allows a program to track cost and schedule by defined deliverables, such as hardware or software components. Finally, where information is limited and judgments are made, the cost estimate should document all cost-influencing assumptions.

Well-documented. A good cost estimate—while taking the form of a single number—is supported by detailed documentation that describes how it was derived and how the expected funding will be spent in order to achieve a given objective. Therefore, the documentation should capture in writing such things as the source data used, the calculations performed and their results, and the estimating methodology used to derive each WBS element’s cost.

Moreover, this information should be captured in such a way that the data used to derive the estimate can be traced back to, and verified against, their sources so that the estimate can be easily replicated and updated. The documentation should also discuss the technical baseline description and how the data were normalized. Finally, the documentation should include evidence that the cost estimate was reviewed and accepted by management.

Accurate. The cost estimate should provide for results that are unbiased, and it should not be overly conservative or optimistic. An estimate is accurate when it is based on an assessment of most likely costs; adjusted properly for inflation; and contains few, if any, minor mistakes. In addition, a cost estimate should be updated regularly to reflect significant changes in the program—such as when schedules or other assumptions change—and actual costs, so that it is always reflecting current status. During the update process, variances between planned and actual costs should be documented, explained, and reviewed. Among other things, the estimate should be grounded in a historical record of cost estimating and actual experiences on other comparable programs.

Credible. The cost estimate should discuss any limitations of the analysis because of uncertainty or biases surrounding data or assumptions. Major assumptions should be varied, and other outcomes recomputed to determine how sensitive they are to changes in the assumptions. Risk and uncertainty analysis should be performed to determine the level of risk associated with the estimate. Further, the estimate’s cost drivers should be cross-checked, and an independent cost estimate conducted by a group outside the acquiring organization should be developed to determine whether other estimating methods produce similar results.

If any of the characteristics are not met, minimally met, or partially met, then the cost estimate does not fully reflect the characteristics of a high- quality estimate and cannot be considered reliable.

We also analyzed the Bureau’s cost estimation and analysis guidance and evaluated them against a 12-step process outlined in our Cost Estimation and Assessment Guide. A high-quality cost estimating process integrates the following: 1. Define estimate’s purpose. 2. Develop estimating plan. 3. Define program characteristics. 4. Determine estimating structure. 5. Identify ground rules and assumptions. 6. Obtain data. 7. Develop point estimate and compare it to an independent cost estimate. 8. Conduct sensitivity analysis. 9. Conduct risk and uncertainty analysis. 10. Document the estimate. 11. Present estimate to management for approval. 12. Update the estimate to reflect actual costs and changes.

These 12 steps, when followed correctly, should result in reliable and valid cost estimates that management can use for making informed decisions. If any of the steps in the Bureau’s process do not meet, minimally meet, or partially meet the 12 steps, then the cost estimate guidance does not fully reflect best practices for developing a high-quality estimate and cannot be considered reliable.

Lastly, to describe key drivers of cost growth, we compared cost information included in the 2015 and 2017 cost estimates. We analyzed both summary and detailed cost information to assess key changes in totals overall, by WBS category, and by information technology (IT) vs. Non-IT costs. We used this analysis in conjunction with information received from the Bureau during interviews and through document transfers to describe overall changes in the cost estimate from 2015 to 2017.

We conducted this performance audit from December 2017 to August 2018 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Commerce

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Lisa Pearson (Assistant Director), Karen Cassidy (Analyst in Charge), Brian Bothwell, Jackie Chapin, Ann Czapiewski, Jason Lee, Ty Mitchell, Kayla Robinson, and Tim Wexler made significant contributions to this report.