2020 Census: Census Bureau Needs to Improve Its Life-Cycle Cost Estimating Process
Released: 2016-06-30T00:00:00

Letter

Background

Our work on the Bureau’s 2010 Census life-cycle cost estimate found that it was not reliable, because it lacked adequate documentation and was not comprehensive, accurate, or credible. For example, in our 2008 report on the Bureau’s cost estimation process, Bureau officials were unable to provide documentation that supported the assumptions for the initial 2001 life-cycle cost estimate as well as the updates. We reported that without improvements to the cost estimation process, the Bureau's ability to effectively manage operations would be hampered and Congress's ability to oversee the 2010 Census would be constrained. Consequently, we recommended that the Bureau establish guidance, policies, and procedures for conducting cost estimation that would meet best practices criteria. The Bureau agreed with the recommendation and said at the time that it already had efforts underway to improve its future cost estimation methods and systems. Moreover, weaknesses in the life-cycle cost estimate were one reason we designated the 2010 Census a GAO High Risk Area in 2008.

Bureau officials stated they have been working toward implementing the standards of our Cost Estimating and Assessment Guide since 2008. However, in a 2012 report, we found that while the Bureau was taking steps to strengthen its life-cycle cost estimates, it had not yet established guidance for developing cost estimates, and we recommended that the Bureau finalize its guidance, policies, and procedures for cost estimation in accordance with best practices. In its response to that report the Bureau agreed with the overall theme of the report, but did not directly comment on that recommendation, which remains open, as discussed later in this report.

Reengineering the 2020 Census

To help control costs while maintaining accuracy, the Census Bureau is introducing significant change to how it conducts the decennial census in 2020. Its planned innovations include reengineering how it builds its address list, improving self-response by encouraging the use of the Internet and telephone, using administrative records to reduce field work, and reengineering field operations using technology to reduce manual effort and improve productivity. The Bureau estimates that if it succeeds with these innovations it can conduct the 2020 Census for $12.5 billion in constant 2020 dollars. By contrast, the 2020 Census would cost $17.8 billion in constant 2020 dollars if the Bureau repeats the 2010 Census design and methods, according to the Bureau’s estimates.

Major innovation does not come without risks and in order to manage and mitigate them the Bureau has established a decennial risk management process governing two types of risks: project risks and program risks. The Bureau defines project risks as those that could jeopardize success of an individual project, such as the Bureau's operation to collect responses door-to-door from households that do not respond to Bureau mailings. The Bureau defines broader program risks as those that jeopardize the success of the 2020 Census program, typically spanning several years with many potential risk events over the period. These may have elevated from project level risks and include risks such as the possibility that external stakeholders do not support the planned design changes.

As early as 2011, the Bureau began developing preliminary cost estimates of the 2020 Census in order to approximate potential savings from its plans to reengineer the census, and, according to the Bureau, to begin developing the methodology for producing the decennial life-cycle cost estimates.

The Bureau’s October 2015 release of the latest cost estimate marked the transition from the “research” to “implementation” phases of the 2020 Census. According to the Bureau, this was the Bureau’s first attempt to model the life-cycle cost of its planned 2020 Census, in contrast to its earlier 2011 estimate which the Bureau said was intended to produce an approximation of potential savings and to begin developing the methodology for producing decennial life-cycle cost estimates covering all phases of the decennial life cycle (see fig. 1).

How the Bureau Produced Its Latest Cost Estimate

According to Bureau officials, they recently implemented a software upgrade they believe has better positioned them to deliver a quality cost estimate. They stated that they relied primarily on a cost estimation team of five individuals in the Decennial Programs Directorate to develop the 2020 Census life-cycle cost estimate. This team drew in part on subject matter specialists involved in the Bureau’s research and testing of its planned innovations in order to help develop a cost model and to obtain data to inform key assumptions within the model. The specialists typically provided the cost estimation team with data on each assumption using a three-point method, whereby the cost estimation team received a “minimum,” “value,” and “maximum” input for each assumption, representing whatever uncertainty or risks the provider had considered. Figure 2 shows the Bureau’s cost estimation process.

Because cost estimates predict future program costs, uncertainty is always associated with them. For example, data from the past (such as fuel prices) may not always be relevant in the future. Risk and uncertainty refer to the fact that because a cost estimate is a forecast, there is always a chance that the actual cost will differ from the estimate.

One way to determine whether a program is realistically budgeted is to perform an uncertainty analysis, so that the probability associated with achieving its point estimate can be determined, usually relying on simulations such as those of Monte Carlo methods. This can be particularly useful in portraying the uncertainty implications of various cost estimates.

Consistent with prevailing cost estimation practices, the Bureau’s Office of Cost Estimation, Analysis, and Assessment (OCEAA), in the Deputy Director’s Office (see fig. 3), began generating an independent cost estimate in fiscal year 2015, and shared it with the cost estimation team in April 2016. OCEAA and the cost estimate team worked together to examine the process they each used, an effort known as the reconciliation process. According to Bureau officials, reconciliation is a major step in the cost estimation process, and it may help identify areas of improvement for the Bureau’s cost estimation process. Officials also said that this reconciliation was to be shared with the Department of Commerce in June 2016 as part of its 2020 Census budget process.

The Bureau plans annual updates of its cost estimate, leaving three updates before the Bureau begins implementing its earliest operations in 2019 for the 2020 Census.

The Bureau’s October 2015 Cost Estimate Does Not Reflect Key Best Practices

Since our January 2012 report in which we reviewed the Bureau’s initial estimate of the total cost of the decennial census, the Bureau has taken significant steps to improve its capacity for cost estimating. For example, the Bureau established OCEAA as an enterprise-level cost estimation office and hired certified cost analysts. Despite this progress, the Bureau’s October 2015 cost estimate for the 2020 Census does not fully reflect characteristics of a high-quality estimate as described in our 2009 Cost Estimating and Assessment Guide and cannot be considered reliable.

A reliable cost estimate is critical to the success of any program. Such an estimate provides the basis for realistic budget formulation and program resourcing, meaningful progress measurement, proactive course correction when warranted, and accountability for results. According to the Office of Management and Budget (OMB), programs must maintain current and well-documented estimates of program costs, and these estimates must encompass the full life-cycle of the program. Without this capability, agencies are at risk of experiencing program cost overruns, missed deadlines, and performance shortfalls. The Cost Estimating and Assessment Guide describes best practices for the development of reliable cost estimates. For our reporting needs, in the cost guide we collapsed these best practices into four general characteristics for sound cost estimating—comprehensive, well-documented, accurate, and credible—and identified specific best practices for each characteristic.

To reflect these characteristics, an organization must meet or substantially meet each best practice. Our review found the Bureau partially or minimally met the cost estimating best practices. More specifically, it partially met two characteristics and minimally met two characteristics (see table 1).

According to our best practices, an estimate is comprehensive if it has enough detail to ensure that cost elements are neither omitted nor double-counted, all cost-influencing assumptions are detailed in the estimate’s documentation, and a single work breakdown structure (WBS) is defined and all WBS elements are described in a WBS dictionary. While Bureau officials were able to provide us with several documents that included projections and assumptions that were used in the cost estimate, we found the estimate to be partially comprehensive because it is unclear if all life-cycle costs are included in the estimate or if the cost estimate completely defines the program. Additionally, we found the Bureau had four versions of WBSs with inconsistencies among them. For example, the Bureau’s standard WBS, operational plan WBS, decennial budget WBS, and the life-cycle cost estimate WBS contained differing numbers of major areas (e.g., program management and response data) and we could not determine how all the different areas fit together.

Bureau officials stated that they have worked to create an agency-wide standard WBS and that the different versions of the WBS they provided to us were similar. Yet the Bureau did not provide evidence that demonstrated how the WBS related to each other. These issues reduce the reliability of the Bureau’s current life-cycle cost estimate, because as a result, we could not determine if all life-cycle costs are included in the cost estimate, and decision makers and others cannot know whether the total estimate fully accounts for all costs. However, the Bureau’s use of WBSs is an improvement over what we found in 2008 when the Bureau did not have a WBS in place at all.

In addition, about $3 billion, or one-quarter, of the total cost of the census is represented by cost breakouts that are not in the Bureau’s WBS. These comprise dollar amounts for the annual 2020 Census enacted or requested budget line for fiscal years 2012 through 2017 and their out year estimates for 2021 through 2023. Bureau officials justified using such aggregate budget figures by describing much of these years’ costs as staff cost at headquarters and as relatively fixed in nature—basic requirements to conduct a decennial census. We reported on the limitations associated with relying on aggregate budget numbers instead of detailed actual costs to support cost estimation in 2008 and on the limitations of isolating cost drivers (i.e., cost of specific field operations) within overly broad categories of cost in 2012. Although the Bureau generally agreed with the findings in both reports, it maintained that relying on appropriations figures within cost estimates was appropriate and expressed belief that the weaknesses we identified would not affect the Bureau’s ability to control future costs. However, according to best practices, the description of the costs should be tied to the WBS so that the Bureau or others can measure actual variances on specific cost elements from estimates, identify possible cost drivers, and inform life- cycle cost estimation activity for future censuses.

Well-Documented

Cost estimates are considered valid if they are well-documented to the point they can be easily repeated or updated and can be traced to original sources through auditing, according to best practices. Rigorous documentation also increases an estimate’s reliability and helps support an organization’s decision making. The documentation should explicitly identify the primary methods, calculations, results, rationales or assumptions, and sources of the data used to generate each cost element.

One reason why our overall assessment is low is because the estimate is not well-documented. While the Bureau provided some documentation of supporting data, it did not describe how the source data were incorporated. Additionally, while officials provided documentation to show the estimating approach used for the majority of the cost elements and the estimate results, it did not show the estimate calculations.

Moreover, the majority of methodologies used to develop the cost estimate relied on expert opinion instead of more quantitative methods of estimating. While expert opinion can be valuable in the absence of other data, according to best practices it should be used sparingly because of its subjectivity, potential to introduce bias, and lack of supporting documentation.

Similar to our findings, a contractor hired by the Bureau to assess the Bureau’s cost estimate identified the Bureau’s primary estimation weakness as a lack of formal, stand-alone documentation. The contractor’s January 2016 assessment noted that cost estimate documentation was available to the cost estimate team, but not contained in a formal document accessible to outside analysts. The contractor also noted that the cost estimate team acknowledged the lack of a documented cost estimating process.

While Bureau officials discussed with us how they believe they implemented best practices in producing the Bureau’s 2020 life-cycle cost estimate, those efforts have generally not been well-documented. Best practices state that thorough documentation is essential for validating and defending a cost estimate. A poorly documented estimate does not provide convincing support for the estimate’s validity and fails to answer decision makers’ and oversight groups’ questions.

Failure to document an estimate in enough detail for someone unfamiliar with the program to recreate or update the estimate makes it more difficult to detect possible errors in the estimate, reduces transparency of the estimation process, and can undermine the ability to use the information to improve future cost estimates or even to reconcile the estimate with another independent cost estimate. Bureau officials acknowledged throughout our review the importance of documenting the cost estimate and stated that facing resource constraints, they had prioritized their research and testing efforts, as well as completing the 2020 Census Operational Plan with the associated cost estimate, over documenting the cost estimate.

In April 2016, Bureau officials provided us with a summary of a documentation strategy they plan to have implemented in summer 2016. That strategy includes a cost estimation plan and schedule for future updates, a framework describing the fundamental basis of the estimate, and a comprehensive list of artifacts and other documents to be created. This strategy indicates that much progress might be made during the coming year to improve the Bureau’s documentation; however, it contains insufficient details to assess the extent to which that progress may help the Bureau meet documentation best practices.

Accurate

According to best practices, an estimate that is accurate is unbiased, is not overly conservative or overly optimistic, and is based on an assessment of most likely costs. Few, if any, mathematical mistakes are present and those that are present are minor. We found the estimate partially met best practices for this characteristic. For example, Bureau officials said that they only applied risk and uncertainty analysis to the portion of the 2020 Census estimate for the years 2018 to 2020. Officials said they focused on cost estimating methodologies and data collection and normalization for that period because the majority of costs, roughly 80 percent of the total estimate, occur in those 3 years. However, it appears the balance of the estimate was not adjusted for any specific level of confidence, so we cannot determine the confidence level of the total life-cycle estimate. Therefore, we cannot determine if the estimate is unbiased, overly conservative, or overly optimistic.

Additionally, we could not independently verify the calculations the Bureau used within its cost model to ensure there were no major mathematical mistakes. While the Bureau’s new cost estimation software upgrade contains the Bureau’s calculations within it, access to the software is limited, restricting the accessibility of these calculations or related notes. In one example, Bureau officials provided us with the inflation rates they used, but we could not verify the application of the rates because the documentation provided did not include the calculations.

Credible

Credible cost estimates clearly identify limitations due to uncertainty or bias surrounding the data or assumptions, according to best practices. Major assumptions should be varied and other outcomes should be recomputed to determine how sensitive outcomes are to changes in the assumptions. In addition, a risk and uncertainty analysis should be performed to determine the level of risk associated with the estimate. Finally, the results of the estimate should be cross-checked and an independent cost estimate should be performed to determine whether alternative estimate views produce similar results.

We found the estimate minimally met best practices for this characteristic. Bureau officials acknowledged that while a contractor they hired performed cross-checks of selected cost elements, overall cross-checks were not used. Also, although OCEAA began generating an independent cost estimate in fiscal year 2015 and shared it with the cost estimation team in April 2016, it was not made available to us at the time of this review. Bureau officials said they conducted a sensitivity analysis in order to identify cost drivers, but had not documented it by the time of our review. Additionally the Bureau carried out its risk and uncertainty analysis only for a portion of costs in fiscal years 2018 to 2020, telling us it scoped it narrowly by design to those 3 years when most of the census costs—and predominantly variable costs—occur. We found that the Bureau’s risk and uncertainty analysis (modeled costs) covered $4.6 billion, only about 37 percent of the $12.5 billion total estimated life-cycle cost, and less than one-half of the total estimated future cost of the census, which would include fiscal years 2017 to 2023 (see figure 4). Based on its risk and uncertainty analysis, the Bureau included a contingency of about $500 million to establish a total estimated value that it believed will equal or exceed the true cost with 80 percent certainty. testified on our concerns about the management and progress of the CEDCaP program that could contribute to cost overruns.

Other costs excluded from the modeling were costs of advertising ($347 million), various operations such as coverage improvement ($204 million) and in-office canvassing ($22 million), non-CEDCaP information technology systems ($221 million), as well as nearly all decennial costs for fiscal years 2021 through 2023 ($877 million). Bureau officials stated that much of the omitted costs are for fixed contracts or Bureau headquarters staff that are not susceptible to cost-related risks; however, these costs also include uncertain cost drivers, such as CEDCaP and other systems, which could have been included in the uncertainty analyses.

The Bureau used management discretion to determine how much contingency to add on top of the remaining costs. An additional 10 percent was added for fiscal years 2018 through 2020, for a total additional contingency of $829 million. However, officials were not able to justify the 10 percent factor and there was no Bureau documentation justifying the additional contingency. Because the Bureau only carried out its uncertainty analysis on a portion of the cost estimate, we cannot determine if it fully identified the level of risk associated with the estimate. Nor can we validate the Bureau’s reported confidence level of the total life-cycle cost estimate or how it relates to the Bureau’s total contingency.

We found the Bureau had little planning information among its documents supporting its cost estimate. Early fundamental planning and guidance documents such as general policies and procedures for cost estimation— in contrast to polished final process descriptions that might be produced later—can contribute to consistent control over the process used to develop a cost estimate and help ensure that desired standards and practices are implemented. Internal controls for the federal government state that management should design control activities to achieve objectives such as the development of a reliable cost estimate. These internal controls could take many forms, such as an operational plan, guidance on specific steps, and job aids for staff involved in the process. Internal controls would help the Bureau ensure continuity of operations across turnover in staff during the decennial life cycle, ensure that its cost estimation process follows best practices, and help it meet its objective of a reliable cost estimate. We recommended the Bureau take these steps to put guidance, policies, and procedures in place in our 2008 and 2012 reports on the Bureau’s cost estimation process. While the Bureau has not yet implemented the recommendation, we continue to believe that it is valid in order to ensure the Bureau improves the reliability of its cost estimate.

The Bureau Could Demonstrate That About Half of Its Key Assumptions Were Supported by Field Tests, Prior Studies, or Other Research

Cost estimates typically include a number of unknowns because they are based on future events. To account for these unknowns, cost estimators include various assumptions that are often built with limited information. These assumptions represent a set of judgments about past, present, or future conditions. Best cost estimation practices state that assumptions should be realistic and valid, meaning that historical data should back them up to minimize uncertainty and risk. Analysts must ensure that assumptions are not arbitrary, and that they are founded on expert judgments rendered by experienced program and technical personnel. Further, well-supported assumptions should include documentation of an assumption’s source.

Recognizing that not all assumptions are equal in their impact on cost, we worked with the Bureau’s cost team to identify key Bureau assumptions that were included within the Bureau’s model for its life-cycle cost estimate. We identified 41 key assumptions: examples included staffing ratios, pay rates, and anticipated production rates for the major field operations Address Canvassing and Nonresponse Follow-up (NRFU). Not all of these assumptions are likely to have the same impact on the cost model or are as directly testable in the field. According to the Bureau, it prioritized field testing for the most sensitive or high-impact assumptions, based on the availability of resources, and relied on subject matter expertise for assumptions that are more difficult to test or that affect relatively low-cost operations.

The Bureau’s model required each of these 41 assumptions to be associated with three inputs: a minimum, value, and maximum. The cost team told us that many of the assumptions were developed by subject matter specialists who provided them to the cost team. In other cases, the cost team said they developed the assumptions themselves.

We requested the source for each of the 41 key assumptions so we could determine if the key assumptions were based on historical data or expert judgments rendered by experienced program personnel. Figure 5 shows the sources of support for the assumptions we could determine.

Of the 41 key assumptions, Bureau officials provided evidence that 18 were supported by field tests, prior studies, other research, or a combination of these sources, as shown in the following examples:

Assumptions related to the use of administrative records were derived from the results of the 2015 Census Test.

NRFU assumptions relied on recent test results from the 2014 and 2015 Census site tests.

Address canvassing productivity assumptions drew on operational results from the 2010 Census.

By contrast, in our 2008 assessment, we had difficulty linking any assumptions to specific evidentiary support.

The other 23 assumptions did not have evidence to which the assumptions could be fully traced, though the cost estimation team provided verbal explanations for the assumptions. When we interviewed the subject matter specialists who developed the assumptions, they provided us with the research and information they used to develop their assumptions. However, for 10 assumptions, data provided by subject matter specialists did not include minimum and maximum inputs or had been changed by the cost estimation team. The cost estimation team did not record how and why it changed assumptions that were provided to it, how the range was determined if specialists had not provided a range, or specific sources of support for assumptions that were said to be 2010 actual census numbers, 2010 assessments, or management decisions.

According to Standards for Internal Control in the Federal Government, management should design control activities to achieve objectives. However, the Bureau had no guidance for how the cost team was supposed to handle information provided to it. In particular, the cost estimation team had no clear guidance for determining when to adjust or augment information provided to it. Nor did the team have procedures for recording its adjustments or their justification. When we found that data in the cost model differed from what subject matter specialists provided, we were unable to determine why it differed and, in some cases, neither could the cost estimation team. Lack of a record of how and why changes were made introduces uncertainty as to the origin and credibility of the numbers in the cost estimate. Additionally, a lack of information on these changes makes it difficult to determine the significance or impact of the changes.

According to the cost estimation team, six of the assumptions were based on management decisions, such as staffing ratios for field operations, but the cost team could not provide documentation of these decisions. The cost team said seven assumptions were based on “2010 actual” data and related assessments or from previously developed preliminary cost estimates. For example, they said they used operational data (such as 2010 NRFU pay rates) and assessments of the 2010 Census (such as the 2010 Address Canvassing Operational Assessment) to develop assumptions. We tried to independently verify these assumptions but could not.

We asked the cost estimation team for the source or citation to the actual historical data or assessments that the team may have relied on, but it could not provide them. In several cases the team instead directed us to the Bureau’s 2011 estimate. Documentation of that earlier estimate also referred to the 2010 Census (and assessments) as a primary source, but did not provide specific citations or traceable references. Without support for these assumptions we cannot determine if they are in fact realistic and valid or if they are based on historical data.

The Bureau Identified a Broad Range of Risks, but Did Not Clearly Account for Them in the Cost Estimate

The Bureau Identified a Broad Range of Risks Potentially Affecting 2020 Census Cost

The Bureau identified 158 different project-level risks for the 2020 Census. For example, these risks include delays in the acquisition of imagery needed to help build the Bureau’s national address list, and schedule delays in the delivery of information technology systems. The Bureau ranked its project-level risks on a scale of potential impact in five different areas, including cost. About 35 percent of the risks were identified as having a potential impact of at least “3” on a 5-point scale, equivalent to a potential cost impact of greater than or equal to 5 percent. (Bureau guidance does not specify against what cost element this impact rating potentially applies.) Of the 158 project-level risks, Bureau officials flagged 55 as having a significant potential impact on cost (see figure 6).

A review of our prior work and Department of Commerce Inspector General reporting on the decennial census, as well as our interviews with National Academies of Science staff selected for their involvement in study panels of the 2020 Census, did not identify any risks not already included in the Bureau’s list of risks. The Bureau designed and documented risk identification processes. For example, it has a risk management plan laying out roles, responsibilities, and processes and staff receive training on it. Project teams working on implementation of the census design, and research teams preceding them, had requirements to identify and report on risks potentially affecting their areas. Additionally, the Decennial Directorate has guidance for generating ideas about risks through brainstorming. This guidance was designed to help ensure inclusivity.

A broad list of identified risks can help provide a solid basis for assessing and mitigating the susceptibility of project implementation to uncertain conditions facing the decennial program. This can help the Bureau make informed decisions to help control cost.

Improving Controls Could Help the Bureau More Fully Account for Risks in Its Future Cost Estimation

Standards for Internal Controls in the Federal Government call for management to document in its policies or day-to-day procedures the responsibilities needed to attain its objectives. Clear guidance on what risks are being accounted for and how those are communicated through the cost estimation process will help ensure that resulting cost estimates reflect the best information available. One of the objectives of a robust cost estimation process is to determine the uncertainty of the cost estimate with an understanding of how much of that uncertainty is introduced by significant risks. This includes knowing what risks, if any, have been accounted for as part of any supporting sensitivity analysis or later management evaluation of risk and uncertainty.

However, we found the Bureau had no guidance, policies, or procedures in place on how to account for risk, develop boundaries around key assumptions, or determine contingencies. Furthermore, subject matter specialists did not receive training or other guidance on what risks, if any, they were to consider as part of their support of the cost estimation process; how to incorporate, document, or communicate what they had accounted for; or how they would account for uncertainty within the assumptions. In addition, specialists used a variety of methods to account for risk and uncertainty in the cost assumptions they were responsible for, but there was no standard methodology for doing this. For example, one specialist reported providing the calculated standard deviations resulting from the supporting statistical analysis to the cost estimation team, assuming that the core estimation team would set ranges around the provided value according to whatever methodology the team used. Another reported setting the submitted parameter “conservatively” and described wanting to “include a buffer” or contingency so that the cost in that area was not estimated too low. Varying treatments of uncertainty, contingencies, or risk can make it difficult to determine later what risk has been fully accounted for already within cost estimates.

Some of the subject matter specialists we spoke with had contributed directly to the identification of various risks within the Bureau’s risk management process, and they said that they had some of those in mind when determining ranges for assumptions they provided. However, none had reviewed the range of risks covered in the 2020 Census program or project risk registers when doing so or had communicated any risks that they may have considered to the cost estimation team. None reported having received guidance or direction on what risk lists, if any, to consider.

The cost estimation team said that it was aware of the risk registers, but had not consulted them. It also reported not having examined specific risks directly for accounting in the cost model. The cost estimation team also said that it was unaware whether or what risk subject matter specialists may have considered; what the basis for the range between minimum, value, and maximum inputs provided to it on each assumption may have been, or to what extent some specialists had already built contingency for general uncertainty into the data the specialist provided. The absence of procedures for the cost estimation team and subject matter specialists could have led to the over-specification of some uncertainty around the cost estimates we observed, or to missing the potential impact of some key risks.

The Bureau had mechanisms in place to make its identified risks known within the agency. For example, the Bureau’s Office of Risk Management and Program Evaluation told us that the Bureau made available to Bureau project managers—including some of the subject matter specialists—an online “risk dashboard” presenting risks documented across agency projects. This is intended to function as a communication tool, increasing risk visibility across project teams. Yet according to Bureau officials, the Bureau relied on its certified cost estimators to account for risks and did not have requirements that they specifically consider these tools or the risk registers.

Improved control over which risks and uncertainty are accounted for in cost estimates will better position Bureau managers to know which risks among those the Bureau has separately identified have or have not already contributed to the Bureau’s measured uncertainty in its cost estimate, and to know how much resources, if any, may thus be worth allocating to mitigate specific risks in order to control cost. Controls that can help with this accounting may include the implementation of processes or methods, for example institutionalized in the form of clearly documented guidance for those involved. In the absence of such accounting, management cannot be confident that (1) the contingency it selected is adequate to cover significant risks in its risk registers, or (2) the selected contingency is not overly conservative, which ties up funding that could be allocated to other projects.

Conclusions

In order for the Bureau to improve its ability to control the cost of the 2020 Census, it will be critical for it to have better control over its cost estimation process. According to Bureau officials, an example of the steps they have taken to improve their cost estimation capabilities is the recent implementation of a software upgrade they believe has better positioned them to deliver a quality cost estimate. While the Bureau has taken significant steps toward improving its capacity to produce reliable cost estimates, those efforts have not yet resulted in a reliable decennial cost estimate. Among the four broad characteristics of a reliable cost estimate—none of which the Bureau fully met—the Bureau has reported it is focusing its attention on improving the documentation of the cost estimate, in order to help improve other characteristics as well. While poor documentation has affected our ability to assess the reliability of the Bureau’s cost estimate’s other characteristics, we believe the problems we observed relate to an absence of internal control procedures over the cost estimation process, which in turn have resulted in the weakness in documentation.

In June 2008 we recommended the Bureau establish guidance, policies, and procedures for conducting cost estimation that would meet best practices criteria. In response, the Bureau agreed with the recommendation and said at the time that it already had efforts underway to improve its future cost estimation methods and systems. Eight years later, the absence of guidance to control the cost estimation process persists. Investment in the planning documents to help control and support cost estimation early in the estimation cycle, such as with an operational plan, guidance on key steps and process flows, assignment of responsibilities, and job aids for staff can help institutionalize practices and ensure that otherwise disparate parties in the process operate consistently. We continue to believe that establishing guidance, policies and procedures could help the Bureau incorporate the four characteristics of reliable cost estimates into future updates to the 2020 life-cycle cost estimate. Further, taking steps to ensure its cost estimate is reliable would help improve decision making, budget formulation, progress measurement, course correction when warranted, and accountability for results. Following the specific steps laid out in the four characteristics discussed in this report would position the Bureau to produce a reliable cost estimate for the 2020 Census.

In addition, we found that the cost estimation team did not record how and why it changed assumptions that were provided to it or traceable sources of all data it used. The Bureau lacked written guidance and procedures for the cost estimation team to follow. Clear guidance on when information for cost assumptions can and should be changed as well as the procedures for documenting such changes and traceable sources for information being used can reduce uncertainty on where data are coming from and improve their credibility.

Finally, the Bureau had risk management processes for identifying a broad range of risks that could affect the cost of the 2020 Census. Yet this institutional awareness of risk was not fully leveraged in the Bureau’s cost estimation process. As a result, the Bureau is unable to determine with confidence what risks the Bureau with its $12.5 billion cost estimate is prepared to mitigate or address. The Bureau determined that it is “80 percent” confident that its cost estimate will cover the actual cost of the 2020 Census, but given the inability to know what risk is accounted for, we do not see how the Bureau can be that sure.

The Department of Commerce will soon release its independent cost estimate as well as results of its reconciliation with the 2020 Census program estimate as part of its budget formulation process. Whether the independent estimate is higher or lower than the $12.5 billion estimate, the concerns we have raised about management attention to cost estimation remain. Improving control over how risk and uncertainty are accounted for and communicated with the Bureau’s decennial cost estimation process, such as by implementing and institutionalizing processes or methods with clear guidance, will improve Bureau and congressional confidence that the Bureau’s budgeted contingencies are at appropriate levels.

Recommendations for Executive Action

We recommend that the Secretary of Commerce and Under Secretary for Economic Affairs direct the Census Bureau to take the following actions: 1. To help ensure the Bureau produces a reliable cost estimate for the 2020 Census, take the following steps to meet the characteristics of a high-quality estimate:

Comprehensive—among other practices, ensure the estimate includes all life-cycle costs and documents all cost-influencing assumptions.

Well-documented—among other practices, ensure that its planned documentation plan captures the source data used; contains the calculations performed and the estimating methodologies used for each element; and describes step by step how the estimate was developed.

Accurate—among other practices, ensure the estimating technique for each cost element is used appropriately and that variances between planned and actual cost are documented, explained, and reviewed.

Credible—among other practices, ensure the estimate includes a sensitivity analysis, major cost elements are cross-checked to see whether results are similar, and an independent cost estimate is conducted to determine whether other estimating methods produce similar results. 2. To further ensure the credibility of data used in cost estimation, establish clear guidance on when information for cost assumptions can and should be changed as well as the procedures for documenting such changes and traceable sources for information being used. 3. To ensure Bureau and congressional confidence that the Bureau’s budgeted contingencies are at appropriate levels, improve control over how risk and uncertainty are accounted for and communicated with the Bureau’s decennial cost estimation process, such as by implementing and institutionalizing processes or methods for doing so with clear guidance.

Agency Comments and Our Evaluation

We provided a draft of this report to the Secretary of the Department of Commerce for comment. In written comments, reproduced in appendix II, the Department of Commerce agreed with our recommendations. It also provided additional context that we incorporated, as appropriate.

The Department of Commerce also noted that while it fully recognizes the Census Bureau can further improve its process under the Cost Estimating and Assessment Guide as well as the Standards for Internal Control in the Federal Government, it stands behind the quantitative integrity of the current life-cycle cost estimates for the 2020 Census.

We maintain that the process used to develop a cost estimate is key to its quantitative integrity. Unless and until the Bureau develops a cost estimate that fully reflects the characteristics of a high-quality estimate such as being comprehensive, well-documented, accurate, and credible, the estimate itself cannot be considered reliable.

We are sending copies of report to the Secretary of Commerce, the Counselor to the Secretary with Delegated Duties of the Undersecretary of Commerce for Economic Affairs, the Director of the U.S. Census Bureau, and interested congressional committees. The report also will be available at no charge on GAO's website at http://www.gao.gov.

If you have any questions about this report please contact me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. The GAO staff that made major contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

The purpose of our review was to evaluate the reliability of the Census Bureau’s (Bureau) life-cycle cost estimate using our Cost Estimating and Assessment Guide (GAO-09-3SP, or GAO Cost Guide). We reviewed (1) the extent to which the Bureau's life-cycle cost estimate met our best practices for cost estimation; (2) the extent to which the Bureau's key cost assumptions were supported by field tests, prior studies, and other evidence-based analysis; and (3) the extent to which the Bureau has identified and accounted for key risks facing the 2020 Census within its risk and uncertainty analyses of its life-cycle cost estimate. For all objectives, we reviewed documentation from the Bureau on the 2020 life- cycle cost estimate.

For the first objective, we relied on the cost guide as criteria. For the cost guide, our cost specialists assessed measures consistently applied by cost-estimating organizations throughout the federal government and industry and considered best-practices for the development of reliable cost-estimates. We analyzed the cost estimating practices used by the Bureau against these best practices and evaluated them in four categories: comprehensive, well-documented, accurate, and credible.

Comprehensive: The cost estimate should include both government and contractor costs of the program over its full life-cycle, from inception of the program through design, development, deployment, and operation and maintenance to retirement of the program. It should also completely define the program, reflect the current schedule, and be technically reasonable. Comprehensive cost estimates should be structured in sufficient detail to ensure that cost elements are neither omitted nor double-counted. Specifically, the cost estimate should be based on a product-oriented work breakdown structure (WBS) that allows a program to track cost and schedule by defined deliverables, such as hardware or software components. Finally, where information is limited and judgments are made, the cost estimate should document all cost-influencing assumptions.

Well-documented: A good cost estimate—while taking the form of a single number—is supported by detailed documentation that describes how it was derived and how the expected funding will be spent in order to achieve a given objective. Therefore, the documentation should capture in writing such things as the source data used, the calculations performed and their results, and the estimating methodology used to derive each WBS element’s cost. Moreover, this information should be captured in such a way that the data used to derive the estimate can be traced back to and verified against their sources so that the estimate can be easily replicated and updated. The documentation should also discuss the technical baseline description and how the data were normalized. Finally, the documentation should include evidence that the cost estimate was reviewed and accepted by management.

Accurate: The cost estimate should provide for results that are unbiased, and it should not be overly conservative or optimistic. An estimate is accurate when it is based on an assessment of most likely costs, adjusted properly for inflation, and contains few, if any, minor mistakes. In addition, a cost estimate should be updated regularly to reflect significant changes in the program—such as when schedules or other assumptions change—and actual costs, so that it is always reflecting current status. During the update process, variances between planned and actual costs should be documented, explained, and reviewed. Among other things, the estimate should be grounded in a historical record of cost estimating and actual experiences on other comparable programs.

Credible: The cost estimate should discuss any limitations of the analysis because of uncertainty or biases surrounding data or assumptions. Major assumptions should be varied, and other outcomes recomputed to determine how sensitive they are to changes in the assumptions. Risk and uncertainty analysis should be performed to determine the level of risk associated with the estimate. Further, the estimate’s cost drivers should be cross-checked, and an independent cost estimate conducted by a group outside the acquiring organization should be developed to determine whether other estimating methods produce similar results.

If any of the characteristics are not met, minimally met, or partially met, then the cost estimate does not fully reflect the characteristics of a high- quality estimate and cannot be considered reliable.

For the second objective we inventoried cost assumptions documented by the Bureau in support of its October 2015 life-cycle cost estimate and identified those associated with major changes from the Bureau's historical census design and with significant cost-saving potential. Additionally, we assessed the reliability of key assumptions by determining to what extent they are based on prior Bureau experience and testing, such as related results of the 2014 and 2015 Census Tests and other historical support.

Finally, for the third objective, we analyzed the Bureau's project and program risk registers and leveraged prior GAO work in this area to determine the range of risks and the adequacy of the Bureau's uncertainty analysis. We relied on our cost assessment guide and our Standards for Internal Control in the Federal Government as criteria.

We conducted this performance audit from November 2015 to June 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

GAO, Standards for Internal Control in the Federal Government. GAO-14-704G (Washington, D.C.: Sept. 10, 2014).

Appendix II: Comments from the Department of Commerce

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Ty Mitchell, Assistant Director; Brian Bothwell; Brett Caloia; Robert Gebhart; Jason Lee; Andrea Levine; Donna Miller; Cynthia Saunders; and Timothy Wexler.