2020 Census: Census Bureau Needs to Improve Its Life-Cycle Cost Estimating Process
Published: 2016-06-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In October 2015, the Bureau estimated that with its new approach it can conduct the 2020 Census for $12.5 billion, $5 billion less than the $17.8 billion it estimated it would cost to repeat the design and methods of the 2010 Census. Reliable cost estimates can help an agency manage large complex activities like the 2020 Census, as well as help Congress make funding decisions and provide oversight.
GAO was asked to evaluate the reliability of the Bureau's life-cycle cost estimate. Among other objectives, this report assesses the extent to which (1) the Bureau's life-cycle cost estimate met GAO's best practices for cost estimation and (2) the Bureau identified and accounted for key risks facing the 2020 Census. To meet these objectives, GAO reviewed documentary and testimonial evidence from Bureau officials responsible for developing the 2020 Census cost estimate. GAO used its cost assessment guide ( GAO-09-3SP ) and Standards for Internal Control in the Federal Government ( GAO-14-704G ) as criteria.
section_title: What GAO Found
paragraphs: Since 2012, the U.S. Census Bureau (Bureau) has taken significant steps to improve its capacity to carry out an effective cost estimate; however, its October 2015 cost estimate for the 2020 Census does not fully reflect characteristics of a high-quality estimate and cannot be considered reliable. To reflect these characteristics, an organization must meet or substantially meet four best practices. Overall, GAO found the cost estimate partially met the characteristics of two best practices (comprehensive and accurate) and minimally met the other two (well-documented and credible). One reason why GAO's overall assessment is low is because the estimate is not well-documented. Improving cost estimation practices will increase the reliability of the Bureau's cost estimate, which will in turn help improve decision making, budget formulation, progress measurement, course correction when warranted, and accountability for results.
Best practices state a risk and uncertainty analysis should be performed to determine the level of risk associated with the cost estimate. The Bureau carried out such an analysis only for a portion of estimated costs for fiscal years 2018 to 2020. According to Bureau officials, they scoped the analysis narrowly to those 3 years when most of the census costs occur. GAO found that, as a result, the Bureau's risk and uncertainty analysis (modeled costs) covered $4.6 billion, only about 37 percent of the $12.5 billion total estimated life-cycle cost, and less than one-half of the total estimated cost of the census during future fiscal years.
Note: All figures are in constant 2020 dollars.
The Bureau has risk identification processes, which identify a broad range of risks that could affect the cost of the 2020 Census. Yet this awareness of risk is not leveraged in the Bureau's cost estimation. The cost estimation team did not consult risk registers or examine specific risks directly for inclusion in the cost model or risk and uncertainty analysis. It was not known what risks, if any, had been accounted for in other data in the cost model. As a result, neither the Bureau nor GAO are able to determine with confidence what risks the Bureau is prepared to mitigate or address within its $12.5 billion cost estimate. Improving control over how risk is accounted for will improve confidence that the Bureau's budgeted contingencies are at appropriate levels.
section_title: What GAO Recommends
paragraphs: GAO is making three recommendations including that the Secretary of Commerce direct the Bureau to take specific steps to ensure its cost estimate meets the characteristics of a high-quality estimate and improve control over how risk and uncertainty are accounted for in cost estimation. The Department of Commerce agreed with GAO's recommendations and provided additional context that was incorporated, as appropriate.

section_title: Letter
paragraphs: With a cost of about $13 billion, the 2010 Census was the most expensive  population count in U.S. history, costing over 50 percent more than the  $8.1 billion 2000 Census (in constant 2010 dollars). Some cost growth is  to be expected because the population is growing and becoming more  complex and difficult to count, which increases the workload of the  Census Bureau (Bureau). However, the cost of counting each housing  unit has escalated from $16 in 1970 to $94 in 2010 (in constant 2010  dollars).
For the 2020 Census, the Bureau intends to limit its per-household cost to  not more than that of the 2010 Census, adjusted for inflation. To achieve  this goal, the Bureau is significantly changing how it conducts the census,  in part by re-engineering key census-taking methods and infrastructure. In  October 2015, the Bureau estimated that with its new approach it can  conduct the 2020 Census for a life-cycle cost of $12.5 billion in contrast to  its estimate of $17.8 billion to repeat the design and methods of the 2010  Census (both in constant 2020 dollars).
Reliable cost estimates that appropriately account for risks facing an  agency can help an agency manage large complex activities like the 2020  Census, as well as help Congress make funding decisions and provide  oversight. Cost estimates are also necessary to support decisions about  funding one program over another, to develop annual budget requests, to  determine what resources are needed, and to develop baselines for  measuring performance. Having a realistic estimate of projected costs  makes for effective resource allocation, and it increases the probability of  a program’s success.
Despite their importance, our work has shown agencies across the  federal government have had difficulty developing reliable cost estimates.  Too often, programs cost more than expected and deliver results that do  not satisfy all requirements.
You asked us to evaluate the reliability of the life-cycle cost estimate the  Bureau submitted to Congress in October 2015. We reviewed (1) the  extent to which the Bureau's life-cycle cost estimate met our best  practices for cost estimation; (2) the extent to which the Bureau's key  cost assumptions were supported by field tests, prior studies, and other  evidence-based analysis; and (3) the extent to which the Bureau has  identified and accounted for key risks facing the 2020 Census within its  risk and uncertainty analyses of its life-cycle cost estimate.
For all three objectives, we reviewed documentation related to the cost  estimate and interviewed Bureau officials responsible for developing the  2020 Decennial life-cycle cost estimate. For the first question we  interviewed Bureau cost analysts and evaluated whether the Bureau's  cost estimate was generated according to best practices of our Cost  Estimating and Assessment Guide. For the second question we  inventoried cost assumptions documented by the Bureau in support of its  October 2015 life-cycle cost estimate and identified those associated with  major changes from the Bureau's historical census design and with  significant cost-saving potential. We then attempted to locate these key  assumptions’ sources in order to determine if they were supported by field  tests, prior studies, and other evidence-based analysis. Finally, for the  third question we analyzed the Bureau's project and program risk  registers to determine if the range of risks and how they are incorporated  in the Bureau's uncertainty analyses is adequate. We relied on our cost  assessment guide and Standards for Internal Control in the Federal  Government as criteria. More information on our scope and methodology  can be found in appendix I.
We conducted this performance audit from November 2015 to June 2016  in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Our work on the Bureau’s 2010 Census life-cycle cost estimate found that  it was not reliable, because it lacked adequate documentation and was  not comprehensive, accurate, or credible. For example, in our 2008 report  on the Bureau’s cost estimation process, Bureau officials were unable to  provide documentation that supported the assumptions for the initial 2001  life-cycle cost estimate as well as the updates. We reported that without  improvements to the cost estimation process, the Bureau's ability to  effectively manage operations would be hampered and Congress's ability  to oversee the 2010 Census would be constrained. Consequently, we  recommended that the Bureau establish guidance, policies, and  procedures for conducting cost estimation that would meet best practices  criteria. The Bureau agreed with the recommendation and said at the time  that it already had efforts underway to improve its future cost estimation  methods and systems. Moreover, weaknesses in the life-cycle cost  estimate were one reason we designated the 2010 Census a GAO High  Risk Area in 2008.
Bureau officials stated they have been working toward implementing the  standards of our Cost Estimating and Assessment Guide since 2008.  However, in a 2012 report, we found that while the Bureau was taking  steps to strengthen its life-cycle cost estimates, it had not yet established  guidance for developing cost estimates, and we recommended that the  Bureau finalize its guidance, policies, and procedures for cost estimation  in accordance with best practices. In its response to that report the  Bureau agreed with the overall theme of the report, but did not directly  comment on that recommendation, which remains open, as discussed  later in this report.
subsections: section_title: Reengineering the 2020 Census
paragraphs: To help control costs while maintaining accuracy, the Census Bureau is  introducing significant change to how it conducts the decennial census in  2020. Its planned innovations include reengineering how it builds its  address list, improving self-response by encouraging the use of the  Internet and telephone, using administrative records to reduce field work,  and reengineering field operations using technology to reduce manual  effort and improve productivity. The Bureau estimates that if it succeeds  with these innovations it can conduct the 2020 Census for $12.5 billion in  constant 2020 dollars. By contrast, the 2020 Census would cost $17.8  billion in constant 2020 dollars if the Bureau repeats the 2010 Census  design and methods, according to the Bureau’s estimates.
Major innovation does not come without risks and in order to manage and  mitigate them the Bureau has established a decennial risk management  process governing two types of risks: project risks and program risks. The  Bureau defines project risks as those that could jeopardize success of an  individual project, such as the Bureau's operation to collect responses  door-to-door from households that do not respond to Bureau mailings.  The Bureau defines broader program risks as those that jeopardize the  success of the 2020 Census program, typically spanning several years  with many potential risk events over the period. These may have elevated  from project level risks and include risks such as the possibility that  external stakeholders do not support the planned design changes.
As early as 2011, the Bureau began developing preliminary cost  estimates of the 2020 Census in order to approximate potential savings  from its plans to reengineer the census, and, according to the Bureau, to  begin developing the methodology for producing the decennial life-cycle  cost estimates.
The Bureau’s October 2015 release of the latest cost estimate marked  the transition from the “research” to “implementation” phases of the 2020  Census. According to the Bureau, this was the Bureau’s first attempt to  model the life-cycle cost of its planned 2020 Census, in contrast to its  earlier 2011 estimate which the Bureau said was intended to produce an  approximation of potential savings and to begin developing the  methodology for producing decennial life-cycle cost estimates covering all  phases of the decennial life cycle (see fig. 1).
section_title: How the Bureau Produced Its Latest Cost Estimate
paragraphs: According to Bureau officials, they recently implemented a software  upgrade they believe has better positioned them to deliver a quality cost  estimate. They stated that they relied primarily on a cost estimation team  of five individuals in the Decennial Programs Directorate to develop the  2020 Census life-cycle cost estimate. This team drew in part on subject  matter specialists involved in the Bureau’s research and testing of its  planned innovations in order to help develop a cost model and to obtain  data to inform key assumptions within the model. The specialists typically  provided the cost estimation team with data on each assumption using a  three-point method, whereby the cost estimation team received a  “minimum,” “value,” and “maximum” input for each assumption,  representing whatever uncertainty or risks the provider had considered.  Figure 2 shows the Bureau’s cost estimation process.
Because cost estimates predict future program costs, uncertainty is  always associated with them. For example, data from the past (such as  fuel prices) may not always be relevant in the future. Risk and uncertainty  refer to the fact that because a cost estimate is a forecast, there is always  a chance that the actual cost will differ from the estimate.
One way to determine whether a program is realistically budgeted is to  perform an uncertainty analysis, so that the probability associated with  achieving its point estimate can be determined, usually relying on  simulations such as those of Monte Carlo methods. This can be  particularly useful in portraying the uncertainty implications of various cost  estimates.
Consistent with prevailing cost estimation practices, the Bureau’s Office  of Cost Estimation, Analysis, and Assessment (OCEAA), in the Deputy  Director’s Office (see fig. 3), began generating an independent cost  estimate in fiscal year 2015, and shared it with the cost estimation team in  April 2016. OCEAA and the cost estimate team worked together to  examine the process they each used, an effort known as the  reconciliation process. According to Bureau officials, reconciliation is a  major step in the cost estimation process, and it may help identify areas  of improvement for the Bureau’s cost estimation process. Officials also  said that this reconciliation was to be shared with the Department of  Commerce in June 2016 as part of its 2020 Census budget process.
The Bureau plans annual updates of its cost estimate, leaving three  updates before the Bureau begins implementing its earliest operations in  2019 for the 2020 Census.
section_title: The Bureau’s October 2015 Cost Estimate Does Not Reflect Key Best Practices
paragraphs: Since our January 2012 report in which we reviewed the Bureau’s initial  estimate of the total cost of the decennial census, the Bureau has taken  significant steps to improve its capacity for cost estimating. For  example, the Bureau established OCEAA as an enterprise-level cost  estimation office and hired certified cost analysts. Despite this progress,  the Bureau’s October 2015 cost estimate for the 2020 Census does not  fully reflect characteristics of a high-quality estimate as described in our  2009 Cost Estimating and Assessment Guide and cannot be  considered reliable.
A reliable cost estimate is critical to the success of any program. Such an  estimate provides the basis for realistic budget formulation and program  resourcing, meaningful progress measurement, proactive course  correction when warranted, and accountability for results. According to  the Office of Management and Budget (OMB), programs must maintain  current and well-documented estimates of program costs, and these  estimates must encompass the full life-cycle of the program. Without this  capability, agencies are at risk of experiencing program cost overruns,  missed deadlines, and performance shortfalls. The Cost Estimating and  Assessment Guide describes best practices for the development of  reliable cost estimates. For our reporting needs, in the cost guide we  collapsed these best practices into four general characteristics for sound  cost estimating—comprehensive, well-documented, accurate, and  credible—and identified specific best practices for each characteristic.
To reflect these characteristics, an organization must meet or  substantially meet each best practice. Our review found the Bureau  partially or minimally met the cost estimating best practices. More  specifically, it partially met two characteristics and minimally met two  characteristics (see table 1).
According to our best practices, an estimate is comprehensive if it has  enough detail to ensure that cost elements are neither omitted nor  double-counted, all cost-influencing assumptions are detailed in the  estimate’s documentation, and a single work breakdown structure (WBS)  is defined and all WBS elements are described in a WBS  dictionary. While Bureau officials were able to provide us with several  documents that included projections and assumptions that were used in  the cost estimate, we found the estimate to be partially comprehensive  because it is unclear if all life-cycle costs are included in the estimate or if  the cost estimate completely defines the program. Additionally, we found  the Bureau had four versions of WBSs with inconsistencies among them.  For example, the Bureau’s standard WBS, operational plan WBS,  decennial budget WBS, and the life-cycle cost estimate WBS contained  differing numbers of major areas (e.g., program management and  response data) and we could not determine how all the different areas fit  together.
Bureau officials stated that they have worked to create an agency-wide  standard WBS and that the different versions of the WBS they provided to  us were similar. Yet the Bureau did not provide evidence that  demonstrated how the WBS related to each other. These issues reduce  the reliability of the Bureau’s current life-cycle cost estimate, because as  a result, we could not determine if all life-cycle costs are included in the  cost estimate, and decision makers and others cannot know whether the  total estimate fully accounts for all costs. However, the Bureau’s use of  WBSs is an improvement over what we found in 2008 when the Bureau  did not have a WBS in place at all.
In addition, about $3 billion, or one-quarter, of the total cost of the census  is represented by cost breakouts that are not in the Bureau’s WBS. These  comprise dollar amounts for the annual 2020 Census enacted or  requested budget line for fiscal years 2012 through 2017 and their out  year estimates for 2021 through 2023. Bureau officials justified using  such aggregate budget figures by describing much of these years’ costs  as staff cost at headquarters and as relatively fixed in nature—basic  requirements to conduct a decennial census. We reported on the  limitations associated with relying on aggregate budget numbers instead  of detailed actual costs to support cost estimation in 2008 and on the  limitations of isolating cost drivers (i.e., cost of specific field operations)  within overly broad categories of cost in 2012. Although the Bureau  generally agreed with the findings in both reports, it maintained that  relying on appropriations figures within cost estimates was appropriate  and expressed belief that the weaknesses we identified would not affect  the Bureau’s ability to control future costs. However, according to best  practices, the description of the costs should be tied to the WBS so that  the Bureau or others can measure actual variances on specific cost  elements from estimates, identify possible cost drivers, and inform life- cycle cost estimation activity for future censuses.
subsections: section_title: Well-Documented
paragraphs: Cost estimates are considered valid if they are well-documented to the  point they can be easily repeated or updated and can be traced to original  sources through auditing, according to best practices. Rigorous  documentation also increases an estimate’s reliability and helps support  an organization’s decision making. The documentation should explicitly  identify the primary methods, calculations, results, rationales or  assumptions, and sources of the data used to generate each cost  element.
One reason why our overall assessment is low is because the estimate is  not well-documented. While the Bureau provided some documentation of  supporting data, it did not describe how the source data were  incorporated. Additionally, while officials provided documentation to show  the estimating approach used for the majority of the cost elements and  the estimate results, it did not show the estimate calculations.
Moreover, the majority of methodologies used to develop the cost  estimate relied on expert opinion instead of more quantitative methods of  estimating. While expert opinion can be valuable in the absence of other  data, according to best practices it should be used sparingly because of  its subjectivity, potential to introduce bias, and lack of supporting  documentation.
Similar to our findings, a contractor hired by the Bureau to assess the  Bureau’s cost estimate identified the Bureau’s primary estimation  weakness as a lack of formal, stand-alone documentation. The  contractor’s January 2016 assessment noted that cost estimate  documentation was available to the cost estimate team, but not contained  in a formal document accessible to outside analysts. The contractor also  noted that the cost estimate team acknowledged the lack of a  documented cost estimating process.
While Bureau officials discussed with us how they believe they  implemented best practices in producing the Bureau’s 2020 life-cycle cost  estimate, those efforts have generally not been well-documented. Best  practices state that thorough documentation is essential for validating and  defending a cost estimate. A poorly documented estimate does not  provide convincing support for the estimate’s validity and fails to answer  decision makers’ and oversight groups’ questions.
Failure to document an estimate in enough detail for someone unfamiliar  with the program to recreate or update the estimate makes it more difficult  to detect possible errors in the estimate, reduces transparency of the  estimation process, and can undermine the ability to use the information  to improve future cost estimates or even to reconcile the estimate with  another independent cost estimate. Bureau officials acknowledged  throughout our review the importance of documenting the cost estimate  and stated that facing resource constraints, they had prioritized their  research and testing efforts, as well as completing the 2020 Census  Operational Plan with the associated cost estimate, over documenting the  cost estimate.
In April 2016, Bureau officials provided us with a summary of a  documentation strategy they plan to have implemented in summer 2016.  That strategy includes a cost estimation plan and schedule for future  updates, a framework describing the fundamental basis of the estimate,  and a comprehensive list of artifacts and other documents to be created.  This strategy indicates that much progress might be made during the  coming year to improve the Bureau’s documentation; however, it contains  insufficient details to assess the extent to which that progress may help  the Bureau meet documentation best practices.
section_title: Accurate
paragraphs: According to best practices, an estimate that is accurate is unbiased, is  not overly conservative or overly optimistic, and is based on an  assessment of most likely costs. Few, if any, mathematical mistakes are  present and those that are present are minor. We found the estimate  partially met best practices for this characteristic. For example, Bureau  officials said that they only applied risk and uncertainty analysis to the  portion of the 2020 Census estimate for the years 2018 to 2020. Officials  said they focused on cost estimating methodologies and data collection  and normalization for that period because the majority of costs, roughly  80 percent of the total estimate, occur in those 3 years. However, it  appears the balance of the estimate was not adjusted for any specific  level of confidence, so we cannot determine the confidence level of the  total life-cycle estimate. Therefore, we cannot determine if the estimate is  unbiased, overly conservative, or overly optimistic.
Additionally, we could not independently verify the calculations the  Bureau used within its cost model to ensure there were no major  mathematical mistakes. While the Bureau’s new cost estimation software  upgrade contains the Bureau’s calculations within it, access to the  software is limited, restricting the accessibility of these calculations or  related notes. In one example, Bureau officials provided us with the  inflation rates they used, but we could not verify the application of the  rates because the documentation provided did not include the  calculations.
section_title: Credible
paragraphs: Credible cost estimates clearly identify limitations due to uncertainty or  bias surrounding the data or assumptions, according to best practices.  Major assumptions should be varied and other outcomes should be  recomputed to determine how sensitive outcomes are to changes in the  assumptions. In addition, a risk and uncertainty analysis should be  performed to determine the level of risk associated with the estimate.  Finally, the results of the estimate should be cross-checked and an  independent cost estimate should be performed to determine whether  alternative estimate views produce similar results.
We found the estimate minimally met best practices for this characteristic.  Bureau officials acknowledged that while a contractor they hired  performed cross-checks of selected cost elements, overall cross-checks  were not used. Also, although OCEAA began generating an independent  cost estimate in fiscal year 2015 and shared it with the cost estimation  team in April 2016, it was not made available to us at the time of this  review. Bureau officials said they conducted a sensitivity analysis in order  to identify cost drivers, but had not documented it by the time of our  review. Additionally the Bureau carried out its risk and uncertainty  analysis only for a portion of costs in fiscal years 2018 to 2020, telling us  it scoped it narrowly by design to those 3 years when most of the census  costs—and predominantly variable costs—occur. We found that the  Bureau’s risk and uncertainty analysis (modeled costs) covered $4.6  billion, only about 37 percent of the $12.5 billion total estimated life-cycle  cost, and less than one-half of the total estimated future cost of the  census, which would include fiscal years 2017 to 2023 (see figure 4).  Based on its risk and uncertainty analysis, the Bureau included a  contingency of about $500 million to establish a total estimated value that  it believed will equal or exceed the true cost with 80 percent certainty.  testified on our concerns about the management and progress of the  CEDCaP program that could contribute to cost overruns.
Other costs excluded from the modeling were costs of advertising ($347  million), various operations such as coverage improvement ($204 million)  and in-office canvassing ($22 million), non-CEDCaP information  technology systems ($221 million), as well as nearly all decennial costs  for fiscal years 2021 through 2023 ($877 million). Bureau officials stated  that much of the omitted costs are for fixed contracts or Bureau  headquarters staff that are not susceptible to cost-related risks; however,  these costs also include uncertain cost drivers, such as CEDCaP and  other systems, which could have been included in the uncertainty  analyses.
The Bureau used management discretion to determine how much  contingency to add on top of the remaining costs. An additional 10  percent was added for fiscal years 2018 through 2020, for a total  additional contingency of $829 million. However, officials were not able to  justify the 10 percent factor and there was no Bureau documentation  justifying the additional contingency. Because the Bureau only carried out  its uncertainty analysis on a portion of the cost estimate, we cannot  determine if it fully identified the level of risk associated with the estimate.  Nor can we validate the Bureau’s reported confidence level of the total  life-cycle cost estimate or how it relates to the Bureau’s total contingency.
We found the Bureau had little planning information among its documents  supporting its cost estimate. Early fundamental planning and guidance  documents such as general policies and procedures for cost estimation— in contrast to polished final process descriptions that might be produced  later—can contribute to consistent control over the process used to  develop a cost estimate and help ensure that desired standards and  practices are implemented. Internal controls for the federal government  state that management should design control activities to achieve  objectives such as the development of a reliable cost estimate. These  internal controls could take many forms, such as an operational plan,  guidance on specific steps, and job aids for staff involved in the process.  Internal controls would help the Bureau ensure continuity of operations  across turnover in staff during the decennial life cycle, ensure that its cost  estimation process follows best practices, and help it meet its objective of  a reliable cost estimate. We recommended the Bureau take these steps  to put guidance, policies, and procedures in place in our 2008 and 2012  reports on the Bureau’s cost estimation process. While the Bureau has  not yet implemented the recommendation, we continue to believe that it is  valid in order to ensure the Bureau improves the reliability of its cost  estimate.
section_title: The Bureau Could Demonstrate That About Half of Its Key Assumptions Were Supported by Field Tests, Prior Studies, or Other Research
paragraphs: Cost estimates typically include a number of unknowns because they are  based on future events. To account for these unknowns, cost estimators  include various assumptions that are often built with limited information.  These assumptions represent a set of judgments about past, present, or  future conditions. Best cost estimation practices state that assumptions  should be realistic and valid, meaning that historical data should back  them up to minimize uncertainty and risk. Analysts must ensure that  assumptions are not arbitrary, and that they are founded on expert  judgments rendered by experienced program and technical personnel.  Further, well-supported assumptions should include documentation of an  assumption’s source.
Recognizing that not all assumptions are equal in their impact on cost, we  worked with the Bureau’s cost team to identify key Bureau assumptions  that were included within the Bureau’s model for its life-cycle cost  estimate. We identified 41 key assumptions: examples included staffing  ratios, pay rates, and anticipated production rates for the major field  operations Address Canvassing and Nonresponse Follow-up (NRFU).  Not all of these assumptions are likely to have the same impact on the  cost model or are as directly testable in the field. According to the Bureau,  it prioritized field testing for the most sensitive or high-impact  assumptions, based on the availability of resources, and relied on subject  matter expertise for assumptions that are more difficult to test or that  affect relatively low-cost operations.
The Bureau’s model required each of these 41 assumptions to be  associated with three inputs: a minimum, value, and maximum. The cost  team told us that many of the assumptions were developed by subject  matter specialists who provided them to the cost team. In other cases, the  cost team said they developed the assumptions themselves.
We requested the source for each of the 41 key assumptions so we could  determine if the key assumptions were based on historical data or expert  judgments rendered by experienced program personnel. Figure 5 shows  the sources of support for the assumptions we could determine.
Of the 41 key assumptions, Bureau officials provided evidence that 18  were supported by field tests, prior studies, other research, or a  combination of these sources, as shown in the following examples:
Assumptions related to the use of administrative records were derived  from the results of the 2015 Census Test.
NRFU assumptions relied on recent test results from the 2014 and  2015 Census site tests.
Address canvassing productivity assumptions drew on operational  results from the 2010 Census.
By contrast, in our 2008 assessment, we had difficulty linking any  assumptions to specific evidentiary support.
The other 23 assumptions did not have evidence to which the  assumptions could be fully traced, though the cost estimation team  provided verbal explanations for the assumptions. When we interviewed  the subject matter specialists who developed the assumptions, they  provided us with the research and information they used to develop their  assumptions. However, for 10 assumptions, data provided by subject  matter specialists did not include minimum and maximum inputs or had  been changed by the cost estimation team. The cost estimation team did  not record how and why it changed assumptions that were provided to it,  how the range was determined if specialists had not provided a range, or  specific sources of support for assumptions that were said to be 2010  actual census numbers, 2010 assessments, or management decisions.
According to Standards for Internal Control in the Federal Government,  management should design control activities to achieve objectives.  However, the Bureau had no guidance for how the cost team was  supposed to handle information provided to it. In particular, the cost  estimation team had no clear guidance for determining when to adjust or  augment information provided to it. Nor did the team have procedures for  recording its adjustments or their justification. When we found that data in  the cost model differed from what subject matter specialists provided, we  were unable to determine why it differed and, in some cases, neither  could the cost estimation team. Lack of a record of how and why changes  were made introduces uncertainty as to the origin and credibility of the  numbers in the cost estimate. Additionally, a lack of information on these  changes makes it difficult to determine the significance or impact of the  changes.
According to the cost estimation team, six of the assumptions were based  on management decisions, such as staffing ratios for field operations, but  the cost team could not provide documentation of these decisions. The  cost team said seven assumptions were based on “2010 actual” data and  related assessments or from previously developed preliminary cost  estimates. For example, they said they used operational data (such as  2010 NRFU pay rates) and assessments of the 2010 Census (such as  the 2010 Address Canvassing Operational Assessment) to develop  assumptions. We tried to independently verify these assumptions but  could not.
We asked the cost estimation team for the source or citation to the actual  historical data or assessments that the team may have relied on, but it  could not provide them. In several cases the team instead directed us to  the Bureau’s 2011 estimate. Documentation of that earlier estimate also  referred to the 2010 Census (and assessments) as a primary source, but  did not provide specific citations or traceable references. Without support  for these assumptions we cannot determine if they are in fact realistic and  valid or if they are based on historical data.
section_title: The Bureau Identified a Broad Range of Risks, but Did Not Clearly Account for Them in the Cost Estimate
subsections: section_title: The Bureau Identified a Broad Range of Risks Potentially Affecting 2020 Census Cost
paragraphs: The Bureau identified 158 different project-level risks for the 2020  Census. For example, these risks include delays in the acquisition of  imagery needed to help build the Bureau’s national address list, and  schedule delays in the delivery of information technology systems. The  Bureau ranked its project-level risks on a scale of potential impact in five  different areas, including cost. About 35 percent of the risks were  identified as having a potential impact of at least “3” on a 5-point scale,  equivalent to a potential cost impact of greater than or equal to 5 percent.  (Bureau guidance does not specify against what cost element this impact  rating potentially applies.) Of the 158 project-level risks, Bureau officials  flagged 55 as having a significant potential impact on cost (see figure 6).
A review of our prior work and Department of Commerce Inspector  General reporting on the decennial census, as well as our interviews with  National Academies of Science staff selected for their involvement in  study panels of the 2020 Census, did not identify any risks not already  included in the Bureau’s list of risks. The Bureau designed and  documented risk identification processes. For example, it has a risk  management plan laying out roles, responsibilities, and processes and  staff receive training on it. Project teams working on implementation of  the census design, and research teams preceding them, had  requirements to identify and report on risks potentially affecting their  areas. Additionally, the Decennial Directorate has guidance for generating  ideas about risks through brainstorming. This guidance was designed to  help ensure inclusivity.
A broad list of identified risks can help provide a solid basis for assessing  and mitigating the susceptibility of project implementation to uncertain  conditions facing the decennial program. This can help the Bureau make  informed decisions to help control cost.
section_title: Improving Controls Could Help the Bureau More Fully Account for Risks in Its Future Cost Estimation
paragraphs: Standards for Internal Controls in the Federal Government call for  management to document in its policies or day-to-day procedures the  responsibilities needed to attain its objectives. Clear guidance on what  risks are being accounted for and how those are communicated through  the cost estimation process will help ensure that resulting cost estimates  reflect the best information available. One of the objectives of a robust  cost estimation process is to determine the uncertainty of the cost  estimate with an understanding of how much of that uncertainty is  introduced by significant risks. This includes knowing what risks, if any,  have been accounted for as part of any supporting sensitivity analysis or  later management evaluation of risk and uncertainty.
However, we found the Bureau had no guidance, policies, or procedures  in place on how to account for risk, develop boundaries around key  assumptions, or determine contingencies. Furthermore, subject matter  specialists did not receive training or other guidance on what risks, if any,  they were to consider as part of their support of the cost estimation  process; how to incorporate, document, or communicate what they had  accounted for; or how they would account for uncertainty within the  assumptions. In addition, specialists used a variety of methods to account  for risk and uncertainty in the cost assumptions they were responsible for,  but there was no standard methodology for doing this. For example, one  specialist reported providing the calculated standard deviations resulting  from the supporting statistical analysis to the cost estimation team,  assuming that the core estimation team would set ranges around the  provided value according to whatever methodology the team used.  Another reported setting the submitted parameter “conservatively” and  described wanting to “include a buffer” or contingency so that the cost in  that area was not estimated too low. Varying treatments of uncertainty,  contingencies, or risk can make it difficult to determine later what risk has  been fully accounted for already within cost estimates.
Some of the subject matter specialists we spoke with had contributed  directly to the identification of various risks within the Bureau’s risk  management process, and they said that they had some of those in mind  when determining ranges for assumptions they provided. However, none  had reviewed the range of risks covered in the 2020 Census program or  project risk registers when doing so or had communicated any risks that  they may have considered to the cost estimation team. None reported  having received guidance or direction on what risk lists, if any, to  consider.
The cost estimation team said that it was aware of the risk registers, but  had not consulted them. It also reported not having examined specific  risks directly for accounting in the cost model. The cost estimation team  also said that it was unaware whether or what risk subject matter  specialists may have considered; what the basis for the range between  minimum, value, and maximum inputs provided to it on each assumption  may have been, or to what extent some specialists had already built  contingency for general uncertainty into the data the specialist provided.  The absence of procedures for the cost estimation team and subject  matter specialists could have led to the over-specification of some  uncertainty around the cost estimates we observed, or to missing the  potential impact of some key risks.
The Bureau had mechanisms in place to make its identified risks known  within the agency. For example, the Bureau’s Office of Risk Management  and Program Evaluation told us that the Bureau made available to Bureau  project managers—including some of the subject matter specialists—an  online “risk dashboard” presenting risks documented across agency  projects. This is intended to function as a communication tool, increasing  risk visibility across project teams. Yet according to Bureau officials, the  Bureau relied on its certified cost estimators to account for risks and did  not have requirements that they specifically consider these tools or the  risk registers.
Improved control over which risks and uncertainty are accounted for in  cost estimates will better position Bureau managers to know which risks  among those the Bureau has separately identified have or have not  already contributed to the Bureau’s measured uncertainty in its cost  estimate, and to know how much resources, if any, may thus be worth  allocating to mitigate specific risks in order to control cost. Controls that  can help with this accounting may include the implementation of  processes or methods, for example institutionalized in the form of clearly  documented guidance for those involved. In the absence of such  accounting, management cannot be confident that (1) the contingency it  selected is adequate to cover significant risks in its risk registers, or (2)  the selected contingency is not overly conservative, which ties up funding  that could be allocated to other projects.
section_title: Conclusions
paragraphs: In order for the Bureau to improve its ability to control the cost of the 2020  Census, it will be critical for it to have better control over its cost  estimation process. According to Bureau officials, an example of the  steps they have taken to improve their cost estimation capabilities is the  recent implementation of a software upgrade they believe has better  positioned them to deliver a quality cost estimate. While the Bureau has  taken significant steps toward improving its capacity to produce reliable  cost estimates, those efforts have not yet resulted in a reliable decennial  cost estimate. Among the four broad characteristics of a reliable cost  estimate—none of which the Bureau fully met—the Bureau has reported it  is focusing its attention on improving the documentation of the cost  estimate, in order to help improve other characteristics as well. While  poor documentation has affected our ability to assess the reliability of the  Bureau’s cost estimate’s other characteristics, we believe the problems  we observed relate to an absence of internal control procedures over the  cost estimation process, which in turn have resulted in the weakness in  documentation.
In June 2008 we recommended the Bureau establish guidance, policies,  and procedures for conducting cost estimation that would meet best  practices criteria. In response, the Bureau agreed with the  recommendation and said at the time that it already had efforts underway  to improve its future cost estimation methods and systems. Eight years  later, the absence of guidance to control the cost estimation process  persists. Investment in the planning documents to help control and  support cost estimation early in the estimation cycle, such as with an  operational plan, guidance on key steps and process flows, assignment of  responsibilities, and job aids for staff can help institutionalize practices  and ensure that otherwise disparate parties in the process operate  consistently. We continue to believe that establishing guidance, policies  and procedures could help the Bureau incorporate the four characteristics  of reliable cost estimates into future updates to the 2020 life-cycle cost  estimate. Further, taking steps to ensure its cost estimate is reliable  would help improve decision making, budget formulation, progress  measurement, course correction when warranted, and accountability for  results. Following the specific steps laid out in the four characteristics  discussed in this report would position the Bureau to produce a reliable  cost estimate for the 2020 Census.
In addition, we found that the cost estimation team did not record how  and why it changed assumptions that were provided to it or traceable  sources of all data it used. The Bureau lacked written guidance and  procedures for the cost estimation team to follow. Clear guidance on  when information for cost assumptions can and should be changed as  well as the procedures for documenting such changes and traceable  sources for information being used can reduce uncertainty on where data  are coming from and improve their credibility.
Finally, the Bureau had risk management processes for identifying a  broad range of risks that could affect the cost of the 2020 Census. Yet  this institutional awareness of risk was not fully leveraged in the Bureau’s  cost estimation process. As a result, the Bureau is unable to determine  with confidence what risks the Bureau with its $12.5 billion cost estimate  is prepared to mitigate or address. The Bureau determined that it is “80  percent” confident that its cost estimate will cover the actual cost of the  2020 Census, but given the inability to know what risk is accounted for,  we do not see how the Bureau can be that sure.
The Department of Commerce will soon release its independent cost  estimate as well as results of its reconciliation with the 2020 Census  program estimate as part of its budget formulation process. Whether the  independent estimate is higher or lower than the $12.5 billion estimate,  the concerns we have raised about management attention to cost  estimation remain. Improving control over how risk and uncertainty are  accounted for and communicated with the Bureau’s decennial cost  estimation process, such as by implementing and institutionalizing  processes or methods with clear guidance, will improve Bureau and  congressional confidence that the Bureau’s budgeted contingencies are  at appropriate levels.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Commerce and Under Secretary for  Economic Affairs direct the Census Bureau to take the following actions: 1.  To help ensure the Bureau produces a reliable cost estimate for the  2020 Census, take the following steps to meet the characteristics of a  high-quality estimate:
Comprehensive—among other practices, ensure the estimate  includes all life-cycle costs and documents all cost-influencing  assumptions.
Well-documented—among other practices, ensure that its planned  documentation plan captures the source data used; contains the  calculations performed and the estimating methodologies used for  each element; and describes step by step how the estimate was  developed.
Accurate—among other practices, ensure the estimating  technique for each cost element is used appropriately and that  variances between planned and actual cost are documented,  explained, and reviewed.
Credible—among other practices, ensure the estimate includes a  sensitivity analysis, major cost elements are cross-checked to see  whether results are similar, and an independent cost estimate is  conducted to determine whether other estimating methods  produce similar results.  2.  To further ensure the credibility of data used in cost estimation,  establish clear guidance on when information for cost assumptions  can and should be changed as well as the procedures for  documenting such changes and traceable sources for information  being used.  3.  To ensure Bureau and congressional confidence that the Bureau’s  budgeted contingencies are at appropriate levels, improve control  over how risk and uncertainty are accounted for and communicated  with the Bureau’s decennial cost estimation process, such as by  implementing and institutionalizing processes or methods for doing so  with clear guidance.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Secretary of the Department of  Commerce for comment. In written comments, reproduced in appendix II,  the Department of Commerce agreed with our recommendations. It also  provided additional context that we incorporated, as appropriate.
The Department of Commerce also noted that while it fully recognizes the  Census Bureau can further improve its process under the Cost Estimating  and Assessment Guide as well as the Standards for Internal Control in  the Federal Government, it stands behind the quantitative integrity of the  current life-cycle cost estimates for the 2020 Census.
We maintain that the process used to develop a cost estimate is key to its  quantitative integrity. Unless and until the Bureau develops a cost  estimate that fully reflects the characteristics of a high-quality estimate  such as being comprehensive, well-documented, accurate, and credible,  the estimate itself cannot be considered reliable.
We are sending copies of report to the Secretary of Commerce, the  Counselor to the Secretary with Delegated Duties of the Undersecretary  of Commerce for Economic Affairs, the Director of the U.S. Census  Bureau, and interested congressional committees. The report also will be  available at no charge on GAO's website at http://www.gao.gov.
If you have any questions about this report please contact me at (202)  512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. The GAO staff that made major contributions to this report  are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The purpose of our review was to evaluate the reliability of the Census  Bureau’s (Bureau) life-cycle cost estimate using our Cost Estimating and  Assessment Guide (GAO-09-3SP, or GAO Cost Guide). We reviewed (1)  the extent to which the Bureau's life-cycle cost estimate met our best  practices for cost estimation; (2) the extent to which the Bureau's key cost  assumptions were supported by field tests, prior studies, and other  evidence-based analysis; and (3) the extent to which the Bureau has  identified and accounted for key risks facing the 2020 Census within its  risk and uncertainty analyses of its life-cycle cost estimate. For all  objectives, we reviewed documentation from the Bureau on the 2020 life- cycle cost estimate.
For the first objective, we relied on the cost guide as criteria. For the cost  guide, our cost specialists assessed measures consistently applied by  cost-estimating organizations throughout the federal government and  industry and considered best-practices for the development of reliable  cost-estimates. We analyzed the cost estimating practices used by the  Bureau against these best practices and evaluated them in four  categories: comprehensive, well-documented, accurate, and credible.
Comprehensive: The cost estimate should include both government and contractor costs of the program over its full life-cycle, from inception of the program through design, development, deployment, and operation and maintenance to retirement of the program. It should also completely define the program, reflect the current schedule, and be technically reasonable. Comprehensive cost estimates should be structured in sufficient detail to ensure that cost elements are neither omitted nor double-counted. Specifically, the cost estimate should be based on a product-oriented work breakdown structure (WBS) that allows a program to track cost and schedule by defined deliverables, such as hardware or software components. Finally, where information is limited and judgments are made, the cost estimate should document all cost-influencing assumptions.
Well-documented: A good cost estimate—while taking the form of a single number—is supported by detailed documentation that describes how it was derived and how the expected funding will be spent in order to achieve a given objective. Therefore, the documentation should capture in writing such things as the source  data used, the calculations performed and their results, and the  estimating methodology used to derive each WBS element’s cost.  Moreover, this information should be captured in such a way that the  data used to derive the estimate can be traced back to and verified  against their sources so that the estimate can be easily replicated and  updated. The documentation should also discuss the technical  baseline description and how the data were normalized. Finally, the  documentation should include evidence that the cost estimate was  reviewed and accepted by management.
Accurate: The cost estimate should provide for results that are unbiased, and it should not be overly conservative or optimistic. An estimate is accurate when it is based on an assessment of most likely costs, adjusted properly for inflation, and contains few, if any, minor mistakes. In addition, a cost estimate should be updated regularly to reflect significant changes in the program—such as when schedules or other assumptions change—and actual costs, so that it is always reflecting current status. During the update process, variances between planned and actual costs should be documented, explained, and reviewed. Among other things, the estimate should be grounded in a historical record of cost estimating and actual experiences on other comparable programs.
Credible: The cost estimate should discuss any limitations of the  analysis because of uncertainty or biases surrounding data or  assumptions. Major assumptions should be varied, and other  outcomes recomputed to determine how sensitive they are to changes  in the assumptions. Risk and uncertainty analysis should be  performed to determine the level of risk associated with the estimate.  Further, the estimate’s cost drivers should be cross-checked, and an  independent cost estimate conducted by a group outside the acquiring  organization should be developed to determine whether other  estimating methods produce similar results.
If any of the characteristics are not met, minimally met, or partially met,  then the cost estimate does not fully reflect the characteristics of a high- quality estimate and cannot be considered reliable.
For the second objective we inventoried cost assumptions documented  by the Bureau in support of its October 2015 life-cycle cost estimate and  identified those associated with major changes from the Bureau's  historical census design and with significant cost-saving potential.  Additionally, we assessed the reliability of key assumptions by  determining to what extent they are based on prior Bureau experience  and testing, such as related results of the 2014 and 2015 Census Tests  and other historical support.
Finally, for the third objective, we analyzed the Bureau's project and  program risk registers and leveraged prior GAO work in this area to  determine the range of risks and the adequacy of the Bureau's  uncertainty analysis. We relied on our cost assessment guide and our  Standards for Internal Control in the Federal Government as criteria.
We conducted this performance audit from November 2015 to June 2016  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
GAO, Standards for Internal Control in the Federal Government. GAO-14-704G  (Washington, D.C.: Sept. 10, 2014).
section_title: Appendix II: Comments from the Department of Commerce
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Ty Mitchell, Assistant Director;  Brian Bothwell; Brett Caloia; Robert Gebhart; Jason Lee; Andrea Levine;  Donna Miller; Cynthia Saunders; and Timothy Wexler.