2010 Census: Census Bureau Should Take Action to Improve the Credibility and Accuracy of Its Cost Estimate for the Decennial Census
Published: 2008-06-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The 2010 Census will be the most expensive census in our nation's history, even after adjusting for inflation. The Census Bureau (Bureau) estimates that the life cycle cost of the 2010 Census will be from $13.7 billion to $14.5 billion. GAO was asked to (1) assess the extent to which the Bureau's 2010 Census life cycle cost estimate adheres to characteristics defined for high-quality cost estimation, (2) report on the relationship between the estimate and the Bureau's budget, and (3) assess whether the Bureau's existing policies and resources are sufficient to conduct cost estimation. To assess the reliability of the Bureau's cost estimate, GAO analyzed the Bureau's methods and approaches to determine if the estimate is well-documented, comprehensive, accurate, and credible.
section_title: What GAO Found
paragraphs: The Bureau's 2010 Census life cycle cost estimate is not reliable because it lacks adequate documentation and is not comprehensive, accurate, or credible. The Bureau could not provide detailed documentation on data sources, significant assumptions, or changes in assumptions for the cost estimate. The cost estimate is not comprehensive because the Bureau did not include the potential cost to fingerprint temporary workers or clearly define some of the cost elements in the model. The cost estimate is not accurate because it does not reflect updated information on address canvassing productivity that was identified during the dress rehearsal and that should result in a significant cost increase. Further, the Bureau does not maintain historical data in a centralized way that is easily accessible for analysis. The cost estimate is not credible because the Bureau did not perform sensitivity or uncertainty analyses, which would have helped quantify the risk and uncertainty associated with the cost model and provided a level of confidence for the estimate. The Bureau also did not validate the estimate with an independent cost estimate. The Bureau uses the life cycle cost estimate as the starting point for annual budget formulation and revises the life cycle cost estimate based on appropriations received and updated budget information. However, the Bureau does not update the cost estimate to reflect actual costs. Further, because the life cycle cost estimate is not reliable, annual budget requests based on that estimate are not fully informed. The Bureau has insufficient policies and procedures and inadequately trained staff for conducting high-quality cost estimation for the decennial census. The Bureau does not have established cost estimation guidance and procedures in place or staff certified in cost estimation techniques. While the Bureau is developing a new budget management tool called the Decennial Budget Integration Tool, which will support the cost estimation process, the Bureau will need to establish rigorous cost estimation policies and procedures and use skilled estimators to ensure that future cost estimates are reliable and of high quality. On April 3, 2008, the Secretary of Commerce announced a redesign of the 2010 Census plan that included significant cost increases of $2.2 billion to $3 billion. The details of this cost increase were not available at the time of this review; however, until the Bureau makes fundamental changes to its cost estimation process, uncertainties about the ultimate cost of the 2010 Census will remain. Without improvements to the cost estimation process, the Bureau's ability to effectively manage operations will be hampered and Congress's ability to oversee the 2010 Census will be constrained.

section_title: Letter
paragraphs: At an estimated cost from $13.7 billion to $14.5 billion, the 2010 Census  will be the most expensive census in our nation’s history, after adjusting  for inflation. The U.S. Census Bureau (Bureau) faces the challenge of cost  effectively counting a population that is growing steadily larger, more  diverse, and increasingly difficult to enumerate with a reengineered design  that relies in part on automation to locate housing units. In an  environment of constrained resources, containing costs is a stated goal of  the Bureau’s design for the 2010 Census.
Earlier this year, we designated the 2010 Census as a high-risk area, in part  because of uncertainty over costs and long-standing weaknesses in the  Bureau’s management of information technology intended to automate the  census. In February 2008, the Director of the Bureau initiated a replanning  of the Field Data Collection Automation (FDCA) program, a major  acquisition that includes systems, equipment (including handheld  computers (HHC)), and infrastructure for field staff to use in collecting  data for the 2010 Census. After analyzing several options to revise the  design of the 2010 Decennial Census, the Secretary of Commerce on   April 3, 2008, announced that the Bureau would no longer use HHCs in its  largest field operation—nonresponse follow-up—in which field workers  interview households that did not return census forms. Additionally, the  Bureau decided to have the contractor for the FDCA program reduce  deployment of field technology infrastructure; provide HHCs for address  canvassing, in which field workers verify addresses; and develop the  information system for controlling all field operations. The Bureau  estimated that along with updating its assumptions, this option would  result in a cost increase of $2.2 billion to $3 billion over the previously  reported estimate of $11.5 billion.
Concerned about the challenges of conducting a cost-effective census, you  asked us to assess the Bureau’s life cycle cost estimate for the 2010  Census. As agreed with your offices, our objectives for this report were to  (1) assess the extent to which the Bureau’s 2010 Census life cycle cost  estimate adheres to characteristics defined for high-quality cost  estimation, (2) report on the relationship between the life cycle cost  estimate and the Bureau’s budget, and (3) assess whether the Bureau’s  existing policies and resources are sufficient to conduct high-quality cost  estimation.
For this review, we evaluated the 2010 Census life cycle cost estimate  using the exposure draft of GAO’s Cost Assessment Guide. The guide  contains criteria for developing reliable cost estimates, which are aligned  with four characteristics: high-quality cost estimates should be well- documented, comprehensive, accurate, and credible. During our review  we shared the Cost Assessment Guide with Bureau officials. There are  three components of the life cycle cost estimate: the American Community  Survey (ACS), the Master Address File/Topologically Integrated  Geographic Encoding and Referencing (MAF/TIGER) system, and the  short form census. To address our first objective, we performed sensitivity  analysis to identify significant cost drivers, and limited uncertainty  analysis on a portion of the 2006 version of the 2010 short form census life  cycle cost estimate that the Bureau provided to quantify the uncertainty  and provide a level of confidence for the estimate. We also analyzed  Bureau data sets and documents related to the life cycle cost estimate and  interviewed Bureau officials. To address our second objective, we  reviewed policies and procedures for preparing annual budgets and  analyzed Bureau documents related to the annual budget process,  including Bureau budget estimates and worksheets on life cycle costs  prepared for the budget process. We also interviewed Bureau census  program and budget officials to understand the relationship between the  cost estimate and the annual budget process. To address our third  objective, we analyzed Bureau documents related to the life cycle cost  estimate; attended a demonstration of a new Bureau automated budget  system currently being developed that should enable the Bureau, among  other initiatives, to produce better cost estimates; and interviewed Bureau  officials. Appendix I provides additional information on our scope and  methodology. Our analysis of the 2010 life cycle cost estimate was  conducted prior to the redesign of the census and the subsequent revision  of the FDCA program. Detailed cost information was not available for the  Bureau’s proposed redesign in time to be analyzed as part of our scope.  We conducted our work from October 2006 to June 2008 in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on our  audit objectives.
subsections: section_title: Background
paragraphs: The Bureau’s cost projections for the 2010 decennial census continue an  escalating trend, with the 2010 Census currently estimated to cost  approximately $13.7 billion to $14.5 billion. At a March 2008 hearing,  Commerce and the Bureau stated that the FDCA program was likely to  incur significant cost overruns and that a redesigning effort was under way  to try to get the decennial census back on track. In early April, the  Secretary of Commerce chose from among several alternatives for  redesigning the FDCA program, and elected an option that involves  dropping the HHCs from the nonresponse follow-up operation.  Additionally, he decided that the Bureau would reduce deployment of field  technology infrastructure by the contractor and have the contractor  provide HHCs for address canvassing and develop the information system  for controlling field operations. The Bureau has estimated that, with the  redesign option, the total life cycle cost estimate for the 2010 Census will  be from $13.7 billion to $14.5 billion.
Prior to this recent, major redesign of the FDCA program, the Bureau had  estimated the life cycle cost of the 2010 Census to be $11.8 billion (in  constant 2010 dollars). As shown in figure 1, this estimate of $11.8 billion  represented a more than tenfold increase over the $1 billion spent on the  1970 Census. Although some of the cost increase could be expected  because the number of housing units—and hence the Bureau’s workload— has increased, the cost growth far exceeded this increase. Factors  contributing to the increased costs include an effort to accommodate more  complex households, busier lifestyles, more languages and greater cultural  diversity, and increased privacy concerns. The Bureau estimated that the  number of housing units for the 2010 Census will increase by almost 14  percent over 2000 Census levels (from 117.5 million to 133.8 million  housing units). At the same time, the average cost per housing unit for  2010 was expected to increase by approximately 26 percent over 2000  levels, from $69.79 per housing unit to $88.19 per housing unit (see fig. 2).
The bulk of total life cycle funds for the 2010 Census has yet to be spent.  As shown in table 1, which reflects the Bureau’s life cycle cost estimate for  the 2010 Census that was released in September 2007, the majority of  spending will occur from fiscal year 2009 through fiscal year 2013.
In reengineering the 2010 Census, the Bureau has four goals: to   (1) improve the relevance and timeliness of census long form data,   (2) reduce operational risk, (3) increase the coverage and accuracy of the  census, and (4) contain costs. To achieve these goals, three new  components are key to the Bureau’s plans for 2010:    modernizing and enhancing the nation’s road map through the  MAF/TIGER enhancement program, which includes realigning the  TIGER map to take advantage of global positioning system capabilities,  modernizing the processing system, and expanding geographic  partnerships;  replacing the census long form questionnaire with a more frequent  sample survey, the ACS; and    conducting a short-form-only census using automation to collect  census data.
Prior to 2010, the decennial census collected data using both a short and  long form questionnaire: the short form counted the population, and the  long form obtained demographic, housing, social, and economic  information from a 1-in-6 sample of households. The ACS is a nationwide  survey that is replacing the decennial long form in the reengineered 2010  Census. The ACS, which was first implemented in 2005, collects detailed  characteristics data every year throughout the decade using a large  household survey. The ACS has allowed the Bureau to simplify the 2010  Census since it will now only involve the short form.
In June 2001, the Bureau issued a document describing the process for  reengineering the 2010 Census, which included a life cycle cost estimate  for the 2010 decennial census, projected at $11.28 billion. Since June 2001,  the Bureau has updated the life cycle cost estimate for the reengineered  2010 Census and issued formal documents describing the estimate in June  2003, September 2005, June 2006, and September 2007. In February 2008,  the Bureau’s Budget Estimates as Presented to Congress, Fiscal Year  2009 included a new life cycle cost estimate for the 2010 Decennial  Census of $11.546 billion, an increase of $20 million over the previous  year’s estimate. As described above, in April 2008, the Secretary of  Commerce testified that the current estimate for the 2010 Census is from  $13.7 billion to $14.5 billion.
The 2001 through 2007 life cycle cost documents issued by the Bureau  illustrate that the costs of the three components—the ACS, MAF/TIGER  modernization, and short-form-only census—fluctuated slightly over that  period. The total life cycle costs ranged from a low of $11.2546 billion in  the September 2005 and June 2006 documents to a high of $11.5264 billion  in the September 2007 document. Figure 3 depicts how the estimated life  cycle cost for the 2010 Census fluctuated in Bureau documents from 2001  to 2007. This year, however, the Bureau estimates that the life cycle cost of  the census will increase by up to $3 billion—dwarfing all previous  increases.
In January 2004, we reported that the Bureau’s approach had the potential  to achieve the first three goals for reengineering the 2010 Census, although  reducing operational risk could prove to be difficult because each of the  three components actually introduced new risks, and the Bureau would be  challenged to control the cost of the 2010 Census. To manage the 2010  Census and contain costs, we recommended in our 2004 report that the  Bureau develop a comprehensive, integrated project plan for the 2010  Census that should include itemized estimated costs of each component,  including a sensitivity analysis, and an explanation of significant changes  in the assumptions on which these costs are based. In response, the  Bureau provided us with the 2010 Census Operations and Systems Plan,  dated August 2007. This plan represented an important step forward at the  time. It included inputs and outputs and described linkages among  operations and systems. However, it did not include sensitivity analysis,  risk mitigation plans, a detailed 2010 Census timeline, or itemized  estimated costs of each component. With the redesign, this plan will need  to be updated.
section_title: The Census Bureau’s 2010 Census Life Cycle Cost Estimate Is Not Reliable
subsections: section_title: The 2010 Census Life Cycle Cost Estimate Is Not Adequately Documented
paragraphs: The Bureau’s life cycle cost estimate for the 2010 Census is not adequately  documented. The Bureau could not provide adequate documentation to  support the costs of two of the three components of the total life cycle  cost estimate, the ACS and MAF/TIGER components, which together  accounted for close to 20 percent of total life cycle costs in the 2007 life  cycle cost estimate. Figure 4 shows the three major components of the  total life cycle cost of $11.5 billion as of August 2007.
Costs for the ACS and MAF/TIGER are estimated outside of the cost model  that is used for the 2010 short form census. According to Bureau officials,  they were not able to locate specific documentation for how the Bureau  first calculated the costs for those components of the 2010 Census life  cycle cost estimate in 2001, although they explained that experienced staff  derived the estimates based on professional expertise, similar surveys,  comparative costs for similar projects, and other factors.
We requested documentation that supported the assumptions for the  initial 2001 life cycle cost estimate as well as the updates, but Bureau  officials were unable to demonstrate that support existed. The Bureau did  not provide detailed documentation for data sources and significant  assumptions used in estimating the cost of the short form 2010 Census.  For example, the Bureau’s first document describing the life cycle cost  estimate for the 2010 Census, which was issued in June 2001, includes a  number of assumptions for expected areas of savings that would be  achieved by the reengineered design. One assumption is that the follow-up  workload for the Local Update of Census Addresses (LUCA) program will  be reduced by 25 percent as a result of updating the address list  throughout the decade and using HHCs to reduce time and travel costs.  However, the document does not fully describe how the 25 percent  reduction was calculated or what the associated savings might be.
Similarly, the Bureau did not provide detailed documentation for the  updated life cycle cost estimate for the 2010 short form that was issued in  2003, which includes several changes to assumptions. For example, the  Bureau describes how increases of $128 million have been made to the life  cycle cost estimate based on new requirements, including the need for  more research on ways to reduce duplicate enumerations, more testing  and development on how to identify and enumerate special places/group  quarters, and additional efforts to research and test methods for  enumerating Americans overseas. However, the document does not  describe how the $128 million increase was derived, the costs associated  with each of the new requirements, or the data sources and calculation  methods used.
Cost estimates are well documented when they can be easily repeated or  updated and can be traced to original sources. The documentation should  explicitly identify the primary methods, calculations, assumptions, and  sources of the data used to generate each cost element. The estimating  process should be described and an explanation provided for why  particular methods and data sets were chosen and why these choices are  reasonable. All the steps involved in developing the estimate should be  documented so that a cost analyst unfamiliar with the program could re- create it with the same result. In addition, documentation for the cost  estimate should reflect changes in technical or program assumptions or  new program phases or milestones.
Officials acknowledged that the Bureau does not have a centralized  location in which to keep detailed documentation to support the  assumptions and decision-making process. Further, Bureau officials stated  that there is no written documentation of the process used by  management to agree upon particular assumptions for use in the life cycle  cost estimate, and that there is no systematic documentation regarding  management decisions to make changes to the cost model or life cycle  cost estimates. According to an official, changes to the cost model have  not always been well documented, and sometimes a decision memo is  created to justify a change but not always. Best practices call for  management approval of the cost estimate. Management approval of the  cost estimate should also be documented, including management approval  memorandums or recommendations for change, as well as management  feedback.
Lack of documentation was also identified as an issue for the 2000 Census  life cycle cost estimate. An independent assessment of the prior Census  2000 life cycle cost model that was issued in 1997 found that input data  and assumptions used in the 2000 model typically came from other offices  and were based on historical observation, professional judgment, or both.  The assessment concluded that the Bureau did not have documentation  readily available for external use on the underlying basis of the 2000 cost  estimate, input assumptions, or process characteristics.
As a result of insufficient documentation, the validity and reliability of the  Bureau’s life cycle cost estimate for the 2010 Decennial Census cannot be  verified. Not having adequate documentation also impedes the Bureau’s  ability to support the decennial decision-making process, inform future  estimates, and facilitate oversight by Congress.
Bureau officials indicated that they understand that the Bureau’s cost  estimation process needs to be improved and are currently developing  DBiT, which should improve documentation for the cost estimates. DBiT,  which is described in more detail later in this report, is being  incrementally implemented and will not be fully functional until after 2010.
section_title: The 2010 Census Life Cycle Cost Estimate Is Not Comprehensive
paragraphs: The 2010 Census life cycle cost estimate is not comprehensive because the  Bureau did not include all potential costs or clearly define some of the  cost elements in the model for the short form Census. In February 2008,  Commerce’s Inspector General (IG) reported that the fiscal year 2009  budget request did not include the cost to fingerprint some 900,000  temporary workers to be hired for the 2010 Census. In addition, the  Bureau provided documentation that the life cycle cost estimate did not  include the cost to fingerprint temporary workers. A cost estimate is  comprehensive when it accounts for all possible costs. According to the  IG, the Bureau has not included the cost for fingerprinting temporary staff.  After discussions with Commerce, the Office of Personnel Management,  and the Federal Bureau of Investigation, the Bureau decided to require  fingerprinting. In the previous two decennials, the Bureau obtained  criminal history records for temporary decennial applicants using only a  name check. The Bureau estimates that the cost to fingerprint all  temporary staff could be about $450 million. Thus, there is a risk that the  life cycle cost estimate has been substantially understated.
The Bureau also did not have detailed descriptions for some of the cost  elements in the model or explanations of how individual cost elements  were related. For example, when we analyzed the fiscal year 2006 version  of the cost model for the short form component of the 2010 Census, we  found that $2.7 billion in the Other Objects category were fixed dollar  amounts. The Bureau did not provide detailed descriptions, equations, or  support to explain how individual cost elements within the Other Objects  category were produced or related. Bureau officials stated that most of the  costs in this category were associated with contracts, travel, supplies, and  training. Further, the Bureau does not have a work breakdown structure  (WBS) that clearly identifies and defines all costs contained in the short  form cost estimate. A WBS defines the work necessary to accomplish a  program’s objectives.
A comprehensive cost estimate’s level of detail ensures that all pertinent  cost elements are included and that no costs are double counted. A  comprehensive cost estimate also includes a clearly defined WBS. A WBS  reflects the requirements, resources, and tasks that must be accomplished  to develop a program. The WBS should have a dictionary that defines each  cost element and how it relates to others, clearly describes what is and is  not included in each element, describes resources and processes  necessary to produce the element, and links each element to other  relevant technical documents. A WBS clearly defines the logical  relationship of all program elements and provides a systemic and  standardized way for collecting data, communicates to everyone what  needs to be done and how activities relate to one another, and is an  essential part of developing a cost estimate for a program.
When we asked for documentation for the cost elements in the Other  Objects category, the Bureau was not able to provide support for all the  cost elements and agreed that the documentation supporting cost  elements in the model was not clear. Without clearly defined cost elements  or a well-developed WBS, the Bureau cannot be sure that the cost estimate  captures all relevant costs, which increases the risk of underfunding and  cost overruns. Having cost, schedule, and technical information organized  by the WBS hierarchical structure would allow the Bureau to summarize  data, provide valuable information at any phase of the program, and assess  progress against the cost estimate plan. This would help keep program  status current and visible, so that risks could be managed or mitigated  quickly. Without a WBS, it is difficult (if not impossible) for Bureau  managers to analyze the causes of cost, schedule, or technical problems  and choose an optimum solution to fix the problems. As part of the new  DBiT system, the Bureau expects to have the capacity to develop a WBS  that includes a data dictionary that defines variables, key terms, and  categories.
section_title: The 2010 Census Life Cycle Cost Estimate Is Not Accurate
paragraphs: The Bureau’s 2010 Census life cycle cost estimate is not accurate because  it does not reflect an important change to a key assumption that affects  cost. The assumption for productivity during address canvassing that was  in the Bureau’s fiscal year 2009 President’s Budget life cycle cost estimate  of $11.546 billion did not reflect recent productivity data from last year’s  address canvassing dress rehearsal. According to the Bureau, the 2010  Census cost model initially assumed productivity for address canvassing  to be 25.6 addresses per hour for urban/suburban areas. However, results  from the 2008 address canvassing dress rehearsal showed productivity of  13.4 addresses per hour for urban/suburban areas. According to the 2009  President’s Budget request, the life cycle cost estimate did increase by   $20 million, but this increase was attributed to other factors and not to  lower-than-expected address canvassing productivity.
An estimate is accurate when it is based on an assessment of the costs  most likely to be incurred. When costs change, best practices require cost  model assumptions to be updated as new information becomes available.  Although the Bureau assessed productivity for the address canvassing  operation, it is not clear why the cost estimate was not updated. A senior  Bureau official confirmed that the estimate had not been updated but was  now being updated to reflect changes in assumptions. It is important that  as part of the replan, the Bureau update assumptions for productivity. The  Bureau also expects to update assumptions for the number of hours field  staff may work in a given week. The model assumes 27.5 hours per week,  but the Bureau now expects this to be 18. This will make it necessary to  hire more workers and, therefore, procure more HHCs.
As a result of not updating the cost estimate to reflect an expected  decrease in productivity, the cost estimate for the 2010 address canvassing  operation in the fiscal year 2009 President’s Budget is understated. As part  of our review, we updated the assumption for address canvassing  productivity in the 2006 version of the cost model that was provided to us  by the Bureau to reflect the productivity data for the number of addresses  completed per hour from the dress rehearsal. Updating this productivity  assumption resulted in a significant increase of approximately $270 million  to the cost of the address canvassing operation.
Further, the Bureau cannot readily demonstrate the accuracy of its cost  estimates because it does not maintain historical data, which include  previous versions of the estimate, in a centralized, standard format that is  readily available. We requested all documentation for the life cycle cost  estimate, including support for the initial estimate created in 2001 and for  updates in 2003, 2005, 2006, and 2007. However, the Bureau did not have  previous versions of the estimate available for analysis. Best practices for  ensuring an accurate cost estimate call for historical data to be maintained  for evaluation purposes, documenting lessons learned, and informing  future cost estimates. Bureau officials told us that the Bureau maintains  historical cost data in data warehouses that are separate from the cost  model, and that this information was not easily accessible. Not having  historical data readily available in a standardized, accessible format  hampers the Bureau’s ability to track and evaluate changes in the cost  estimate over time, document lessons learned, and inform future cost  estimates.
section_title: The 2010 Census Life Cycle Cost Estimate Is Not Credible
paragraphs: The Bureau has not carried out analyses that would demonstrate that its  life cycle cost estimate for the 2010 short form Census is credible.  Specifically, Bureau officials told us that the Bureau has not conducted  formal sensitivity analysis to fully assess how sensitive the short form cost  estimate is to changes in key assumptions and parameters. The Bureau  also has not conducted uncertainty analysis to quantify the uncertainty of  its short form cost estimate or provide a level of confidence associated  with the point estimate. Finally, the Bureau did not have the 2010 short  form Census life cycle cost estimate validated through an independent  cost estimate.
Cost estimates are credible when major assumptions have been varied and  other outcomes recomputed to determine how sensitive outcomes are to  changes in the assumptions, when risk and uncertainty analyses have been  performed to determine the level of risk associated with the estimate, and  when the estimate’s results have been cross-checked and an independent  cost estimate has been developed to determine whether other estimating  methods produce similar results. Sensitivity analysis should be included in  all cost estimates as a best practice because all estimates have some  uncertainty. A sensitivity analysis addresses some of the estimating  uncertainty by testing discrete cases of assumptions and other factors that  could change. By examining each assumption or factor independently,  while holding all others constant, the cost estimator can evaluate the  results to discover which assumptions or factors most influence the  estimate. However, because many parameters could change at the same  time, uncertainty analysis should also be performed to capture the  cumulative effect of additional risks. Uncertainty analysis adds to the  credibility of a cost estimate because it quantifies the uncertainty and  provides a level of confidence associated with the point estimate. The  results of a high-quality, reliable cost estimate should also be cross- checked, and an independent cost estimate should be developed to  determine whether other estimating methods produce similar results. An  independent cost estimate is considered to be one of the most reliable  validation methods. An independent cost estimate is typically performed  by organizations higher in the decision-making process than the office  performing the baseline cost estimate, using different estimating  techniques and, where possible, different data sources from those used to  develop the baseline cost estimate.
Bureau officials told us that while staff have not conducted formal  sensitivity analysis, they have carried out some “what-if” analysis to assess  the impact of changes to some assumptions. According to Bureau officials,  DBiT, which is under development, will provide the Bureau with the  capability to perform sensitivity and uncertainty analyses for the life cycle  cost estimate for the decennial census in the future, although officials did  not confirm that the Bureau plans to do these analyses.
Given the importance of sensitivity and uncertainty analyses for producing  a high-quality cost estimate, we conducted these analyses for the short  form census life cycle cost estimate, using cost model data provided by the  Bureau in November 2006. However, as described in earlier sections, the  Bureau provided incomplete documentation on cost elements and  assumptions included in the short form life cycle cost estimate. As a result,  we would only be able to conduct uncertainty analysis on a portion of the  total life cycle costs of the 2010 Census. We determined that the results of  uncertainty analysis conducted on only a portion of the total life cycle  costs would not be meaningful. See appendix I for a more detailed  explanation of the portion of total life cycle costs for which the Bureau  provided information that would permit uncertainty analysis.
Performing sensitivity analysis for the 2010 Census life cycle cost estimate  would help Bureau managers identify and focus on key elements with the  greatest effects on cost and understand the potential for cost growth and  the reasons for it. It could also influence Bureau decisions affecting the  design and operation of the census. Because the Bureau has not  conducted uncertainty analysis, it is unable to provide Congress with a  confidence level for its total 2010 Census life cycle cost estimate.  Performing uncertainty analysis would enable the Bureau to quantify the  risk and uncertainty associated with the cost model; provide a level of  confidence for its cost estimate; and give decision makers perspective on  the potential variability of the cost estimate should facts, circumstances,  and assumptions change. It would also identify the amount of increased  investment needed to reach specific higher levels of certainty and could  help establish a defensible level of contingency reserves. The results of an  independent cost estimate could help validate the Bureau’s 2010 Census  life cycle cost estimate and provide an objective and unbiased assessment  of whether the cost estimate can be achieved, reducing the risk that the  census would be underfunded.
section_title: Unreliable 2010 Census Cost Estimate Does Not Fully Inform Annual Budgets
paragraphs: The Bureau uses the life cycle cost estimate as the starting point for  annual budget formulation. However, the Bureau does not follow best  practices for developing and maintaining the life cycle cost estimate, as  previously described, so annual budget requests based on the cost  estimate are not fully informed. In addition, while the Bureau revises the  life cycle cost estimate based on appropriations received and updated  budget information, the Bureau does not update the cost estimate to  reflect actual costs.
The Bureau uses the 2010 Census life cycle cost estimate to set initial  allocations when preparing the annual budget submission. The decennial  census life cycle cost estimate is the starting point for the budget  formulation process each year. Officials explained that the Decennial  Management Division sends information from the life cycle cost estimate  to the Budget Division, which uses that information to determine program  allocations by subactivity in the “budget call” memo that goes out to  program offices in January or February. However, because the life cycle  cost estimate is not valid and reliable, as described above, budget requests  based on that estimate are not fully informed.
The Bureau updates the 2010 life cycle cost estimate to reflect  appropriations for specific fiscal years but does not update the cost  estimate with actual costs as they take place over the decade. According  to Bureau officials, they continually revise the life cycle cost estimate  based on changes resulting from the Commerce and Office of Management  and Budget (OMB) passback processes, and once they submit the budget  to Congress, they revise the life cycle cost estimate to match what is in the  budget submission, including outyears. Bureau 2010 Census life cycle cost  estimate documents for 2003, 2005, 2006, and 2007 contain tables showing  enacted appropriations figures for past years and text explanations that  the updated estimates reflect actual appropriations and submitted budget  requests. However, Bureau officials said that the Bureau does not analyze  the accuracy of the life cycle cost estimate each year, such as by  comparing the estimate to actual costs at the end of the year, or update the  estimate with actual costs.
A high-quality cost estimate is the foundation of a good budget. A major  purpose of a cost estimate is to support the budget process by providing  an estimate of the funding required to efficiently execute a program.  Because most programs do not remain static but tend to evolve over time,  developing a cost estimate should not be a onetime event but rather a  recurrent process. Our Cost Assessment Guide explains that a cost  estimate should be a “living” document that is continually updated as  actual costs begin to replace original estimates, so that it remains relevant  and current. Effective program and cost control requires ongoing revisions  to the cost estimate and budget.
When we asked why the Bureau does not update the 2010 life cycle cost  estimate with actual cost data, a budget official told us that actual cost  information can be incurred over multiple fiscal years, and it would be  difficult to compare this information to the annual framework used for the  cost estimate. For budget purposes, the Bureau updates the life cycle cost  estimate every year based on appropriations figures instead of actual cost  data, because appropriations data are attributed to single fiscal years and  are easier to work with than actual cost data.
Using a reliable life cycle cost estimate to formulate the budget could help  the Bureau ensure that all costs are fully accounted for so that resources  are adequate to support the program. Credible cost estimates could also  help the Bureau effectively defend budgets to a department secretary,  OMB, or Congress. In addition, the Bureau could use the cost estimate to  help determine how budget cuts might hinder the census program’s  progress or effectiveness. Moreover, because the Bureau does not update  the life cycle cost estimate with actual cost data, the Bureau will not have  the ability to keep the estimate current or document lessons learned for  cost elements whose actual costs differ from the estimate.
Concerns about the soundness of the life cycle cost estimate and the  quality of annual budgets related to the 2010 Census are particularly  important because the bulk of funds will be spent from fiscal years 2009  through 2013, as shown in table 1.
section_title: The Bureau Has Insufficient Policies and Inadequately Trained Staff for Conducting High- Quality Cost Estimation; However, a New System May Help Improve the Bureau’s Cost Estimation Practices
paragraphs: The Bureau has insufficient policies and inadequately trained staff for  conducting high-quality cost estimation. The Bureau does not have formal  cost estimation policies and procedures. The Bureau also does not have  skilled cost estimators or a centralized office dedicated to cost estimation.  According to Bureau officials, although multiple staff members with  various census backgrounds and experiences from across 25 divisions  develop information for use in the cost estimate, we found that staff are  not adequately trained in cost estimation. Bureau officials told us that  most of the managers have project management certificates or training,  which includes classes in cost estimation. However, the classes were  designed more to provide program management with a general  understanding of cost estimation rather than to provide in-depth training  to the actual cost estimators.
In order to consistently develop reliable cost estimates, it is important for  an agency to have defined policies and procedures to govern the process.  Cost estimating best practices were developed to help agencies establish  appropriate policies and procedures for producing estimates that adhere  to the characteristics of high-quality cost estimation. An agency’s cost  assessment team should include members who are experienced and  trained in conducting cost estimation. Further, centralizing the cost  estimating team and process (including cost analysts working in one group  but supporting many programs) represents a cost estimating team best  practice. Since the experience and skills of the members of a cost  estimating team are important, some organizations have chosen to  establish training programs and certification procedures. For example, the  Department of Defense (DOD) established the Defense Acquisition  University (DAU), which provides basic, intermediate, and advanced  certification training as well as continuous learning opportunities.  Although DAU’s primary mission is to train DOD employees, all federal  employees are eligible to attend, including Bureau employees.
Bureau officials understand that the Bureau’s cost estimation process  needs to be improved and are currently developing DBiT, a budget  management tool that will support the process underlying the generation  of cost estimates by facilitating, managing, and documenting changes in  variables and assumptions that support the cost estimates. If properly  implemented, DBiT should secure, standardize, and consolidate budget  information and enable the Bureau to maintain better documentation for  cost estimation. Further, DBIT is supposed to enhance access to budget  data and increase the ability to model, formulate, execute, and report the  decennial census budget. Officials told us that DBiT would have the ability  to download data from the budget database, thus facilitating linkages  between the life cycle cost estimate and budget preparation processes.  Bureau officials also said that the Cost and Progress system, which tracks  the actual cost of operations, would not link to DBiT. Without this  capability, the Bureau will not be able to systematically update the  estimate with actual costs. However, Bureau officials indicated that they  might consider linking the two systems in the future.
Bureau officials also said that DBiT will enable the Bureau to save  different versions of the cost model and will provide them with the  capability to use software packages such as Crystal Ball to perform  sensitivity and uncertainty analyses on its estimates. However, officials did  not assert their intention to conduct these analyses. DBiT is being  incrementally implemented and will not be fully functional until after 2010.
While DBiT should improve the Bureau’s systems for developing budgets  and the life cycle cost estimate for the 2020 Census, the Bureau will still  need established policies and procedures for conducting cost estimation  and skilled estimators. Policies and procedures to govern the process as  well as a dedicated office that is supported by properly trained staff are  the foundation for a reliable cost estimate. Not having the tools and people  in place for the 2010 Census has impeded the Bureau’s ability to produce a  sufficiently documented, comprehensive, accurate, and credible cost  estimate.
section_title: Conclusions
paragraphs: On April 3, 2008, the Secretary of Commerce presented a redesigned 2010  Census plan with significant cost increases. However, until the Bureau  makes fundamental changes to how it estimates and updates cost  information, uncertainties about the ultimate cost of the 2010 Census will  remain. The Bureau’s ability to produce well-documented, comprehensive,  accurate, and credible cost estimates for the 2010 and future decennial  censuses and its ability to effectively manage operations and contain costs  will continue to be hampered unless improvements are made to its cost  estimation processes and systems. Specifically, without full  documentation of the data sources, assumptions, and calculation methods  the Bureau uses, the 2010 life cycle cost estimate cannot be validated, nor  can the Bureau understand and explain differences between estimated and  actual costs—an important step in improving future cost estimates. Also,  without updating assumptions for the 2010 life cycle cost estimate and  making clear what the underlying assumptions are, the Bureau cannot  ensure that it is providing the most up-to-date and accurate cost estimates  to OMB and Congress. Further, without conducting sensitivity and  uncertainty analyses on the 2010 life cycle cost estimate, the Bureau is  unable to identify and focus on major cost drivers, understand the  potential for cost growth, and quantify the risk and uncertainty associated  with the cost estimate. Finally, without established policies and  procedures and qualified staff, the Bureau’s ability to produce high-quality  cost estimates for the 2010 Census and future censuses will be limited.
Along with the new cost estimate, the Secretary of Commerce outlined  major changes to the 2010 Census design. Changes this late in the decade  significantly increase the risk to the success of the 2010 Census. These  changes, during an era of serious national budget challenges, make it  important for Bureau managers to efficiently manage the new design in  order to contain costs. Furthermore, careful monitoring and oversight by  Commerce, Congress, and other key stakeholders are more critical than  ever.
section_title: Recommendations for Executive Action
paragraphs: To improve the Bureau’s life cycle cost estimates for the decennial census,  we recommend that the Secretary of Commerce direct the U.S. Census  Bureau to take the following five actions:  1.  To improve the quality and transparency of the Bureau’s 2010 Census  life cycle cost estimate, assist the Bureau in managing costs during  design revisions resulting from problems with the HHCs, and help  establish a sound basis for the 2020 Census cost estimate, the Bureau  should thoroughly document the 2010 Census life cycle cost estimate.  Specifically, documentation should be maintained in a centralized  standard format and specify all data sources, assumptions, calculation  methods, and cost elements used to prepare the 2010 cost estimate.  2.  To ensure that the life cycle cost estimate reflects current information,  the Bureau should update assumptions as appropriate, including  updating productivity assumptions to reflect results from the address  canvassing dress rehearsal. The Bureau should also document the  basis for prior and future changes made to assumptions used in the life  cycle cost estimate.  3.  To keep the life cycle cost estimate current and to document lessons  learned for cost elements whose actual costs differ from the estimate,  the Bureau should update the estimate to reflect actual costs.  4.  To improve the quality of and provide a confidence level for the 2010  Census life cycle cost estimate, the Bureau should perform sensitivity  and uncertainty analyses on the estimate.  5.  To help ensure that the Bureau produces a reliable, high-quality life  cycle cost estimate for the 2020 decennial census, the Bureau should  establish guidance, policies, and procedures for conducting cost  estimation that would meet best practices criteria and ensure that it  has staff resources qualified in cost estimation.
section_title: Agency Comments and Our Evaluation
paragraphs: The Secretary of Commerce provided written comments on a draft of this  report on May 23, 2008. The comments are reprinted in appendix II.  Although Commerce raised concerns about how GAO characterizes the  accuracy of the estimate, Commerce stated that it agrees with many of our  findings, and that the Bureau will prepare a formal action plan to  document specific steps (with estimated completion dates) it will take in  response to the recommendations. Commerce stated that the Bureau will  determine if it will be possible to make improvements in the short term to  its cost estimates and methods. Further, Commerce made some  suggestions where additional context or clarification was needed and  where appropriate we made those changes.
Commerce commented that our report, in its discussion of increasing  census costs, does not mention other significant factors that have  contributed to substantial cost increases over the last 40 years. We agree  and have clarified in our report that other factors have contributed to  increased costs, such as accommodating more complex households,  busier lifestyles, more languages and greater cultural diversity, and  increased privacy concerns.
Commerce also stated that it would have been premature to include the  cost of fingerprinting temporary workers in the 2010 Census life cycle cost  estimate. However, best practices state that having a realistic estimate of  projected costs makes for effective resource allocation and increases the  probability of a program’s success. To be prudent and conservative, an  agency should include possible program costs that may have an impact on  the overall life cycle cost estimate. We appropriately characterize the cost  for fingerprinting temporary workers for the 2010 Census by stating that it  is a “potential” cost. Also, understanding that the life cycle cost estimate  is, in fact, an estimate, the draft report states that “there is risk that the life  cycle cost estimate has been substantially understated.” We therefore  made no change to the report.
In commenting on our description of how the Bureau updates the estimate  to reflect costs, Commerce stated that it believes that for budget purposes,  using enacted appropriations is the best way to adjust the life cycle cost  estimate. However, best practices require that an estimate be updated to  reflect actual costs when a difference occurs. This enables an agency to  determine the precise reasons why actual costs differ for the estimate and  document lessons learned. Because the draft report already reflected the  Bureau’s practice of using enacted appropriations, we made no changes.
Commerce did not agree with our statement that the 2010 life cycle cost  estimate is not accurate as it relates to assumed productivity rates for the  fiscal year 2009 address canvassing operation. Commerce stated that at  the time of GAO’s review of the life cycle cost estimate, the Bureau had  not completed its analysis of the dress rehearsal productivity data.  However, productivity data from the address canvassing dress rehearsal  were provided to us in December 2007. These productivity data were  significantly different from the assumptions used in the life cycle cost  model. The updated productivity assumptions should have been included  in the estimate that was included in the fiscal year 2009 President’s Budget  request, which was issued in February 2008.
In addressing our recommendations, Commerce generally agreed with our  five recommendations, and indicated that the Bureau will prepare specific  steps in response to the recommendations. Commerce stated that as part  of the Bureau’s action plan it would examine GAO’s Cost Assessment  Guide to determine if it would be possible to make improvements in the  short term to its cost estimate and methods. Commerce further noted that  the Bureau already has efforts under way to improve future cost  estimation methods and systems through the development of DBiT. In  addition, Commerce acknowledged that at the lower level of detail  estimates in the model, the Bureau will need to develop both a way to  input costs in the model (or the DBiT system) in time to inform estimates  for the 2020 cycle, and a way to update this information regularly over the  coming decade. Further, Commerce noted that the Bureau was updating  the life cycle cost estimate to reflect revised assumptions for the address  canvassing operation.
In response to our recommendation to perform sensitivity and uncertainty  analyses on the estimate, Commerce said that the Bureau would use the  Cost Assessment Guide and seek detailed information on the experiences  of other agencies that have used sensitivity analysis. Commerce further  commented that while it does not disagree that conducting sensitivity  analysis and providing a possible range of costs would be useful to  external audiences, the Bureau must submit budget requests and out-year  estimates as fixed amounts rather than ranges. We understand that fixed  amounts rather than ranges are submitted for budget requests. However,  conducting sensitivity analysis on the cost estimate would be beneficial to  the Bureau in its management of costs associated with the census, not just  to external audiences. Sensitivity analysis provides valuable information to  an agency about which assumptions or factors have the biggest effects on  cost. Further, the Bureau should also conduct uncertainty analysis to  quantify the overall uncertainty of the cost estimate and provide a level of  confidence associated with the point estimate. Providing a range of costs  around a point estimate is useful to decision makers because it conveys  the level of confidence in achieving the most likely cost, and uncertainty  analysis can also help managers identify a defensible level of contingency  funding needed to reach a desired confidence level. The Bureau should  ensure that it conforms to best practices by making sensitivity and  uncertainty analyses part of required processes for cost estimation.
In response to our recommendation to establish guidance, policies, and  procedures for conducting cost estimation and ensure that it has staff  resources qualified in cost estimation, Commerce stated that the Bureau  has efforts under way to improve future cost estimation methods and  systems through the development of DBiT. However, these efforts will not  be completed in time to be used in preparing the 2010 budget request—an  effort already underway.  Commerce further stated that the Bureau would  examine the Cost Assessment Guide to determine if it will be possible to  make improvements in the short term to the estimate and methods,  including possibly hiring additional skilled cost estimators.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to other  interested congressional committees, the Secretary of Commerce, the  Director of the U.S. Census Bureau, and the Director of the Office of  Management and Budget. Copies will be made available to others upon  request. This report will also be available at no charge on GAO’s Web site  at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-6806 or sciremj@gao.gov. Key contributors to this  report were Vidhya Ananthakrishnan, Thomas Beall, Jennifer Echard,  Richard Hung, Anne Inserra, Jason Lee, Andrea Levine, Donna Miller, Lisa  Pearson, Michelle Petre, Sonya Phillips, Karen Richey, John Sperry, Niti  Tandon, Shannon VanCleave, and Michael Volpe.  Contact points for our  Office of Congressional Relations and Public Affairs may be found on the  last page of this report.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this report were to (1) assess the extent to which the  U.S. Census Bureau’s (Bureau) 2010 Census life cycle cost estimate  adheres to characteristics defined for high-quality cost estimation,   (2) report on the relationship between the life cycle cost estimate and the  Bureau’s budget, and (3) assess whether the Bureau’s existing policies and  resources are sufficient to conduct high-quality cost estimation.
To address our first objective, we evaluated the Bureau’s 2010 Census life  cycle cost estimate to determine whether it met key characteristics  identified in the exposure draft of our Cost Assessment Guide. Our guide,  which is based on extensive research of best practices for estimating  program schedules and costs, states that a high-quality, valid, and reliable  cost estimate should be well documented, comprehensive, accurate, and  credible. We obtained and analyzed the version of the 2010 Census short  form cost model that was given to us in November 2006. (The Bureau  provided a cost model containing data for fiscal years 2009 through 2013,  and separate models for fiscal years 2007 and 2008. The models, which  were in Excel, did not include costs for fiscal years 2001 through 2006.)  We also analyzed Bureau documents related to the 2010 Census life cycle  cost estimate and cost estimates for previous censuses. We interviewed  Bureau officials from the Decennial Management and Budget Divisions  about the process used to prepare the life cycle cost estimates and the  assumptions used to prepare the estimates. We shared the Cost  Assessment Guide and the criteria against which we would be evaluating  the Bureau’s cost estimate with Bureau officials.
To assess the adequacy of the Bureau’s 2010 Census life cycle cost  estimate, we compared the Bureau’s methods and approaches for  preparing the estimate with the guidance contained in our Cost  Assessment Guide. We assessed whether the 2010 estimate met the four  desired characteristics of being well documented, comprehensive,  accurate, and credible. Given those criteria, the main purpose of this  objective was to assess the reliability of the cost estimate. We assessed the  extent to which the entire life cycle cost estimate was well documented,  including the short form cost estimate, and the estimates for the American  Community Survey (ACS) and the Master Address File/Topologically  Integrated Geographic Encoding and Referencing (MAF/TIGER) system.  Our analysis for the other three characteristics—comprehensiveness,  accuracy, and credibility—was limited to the short form cost estimate, due  to lack of available documentation on the ACS and MAF/TIGER  components of total life cycle costs.
As part of our assessment of the credibility of the Bureau’s 2010 Census  life cycle cost estimate, we conducted sensitivity analysis to identify  significant cost drivers and limited uncertainty analysis on a portion of the  2006 version of the short form cost estimate that the Bureau provided.  Portions of the total $11.3 billion life cycle cost estimate were excluded  from the uncertainty analysis, as detailed below and shown in figure 5:    $1.7 billion of the total had already been spent from fiscal years 2001  through 2007 (these sunk costs had to be excluded from the  uncertainty analysis);    $1.2 billion of the total was allocated to costs for MAF/TIGER   ($0.1 billion) and ACS ($1.1 billion) for fiscal years 2008 through 2013  (since these costs were estimated separately and not included in the  short form cost model, they were excluded from the uncertainty  analysis);    $0.6 billion of the total represented estimated fiscal year 2008 costs for  the short form Census, and the Bureau gave us estimated fiscal year  2008 costs in a separate model; and    $2.3 billion of the total consisted of costs that the Bureau did not model  but instead just provided as “throughput” or fixed costs.
Subtracting the above components from the total left $5.4 billion to be  analyzed. We determined that the results of uncertainty analysis  performed on this limited portion of the total costs would not be  meaningful.
Our analysis of the Bureau’s 2010 life cycle cost estimate was conducted  prior to the redesign of the census and the subsequent revision of the  FDCA program. Detailed information on revised assumptions, cost data,  methods of calculation, or the process used to revise the estimate was not  available for the Bureau’s proposed redesign in time to be analyzed as part  of our scope. Because we did not analyze the Bureau’s updated range for  the total cost estimate of from $13.7 billion to $14.5 billion, we cannot  verify whether the revised estimate is accurate, valid or reliable.
To address the second objective, we reviewed policies and procedures for  preparing annual budgets. We analyzed Bureau documents related to the  budget preparation system and the process for preparing the decennial  census life cycle cost estimate, including the Budget Formulation and  Performance Planning Manual, and internal correspondence from the  Budget Division to other Bureau offices concerning annual budget  preparation. We also analyzed Bureau budget estimates as presented to  Congress, life cycle cost estimate documents, and worksheets on life cycle  costs prepared for budget formulation, and confirmed that appropriations  figures for completed fiscal years did appear in subsequent life cycle cost  estimate documents. Further, we interviewed officials from the Bureau’s  Budget and Decennial Management Divisions about the relationship  between the cost estimate and the annual budget process. We did not  independently verify budget information provided by the Bureau because  that would have been outside the scope of our review.
To address our third objective, we analyzed Bureau documents related to  the life cycle cost estimate. We compiled information on the Bureau’s  process for developing its decennial cost estimate and the credentials of  the Bureau’s cost estimation staff. We requested the Bureau’s policies and  procedures for cost estimation, but determined that the Bureau does not  have specific policies and procedures for cost estimating. We evaluated  Bureau information against the best practices criteria presented in our  Cost Assessment Guide for developing a high-quality cost estimate and  designating an experienced, well-trained cost estimation team. We also  interviewed Bureau officials.
To determine what steps the Bureau is taking to improve its cost  estimation practices, we attended a demonstration of the Decennial  Budget Integration Tool, the new Bureau automated budget system  currently being developed that should enable the Bureau to produce better  cost estimates. We also interviewed Bureau officials.
We conducted our work from October 2006 to June 2008 in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on our  audit objectives.
section_title: Appendix II: Comments from the Department of Commerce