2010 Census: Census Bureau Should Take Action to Improve the Credibility and Accuracy of Its Cost Estimate for the Decennial Census
Released: 2008-07-16T00:00:00

Letter

Background

The Bureau’s cost projections for the 2010 decennial census continue an escalating trend, with the 2010 Census currently estimated to cost approximately $13.7 billion to $14.5 billion. At a March 2008 hearing, Commerce and the Bureau stated that the FDCA program was likely to incur significant cost overruns and that a redesigning effort was under way to try to get the decennial census back on track. In early April, the Secretary of Commerce chose from among several alternatives for redesigning the FDCA program, and elected an option that involves dropping the HHCs from the nonresponse follow-up operation. Additionally, he decided that the Bureau would reduce deployment of field technology infrastructure by the contractor and have the contractor provide HHCs for address canvassing and develop the information system for controlling field operations. The Bureau has estimated that, with the redesign option, the total life cycle cost estimate for the 2010 Census will be from $13.7 billion to $14.5 billion.

Prior to this recent, major redesign of the FDCA program, the Bureau had estimated the life cycle cost of the 2010 Census to be $11.8 billion (in constant 2010 dollars). As shown in figure 1, this estimate of $11.8 billion represented a more than tenfold increase over the $1 billion spent on the 1970 Census. Although some of the cost increase could be expected because the number of housing units—and hence the Bureau’s workload— has increased, the cost growth far exceeded this increase. Factors contributing to the increased costs include an effort to accommodate more complex households, busier lifestyles, more languages and greater cultural diversity, and increased privacy concerns. The Bureau estimated that the number of housing units for the 2010 Census will increase by almost 14 percent over 2000 Census levels (from 117.5 million to 133.8 million housing units). At the same time, the average cost per housing unit for 2010 was expected to increase by approximately 26 percent over 2000 levels, from $69.79 per housing unit to $88.19 per housing unit (see fig. 2).

The bulk of total life cycle funds for the 2010 Census has yet to be spent. As shown in table 1, which reflects the Bureau’s life cycle cost estimate for the 2010 Census that was released in September 2007, the majority of spending will occur from fiscal year 2009 through fiscal year 2013.

In reengineering the 2010 Census, the Bureau has four goals: to (1) improve the relevance and timeliness of census long form data, (2) reduce operational risk, (3) increase the coverage and accuracy of the census, and (4) contain costs. To achieve these goals, three new components are key to the Bureau’s plans for 2010: modernizing and enhancing the nation’s road map through the MAF/TIGER enhancement program, which includes realigning the TIGER map to take advantage of global positioning system capabilities, modernizing the processing system, and expanding geographic partnerships; replacing the census long form questionnaire with a more frequent sample survey, the ACS; and conducting a short-form-only census using automation to collect census data.

Prior to 2010, the decennial census collected data using both a short and long form questionnaire: the short form counted the population, and the long form obtained demographic, housing, social, and economic information from a 1-in-6 sample of households. The ACS is a nationwide survey that is replacing the decennial long form in the reengineered 2010 Census. The ACS, which was first implemented in 2005, collects detailed characteristics data every year throughout the decade using a large household survey. The ACS has allowed the Bureau to simplify the 2010 Census since it will now only involve the short form.

In June 2001, the Bureau issued a document describing the process for reengineering the 2010 Census, which included a life cycle cost estimate for the 2010 decennial census, projected at $11.28 billion. Since June 2001, the Bureau has updated the life cycle cost estimate for the reengineered 2010 Census and issued formal documents describing the estimate in June 2003, September 2005, June 2006, and September 2007. In February 2008, the Bureau’s Budget Estimates as Presented to Congress, Fiscal Year 2009 included a new life cycle cost estimate for the 2010 Decennial Census of $11.546 billion, an increase of $20 million over the previous year’s estimate. As described above, in April 2008, the Secretary of Commerce testified that the current estimate for the 2010 Census is from $13.7 billion to $14.5 billion.

The 2001 through 2007 life cycle cost documents issued by the Bureau illustrate that the costs of the three components—the ACS, MAF/TIGER modernization, and short-form-only census—fluctuated slightly over that period. The total life cycle costs ranged from a low of $11.2546 billion in the September 2005 and June 2006 documents to a high of $11.5264 billion in the September 2007 document. Figure 3 depicts how the estimated life cycle cost for the 2010 Census fluctuated in Bureau documents from 2001 to 2007. This year, however, the Bureau estimates that the life cycle cost of the census will increase by up to $3 billion—dwarfing all previous increases.

In January 2004, we reported that the Bureau’s approach had the potential to achieve the first three goals for reengineering the 2010 Census, although reducing operational risk could prove to be difficult because each of the three components actually introduced new risks, and the Bureau would be challenged to control the cost of the 2010 Census. To manage the 2010 Census and contain costs, we recommended in our 2004 report that the Bureau develop a comprehensive, integrated project plan for the 2010 Census that should include itemized estimated costs of each component, including a sensitivity analysis, and an explanation of significant changes in the assumptions on which these costs are based. In response, the Bureau provided us with the 2010 Census Operations and Systems Plan, dated August 2007. This plan represented an important step forward at the time. It included inputs and outputs and described linkages among operations and systems. However, it did not include sensitivity analysis, risk mitigation plans, a detailed 2010 Census timeline, or itemized estimated costs of each component. With the redesign, this plan will need to be updated.

The Census Bureau’s 2010 Census Life Cycle Cost Estimate Is Not Reliable

The 2010 Census Life Cycle Cost Estimate Is Not Adequately Documented

The Bureau’s life cycle cost estimate for the 2010 Census is not adequately documented. The Bureau could not provide adequate documentation to support the costs of two of the three components of the total life cycle cost estimate, the ACS and MAF/TIGER components, which together accounted for close to 20 percent of total life cycle costs in the 2007 life cycle cost estimate. Figure 4 shows the three major components of the total life cycle cost of $11.5 billion as of August 2007.

Costs for the ACS and MAF/TIGER are estimated outside of the cost model that is used for the 2010 short form census. According to Bureau officials, they were not able to locate specific documentation for how the Bureau first calculated the costs for those components of the 2010 Census life cycle cost estimate in 2001, although they explained that experienced staff derived the estimates based on professional expertise, similar surveys, comparative costs for similar projects, and other factors.

We requested documentation that supported the assumptions for the initial 2001 life cycle cost estimate as well as the updates, but Bureau officials were unable to demonstrate that support existed. The Bureau did not provide detailed documentation for data sources and significant assumptions used in estimating the cost of the short form 2010 Census. For example, the Bureau’s first document describing the life cycle cost estimate for the 2010 Census, which was issued in June 2001, includes a number of assumptions for expected areas of savings that would be achieved by the reengineered design. One assumption is that the follow-up workload for the Local Update of Census Addresses (LUCA) program will be reduced by 25 percent as a result of updating the address list throughout the decade and using HHCs to reduce time and travel costs. However, the document does not fully describe how the 25 percent reduction was calculated or what the associated savings might be.

Similarly, the Bureau did not provide detailed documentation for the updated life cycle cost estimate for the 2010 short form that was issued in 2003, which includes several changes to assumptions. For example, the Bureau describes how increases of $128 million have been made to the life cycle cost estimate based on new requirements, including the need for more research on ways to reduce duplicate enumerations, more testing and development on how to identify and enumerate special places/group quarters, and additional efforts to research and test methods for enumerating Americans overseas. However, the document does not describe how the $128 million increase was derived, the costs associated with each of the new requirements, or the data sources and calculation methods used.

Cost estimates are well documented when they can be easily repeated or updated and can be traced to original sources. The documentation should explicitly identify the primary methods, calculations, assumptions, and sources of the data used to generate each cost element. The estimating process should be described and an explanation provided for why particular methods and data sets were chosen and why these choices are reasonable. All the steps involved in developing the estimate should be documented so that a cost analyst unfamiliar with the program could re- create it with the same result. In addition, documentation for the cost estimate should reflect changes in technical or program assumptions or new program phases or milestones.

Officials acknowledged that the Bureau does not have a centralized location in which to keep detailed documentation to support the assumptions and decision-making process. Further, Bureau officials stated that there is no written documentation of the process used by management to agree upon particular assumptions for use in the life cycle cost estimate, and that there is no systematic documentation regarding management decisions to make changes to the cost model or life cycle cost estimates. According to an official, changes to the cost model have not always been well documented, and sometimes a decision memo is created to justify a change but not always. Best practices call for management approval of the cost estimate. Management approval of the cost estimate should also be documented, including management approval memorandums or recommendations for change, as well as management feedback.

Lack of documentation was also identified as an issue for the 2000 Census life cycle cost estimate. An independent assessment of the prior Census 2000 life cycle cost model that was issued in 1997 found that input data and assumptions used in the 2000 model typically came from other offices and were based on historical observation, professional judgment, or both. The assessment concluded that the Bureau did not have documentation readily available for external use on the underlying basis of the 2000 cost estimate, input assumptions, or process characteristics.

As a result of insufficient documentation, the validity and reliability of the Bureau’s life cycle cost estimate for the 2010 Decennial Census cannot be verified. Not having adequate documentation also impedes the Bureau’s ability to support the decennial decision-making process, inform future estimates, and facilitate oversight by Congress.

Bureau officials indicated that they understand that the Bureau’s cost estimation process needs to be improved and are currently developing DBiT, which should improve documentation for the cost estimates. DBiT, which is described in more detail later in this report, is being incrementally implemented and will not be fully functional until after 2010.

The 2010 Census Life Cycle Cost Estimate Is Not Comprehensive

The 2010 Census life cycle cost estimate is not comprehensive because the Bureau did not include all potential costs or clearly define some of the cost elements in the model for the short form Census. In February 2008, Commerce’s Inspector General (IG) reported that the fiscal year 2009 budget request did not include the cost to fingerprint some 900,000 temporary workers to be hired for the 2010 Census. In addition, the Bureau provided documentation that the life cycle cost estimate did not include the cost to fingerprint temporary workers. A cost estimate is comprehensive when it accounts for all possible costs. According to the IG, the Bureau has not included the cost for fingerprinting temporary staff. After discussions with Commerce, the Office of Personnel Management, and the Federal Bureau of Investigation, the Bureau decided to require fingerprinting. In the previous two decennials, the Bureau obtained criminal history records for temporary decennial applicants using only a name check. The Bureau estimates that the cost to fingerprint all temporary staff could be about $450 million. Thus, there is a risk that the life cycle cost estimate has been substantially understated.

The Bureau also did not have detailed descriptions for some of the cost elements in the model or explanations of how individual cost elements were related. For example, when we analyzed the fiscal year 2006 version of the cost model for the short form component of the 2010 Census, we found that $2.7 billion in the Other Objects category were fixed dollar amounts. The Bureau did not provide detailed descriptions, equations, or support to explain how individual cost elements within the Other Objects category were produced or related. Bureau officials stated that most of the costs in this category were associated with contracts, travel, supplies, and training. Further, the Bureau does not have a work breakdown structure (WBS) that clearly identifies and defines all costs contained in the short form cost estimate. A WBS defines the work necessary to accomplish a program’s objectives.

A comprehensive cost estimate’s level of detail ensures that all pertinent cost elements are included and that no costs are double counted. A comprehensive cost estimate also includes a clearly defined WBS. A WBS reflects the requirements, resources, and tasks that must be accomplished to develop a program. The WBS should have a dictionary that defines each cost element and how it relates to others, clearly describes what is and is not included in each element, describes resources and processes necessary to produce the element, and links each element to other relevant technical documents. A WBS clearly defines the logical relationship of all program elements and provides a systemic and standardized way for collecting data, communicates to everyone what needs to be done and how activities relate to one another, and is an essential part of developing a cost estimate for a program.

When we asked for documentation for the cost elements in the Other Objects category, the Bureau was not able to provide support for all the cost elements and agreed that the documentation supporting cost elements in the model was not clear. Without clearly defined cost elements or a well-developed WBS, the Bureau cannot be sure that the cost estimate captures all relevant costs, which increases the risk of underfunding and cost overruns. Having cost, schedule, and technical information organized by the WBS hierarchical structure would allow the Bureau to summarize data, provide valuable information at any phase of the program, and assess progress against the cost estimate plan. This would help keep program status current and visible, so that risks could be managed or mitigated quickly. Without a WBS, it is difficult (if not impossible) for Bureau managers to analyze the causes of cost, schedule, or technical problems and choose an optimum solution to fix the problems. As part of the new DBiT system, the Bureau expects to have the capacity to develop a WBS that includes a data dictionary that defines variables, key terms, and categories.

The 2010 Census Life Cycle Cost Estimate Is Not Accurate

The Bureau’s 2010 Census life cycle cost estimate is not accurate because it does not reflect an important change to a key assumption that affects cost. The assumption for productivity during address canvassing that was in the Bureau’s fiscal year 2009 President’s Budget life cycle cost estimate of $11.546 billion did not reflect recent productivity data from last year’s address canvassing dress rehearsal. According to the Bureau, the 2010 Census cost model initially assumed productivity for address canvassing to be 25.6 addresses per hour for urban/suburban areas. However, results from the 2008 address canvassing dress rehearsal showed productivity of 13.4 addresses per hour for urban/suburban areas. According to the 2009 President’s Budget request, the life cycle cost estimate did increase by $20 million, but this increase was attributed to other factors and not to lower-than-expected address canvassing productivity.

An estimate is accurate when it is based on an assessment of the costs most likely to be incurred. When costs change, best practices require cost model assumptions to be updated as new information becomes available. Although the Bureau assessed productivity for the address canvassing operation, it is not clear why the cost estimate was not updated. A senior Bureau official confirmed that the estimate had not been updated but was now being updated to reflect changes in assumptions. It is important that as part of the replan, the Bureau update assumptions for productivity. The Bureau also expects to update assumptions for the number of hours field staff may work in a given week. The model assumes 27.5 hours per week, but the Bureau now expects this to be 18. This will make it necessary to hire more workers and, therefore, procure more HHCs.

As a result of not updating the cost estimate to reflect an expected decrease in productivity, the cost estimate for the 2010 address canvassing operation in the fiscal year 2009 President’s Budget is understated. As part of our review, we updated the assumption for address canvassing productivity in the 2006 version of the cost model that was provided to us by the Bureau to reflect the productivity data for the number of addresses completed per hour from the dress rehearsal. Updating this productivity assumption resulted in a significant increase of approximately $270 million to the cost of the address canvassing operation.

Further, the Bureau cannot readily demonstrate the accuracy of its cost estimates because it does not maintain historical data, which include previous versions of the estimate, in a centralized, standard format that is readily available. We requested all documentation for the life cycle cost estimate, including support for the initial estimate created in 2001 and for updates in 2003, 2005, 2006, and 2007. However, the Bureau did not have previous versions of the estimate available for analysis. Best practices for ensuring an accurate cost estimate call for historical data to be maintained for evaluation purposes, documenting lessons learned, and informing future cost estimates. Bureau officials told us that the Bureau maintains historical cost data in data warehouses that are separate from the cost model, and that this information was not easily accessible. Not having historical data readily available in a standardized, accessible format hampers the Bureau’s ability to track and evaluate changes in the cost estimate over time, document lessons learned, and inform future cost estimates.

The 2010 Census Life Cycle Cost Estimate Is Not Credible

The Bureau has not carried out analyses that would demonstrate that its life cycle cost estimate for the 2010 short form Census is credible. Specifically, Bureau officials told us that the Bureau has not conducted formal sensitivity analysis to fully assess how sensitive the short form cost estimate is to changes in key assumptions and parameters. The Bureau also has not conducted uncertainty analysis to quantify the uncertainty of its short form cost estimate or provide a level of confidence associated with the point estimate. Finally, the Bureau did not have the 2010 short form Census life cycle cost estimate validated through an independent cost estimate.

Cost estimates are credible when major assumptions have been varied and other outcomes recomputed to determine how sensitive outcomes are to changes in the assumptions, when risk and uncertainty analyses have been performed to determine the level of risk associated with the estimate, and when the estimate’s results have been cross-checked and an independent cost estimate has been developed to determine whether other estimating methods produce similar results. Sensitivity analysis should be included in all cost estimates as a best practice because all estimates have some uncertainty. A sensitivity analysis addresses some of the estimating uncertainty by testing discrete cases of assumptions and other factors that could change. By examining each assumption or factor independently, while holding all others constant, the cost estimator can evaluate the results to discover which assumptions or factors most influence the estimate. However, because many parameters could change at the same time, uncertainty analysis should also be performed to capture the cumulative effect of additional risks. Uncertainty analysis adds to the credibility of a cost estimate because it quantifies the uncertainty and provides a level of confidence associated with the point estimate. The results of a high-quality, reliable cost estimate should also be cross- checked, and an independent cost estimate should be developed to determine whether other estimating methods produce similar results. An independent cost estimate is considered to be one of the most reliable validation methods. An independent cost estimate is typically performed by organizations higher in the decision-making process than the office performing the baseline cost estimate, using different estimating techniques and, where possible, different data sources from those used to develop the baseline cost estimate.

Bureau officials told us that while staff have not conducted formal sensitivity analysis, they have carried out some “what-if” analysis to assess the impact of changes to some assumptions. According to Bureau officials, DBiT, which is under development, will provide the Bureau with the capability to perform sensitivity and uncertainty analyses for the life cycle cost estimate for the decennial census in the future, although officials did not confirm that the Bureau plans to do these analyses.

Given the importance of sensitivity and uncertainty analyses for producing a high-quality cost estimate, we conducted these analyses for the short form census life cycle cost estimate, using cost model data provided by the Bureau in November 2006. However, as described in earlier sections, the Bureau provided incomplete documentation on cost elements and assumptions included in the short form life cycle cost estimate. As a result, we would only be able to conduct uncertainty analysis on a portion of the total life cycle costs of the 2010 Census. We determined that the results of uncertainty analysis conducted on only a portion of the total life cycle costs would not be meaningful. See appendix I for a more detailed explanation of the portion of total life cycle costs for which the Bureau provided information that would permit uncertainty analysis.

Performing sensitivity analysis for the 2010 Census life cycle cost estimate would help Bureau managers identify and focus on key elements with the greatest effects on cost and understand the potential for cost growth and the reasons for it. It could also influence Bureau decisions affecting the design and operation of the census. Because the Bureau has not conducted uncertainty analysis, it is unable to provide Congress with a confidence level for its total 2010 Census life cycle cost estimate. Performing uncertainty analysis would enable the Bureau to quantify the risk and uncertainty associated with the cost model; provide a level of confidence for its cost estimate; and give decision makers perspective on the potential variability of the cost estimate should facts, circumstances, and assumptions change. It would also identify the amount of increased investment needed to reach specific higher levels of certainty and could help establish a defensible level of contingency reserves. The results of an independent cost estimate could help validate the Bureau’s 2010 Census life cycle cost estimate and provide an objective and unbiased assessment of whether the cost estimate can be achieved, reducing the risk that the census would be underfunded.

Unreliable 2010 Census Cost Estimate Does Not Fully Inform Annual Budgets

The Bureau uses the life cycle cost estimate as the starting point for annual budget formulation. However, the Bureau does not follow best practices for developing and maintaining the life cycle cost estimate, as previously described, so annual budget requests based on the cost estimate are not fully informed. In addition, while the Bureau revises the life cycle cost estimate based on appropriations received and updated budget information, the Bureau does not update the cost estimate to reflect actual costs.

The Bureau uses the 2010 Census life cycle cost estimate to set initial allocations when preparing the annual budget submission. The decennial census life cycle cost estimate is the starting point for the budget formulation process each year. Officials explained that the Decennial Management Division sends information from the life cycle cost estimate to the Budget Division, which uses that information to determine program allocations by subactivity in the “budget call” memo that goes out to program offices in January or February. However, because the life cycle cost estimate is not valid and reliable, as described above, budget requests based on that estimate are not fully informed.

The Bureau updates the 2010 life cycle cost estimate to reflect appropriations for specific fiscal years but does not update the cost estimate with actual costs as they take place over the decade. According to Bureau officials, they continually revise the life cycle cost estimate based on changes resulting from the Commerce and Office of Management and Budget (OMB) passback processes, and once they submit the budget to Congress, they revise the life cycle cost estimate to match what is in the budget submission, including outyears. Bureau 2010 Census life cycle cost estimate documents for 2003, 2005, 2006, and 2007 contain tables showing enacted appropriations figures for past years and text explanations that the updated estimates reflect actual appropriations and submitted budget requests. However, Bureau officials said that the Bureau does not analyze the accuracy of the life cycle cost estimate each year, such as by comparing the estimate to actual costs at the end of the year, or update the estimate with actual costs.

A high-quality cost estimate is the foundation of a good budget. A major purpose of a cost estimate is to support the budget process by providing an estimate of the funding required to efficiently execute a program. Because most programs do not remain static but tend to evolve over time, developing a cost estimate should not be a onetime event but rather a recurrent process. Our Cost Assessment Guide explains that a cost estimate should be a “living” document that is continually updated as actual costs begin to replace original estimates, so that it remains relevant and current. Effective program and cost control requires ongoing revisions to the cost estimate and budget.

When we asked why the Bureau does not update the 2010 life cycle cost estimate with actual cost data, a budget official told us that actual cost information can be incurred over multiple fiscal years, and it would be difficult to compare this information to the annual framework used for the cost estimate. For budget purposes, the Bureau updates the life cycle cost estimate every year based on appropriations figures instead of actual cost data, because appropriations data are attributed to single fiscal years and are easier to work with than actual cost data.

Using a reliable life cycle cost estimate to formulate the budget could help the Bureau ensure that all costs are fully accounted for so that resources are adequate to support the program. Credible cost estimates could also help the Bureau effectively defend budgets to a department secretary, OMB, or Congress. In addition, the Bureau could use the cost estimate to help determine how budget cuts might hinder the census program’s progress or effectiveness. Moreover, because the Bureau does not update the life cycle cost estimate with actual cost data, the Bureau will not have the ability to keep the estimate current or document lessons learned for cost elements whose actual costs differ from the estimate.

Concerns about the soundness of the life cycle cost estimate and the quality of annual budgets related to the 2010 Census are particularly important because the bulk of funds will be spent from fiscal years 2009 through 2013, as shown in table 1.

The Bureau Has Insufficient Policies and Inadequately Trained Staff for Conducting High- Quality Cost Estimation; However, a New System May Help Improve the Bureau’s Cost Estimation Practices

The Bureau has insufficient policies and inadequately trained staff for conducting high-quality cost estimation. The Bureau does not have formal cost estimation policies and procedures. The Bureau also does not have skilled cost estimators or a centralized office dedicated to cost estimation. According to Bureau officials, although multiple staff members with various census backgrounds and experiences from across 25 divisions develop information for use in the cost estimate, we found that staff are not adequately trained in cost estimation. Bureau officials told us that most of the managers have project management certificates or training, which includes classes in cost estimation. However, the classes were designed more to provide program management with a general understanding of cost estimation rather than to provide in-depth training to the actual cost estimators.

In order to consistently develop reliable cost estimates, it is important for an agency to have defined policies and procedures to govern the process. Cost estimating best practices were developed to help agencies establish appropriate policies and procedures for producing estimates that adhere to the characteristics of high-quality cost estimation. An agency’s cost assessment team should include members who are experienced and trained in conducting cost estimation. Further, centralizing the cost estimating team and process (including cost analysts working in one group but supporting many programs) represents a cost estimating team best practice. Since the experience and skills of the members of a cost estimating team are important, some organizations have chosen to establish training programs and certification procedures. For example, the Department of Defense (DOD) established the Defense Acquisition University (DAU), which provides basic, intermediate, and advanced certification training as well as continuous learning opportunities. Although DAU’s primary mission is to train DOD employees, all federal employees are eligible to attend, including Bureau employees.

Bureau officials understand that the Bureau’s cost estimation process needs to be improved and are currently developing DBiT, a budget management tool that will support the process underlying the generation of cost estimates by facilitating, managing, and documenting changes in variables and assumptions that support the cost estimates. If properly implemented, DBiT should secure, standardize, and consolidate budget information and enable the Bureau to maintain better documentation for cost estimation. Further, DBIT is supposed to enhance access to budget data and increase the ability to model, formulate, execute, and report the decennial census budget. Officials told us that DBiT would have the ability to download data from the budget database, thus facilitating linkages between the life cycle cost estimate and budget preparation processes. Bureau officials also said that the Cost and Progress system, which tracks the actual cost of operations, would not link to DBiT. Without this capability, the Bureau will not be able to systematically update the estimate with actual costs. However, Bureau officials indicated that they might consider linking the two systems in the future.

Bureau officials also said that DBiT will enable the Bureau to save different versions of the cost model and will provide them with the capability to use software packages such as Crystal Ball to perform sensitivity and uncertainty analyses on its estimates. However, officials did not assert their intention to conduct these analyses. DBiT is being incrementally implemented and will not be fully functional until after 2010.

While DBiT should improve the Bureau’s systems for developing budgets and the life cycle cost estimate for the 2020 Census, the Bureau will still need established policies and procedures for conducting cost estimation and skilled estimators. Policies and procedures to govern the process as well as a dedicated office that is supported by properly trained staff are the foundation for a reliable cost estimate. Not having the tools and people in place for the 2010 Census has impeded the Bureau’s ability to produce a sufficiently documented, comprehensive, accurate, and credible cost estimate.

Conclusions

On April 3, 2008, the Secretary of Commerce presented a redesigned 2010 Census plan with significant cost increases. However, until the Bureau makes fundamental changes to how it estimates and updates cost information, uncertainties about the ultimate cost of the 2010 Census will remain. The Bureau’s ability to produce well-documented, comprehensive, accurate, and credible cost estimates for the 2010 and future decennial censuses and its ability to effectively manage operations and contain costs will continue to be hampered unless improvements are made to its cost estimation processes and systems. Specifically, without full documentation of the data sources, assumptions, and calculation methods the Bureau uses, the 2010 life cycle cost estimate cannot be validated, nor can the Bureau understand and explain differences between estimated and actual costs—an important step in improving future cost estimates. Also, without updating assumptions for the 2010 life cycle cost estimate and making clear what the underlying assumptions are, the Bureau cannot ensure that it is providing the most up-to-date and accurate cost estimates to OMB and Congress. Further, without conducting sensitivity and uncertainty analyses on the 2010 life cycle cost estimate, the Bureau is unable to identify and focus on major cost drivers, understand the potential for cost growth, and quantify the risk and uncertainty associated with the cost estimate. Finally, without established policies and procedures and qualified staff, the Bureau’s ability to produce high-quality cost estimates for the 2010 Census and future censuses will be limited.

Along with the new cost estimate, the Secretary of Commerce outlined major changes to the 2010 Census design. Changes this late in the decade significantly increase the risk to the success of the 2010 Census. These changes, during an era of serious national budget challenges, make it important for Bureau managers to efficiently manage the new design in order to contain costs. Furthermore, careful monitoring and oversight by Commerce, Congress, and other key stakeholders are more critical than ever.

Recommendations for Executive Action

To improve the Bureau’s life cycle cost estimates for the decennial census, we recommend that the Secretary of Commerce direct the U.S. Census Bureau to take the following five actions: 1. To improve the quality and transparency of the Bureau’s 2010 Census life cycle cost estimate, assist the Bureau in managing costs during design revisions resulting from problems with the HHCs, and help establish a sound basis for the 2020 Census cost estimate, the Bureau should thoroughly document the 2010 Census life cycle cost estimate. Specifically, documentation should be maintained in a centralized standard format and specify all data sources, assumptions, calculation methods, and cost elements used to prepare the 2010 cost estimate. 2. To ensure that the life cycle cost estimate reflects current information, the Bureau should update assumptions as appropriate, including updating productivity assumptions to reflect results from the address canvassing dress rehearsal. The Bureau should also document the basis for prior and future changes made to assumptions used in the life cycle cost estimate. 3. To keep the life cycle cost estimate current and to document lessons learned for cost elements whose actual costs differ from the estimate, the Bureau should update the estimate to reflect actual costs. 4. To improve the quality of and provide a confidence level for the 2010 Census life cycle cost estimate, the Bureau should perform sensitivity and uncertainty analyses on the estimate. 5. To help ensure that the Bureau produces a reliable, high-quality life cycle cost estimate for the 2020 decennial census, the Bureau should establish guidance, policies, and procedures for conducting cost estimation that would meet best practices criteria and ensure that it has staff resources qualified in cost estimation.

Agency Comments and Our Evaluation

The Secretary of Commerce provided written comments on a draft of this report on May 23, 2008. The comments are reprinted in appendix II. Although Commerce raised concerns about how GAO characterizes the accuracy of the estimate, Commerce stated that it agrees with many of our findings, and that the Bureau will prepare a formal action plan to document specific steps (with estimated completion dates) it will take in response to the recommendations. Commerce stated that the Bureau will determine if it will be possible to make improvements in the short term to its cost estimates and methods. Further, Commerce made some suggestions where additional context or clarification was needed and where appropriate we made those changes.

Commerce commented that our report, in its discussion of increasing census costs, does not mention other significant factors that have contributed to substantial cost increases over the last 40 years. We agree and have clarified in our report that other factors have contributed to increased costs, such as accommodating more complex households, busier lifestyles, more languages and greater cultural diversity, and increased privacy concerns.

Commerce also stated that it would have been premature to include the cost of fingerprinting temporary workers in the 2010 Census life cycle cost estimate. However, best practices state that having a realistic estimate of projected costs makes for effective resource allocation and increases the probability of a program’s success. To be prudent and conservative, an agency should include possible program costs that may have an impact on the overall life cycle cost estimate. We appropriately characterize the cost for fingerprinting temporary workers for the 2010 Census by stating that it is a “potential” cost. Also, understanding that the life cycle cost estimate is, in fact, an estimate, the draft report states that “there is risk that the life cycle cost estimate has been substantially understated.” We therefore made no change to the report.

In commenting on our description of how the Bureau updates the estimate to reflect costs, Commerce stated that it believes that for budget purposes, using enacted appropriations is the best way to adjust the life cycle cost estimate. However, best practices require that an estimate be updated to reflect actual costs when a difference occurs. This enables an agency to determine the precise reasons why actual costs differ for the estimate and document lessons learned. Because the draft report already reflected the Bureau’s practice of using enacted appropriations, we made no changes.

Commerce did not agree with our statement that the 2010 life cycle cost estimate is not accurate as it relates to assumed productivity rates for the fiscal year 2009 address canvassing operation. Commerce stated that at the time of GAO’s review of the life cycle cost estimate, the Bureau had not completed its analysis of the dress rehearsal productivity data. However, productivity data from the address canvassing dress rehearsal were provided to us in December 2007. These productivity data were significantly different from the assumptions used in the life cycle cost model. The updated productivity assumptions should have been included in the estimate that was included in the fiscal year 2009 President’s Budget request, which was issued in February 2008.

In addressing our recommendations, Commerce generally agreed with our five recommendations, and indicated that the Bureau will prepare specific steps in response to the recommendations. Commerce stated that as part of the Bureau’s action plan it would examine GAO’s Cost Assessment Guide to determine if it would be possible to make improvements in the short term to its cost estimate and methods. Commerce further noted that the Bureau already has efforts under way to improve future cost estimation methods and systems through the development of DBiT. In addition, Commerce acknowledged that at the lower level of detail estimates in the model, the Bureau will need to develop both a way to input costs in the model (or the DBiT system) in time to inform estimates for the 2020 cycle, and a way to update this information regularly over the coming decade. Further, Commerce noted that the Bureau was updating the life cycle cost estimate to reflect revised assumptions for the address canvassing operation.

In response to our recommendation to perform sensitivity and uncertainty analyses on the estimate, Commerce said that the Bureau would use the Cost Assessment Guide and seek detailed information on the experiences of other agencies that have used sensitivity analysis. Commerce further commented that while it does not disagree that conducting sensitivity analysis and providing a possible range of costs would be useful to external audiences, the Bureau must submit budget requests and out-year estimates as fixed amounts rather than ranges. We understand that fixed amounts rather than ranges are submitted for budget requests. However, conducting sensitivity analysis on the cost estimate would be beneficial to the Bureau in its management of costs associated with the census, not just to external audiences. Sensitivity analysis provides valuable information to an agency about which assumptions or factors have the biggest effects on cost. Further, the Bureau should also conduct uncertainty analysis to quantify the overall uncertainty of the cost estimate and provide a level of confidence associated with the point estimate. Providing a range of costs around a point estimate is useful to decision makers because it conveys the level of confidence in achieving the most likely cost, and uncertainty analysis can also help managers identify a defensible level of contingency funding needed to reach a desired confidence level. The Bureau should ensure that it conforms to best practices by making sensitivity and uncertainty analyses part of required processes for cost estimation.

In response to our recommendation to establish guidance, policies, and procedures for conducting cost estimation and ensure that it has staff resources qualified in cost estimation, Commerce stated that the Bureau has efforts under way to improve future cost estimation methods and systems through the development of DBiT. However, these efforts will not be completed in time to be used in preparing the 2010 budget request—an effort already underway. Commerce further stated that the Bureau would examine the Cost Assessment Guide to determine if it will be possible to make improvements in the short term to the estimate and methods, including possibly hiring additional skilled cost estimators.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to other interested congressional committees, the Secretary of Commerce, the Director of the U.S. Census Bureau, and the Director of the Office of Management and Budget. Copies will be made available to others upon request. This report will also be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-6806 or sciremj@gao.gov. Key contributors to this report were Vidhya Ananthakrishnan, Thomas Beall, Jennifer Echard, Richard Hung, Anne Inserra, Jason Lee, Andrea Levine, Donna Miller, Lisa Pearson, Michelle Petre, Sonya Phillips, Karen Richey, John Sperry, Niti Tandon, Shannon VanCleave, and Michael Volpe. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report.

Appendix I: Objectives, Scope, and Methodology

The objectives of this report were to (1) assess the extent to which the U.S. Census Bureau’s (Bureau) 2010 Census life cycle cost estimate adheres to characteristics defined for high-quality cost estimation, (2) report on the relationship between the life cycle cost estimate and the Bureau’s budget, and (3) assess whether the Bureau’s existing policies and resources are sufficient to conduct high-quality cost estimation.

To address our first objective, we evaluated the Bureau’s 2010 Census life cycle cost estimate to determine whether it met key characteristics identified in the exposure draft of our Cost Assessment Guide. Our guide, which is based on extensive research of best practices for estimating program schedules and costs, states that a high-quality, valid, and reliable cost estimate should be well documented, comprehensive, accurate, and credible. We obtained and analyzed the version of the 2010 Census short form cost model that was given to us in November 2006. (The Bureau provided a cost model containing data for fiscal years 2009 through 2013, and separate models for fiscal years 2007 and 2008. The models, which were in Excel, did not include costs for fiscal years 2001 through 2006.) We also analyzed Bureau documents related to the 2010 Census life cycle cost estimate and cost estimates for previous censuses. We interviewed Bureau officials from the Decennial Management and Budget Divisions about the process used to prepare the life cycle cost estimates and the assumptions used to prepare the estimates. We shared the Cost Assessment Guide and the criteria against which we would be evaluating the Bureau’s cost estimate with Bureau officials.

To assess the adequacy of the Bureau’s 2010 Census life cycle cost estimate, we compared the Bureau’s methods and approaches for preparing the estimate with the guidance contained in our Cost Assessment Guide. We assessed whether the 2010 estimate met the four desired characteristics of being well documented, comprehensive, accurate, and credible. Given those criteria, the main purpose of this objective was to assess the reliability of the cost estimate. We assessed the extent to which the entire life cycle cost estimate was well documented, including the short form cost estimate, and the estimates for the American Community Survey (ACS) and the Master Address File/Topologically Integrated Geographic Encoding and Referencing (MAF/TIGER) system. Our analysis for the other three characteristics—comprehensiveness, accuracy, and credibility—was limited to the short form cost estimate, due to lack of available documentation on the ACS and MAF/TIGER components of total life cycle costs.

As part of our assessment of the credibility of the Bureau’s 2010 Census life cycle cost estimate, we conducted sensitivity analysis to identify significant cost drivers and limited uncertainty analysis on a portion of the 2006 version of the short form cost estimate that the Bureau provided. Portions of the total $11.3 billion life cycle cost estimate were excluded from the uncertainty analysis, as detailed below and shown in figure 5: $1.7 billion of the total had already been spent from fiscal years 2001 through 2007 (these sunk costs had to be excluded from the uncertainty analysis); $1.2 billion of the total was allocated to costs for MAF/TIGER ($0.1 billion) and ACS ($1.1 billion) for fiscal years 2008 through 2013 (since these costs were estimated separately and not included in the short form cost model, they were excluded from the uncertainty analysis); $0.6 billion of the total represented estimated fiscal year 2008 costs for the short form Census, and the Bureau gave us estimated fiscal year 2008 costs in a separate model; and $2.3 billion of the total consisted of costs that the Bureau did not model but instead just provided as “throughput” or fixed costs.

Subtracting the above components from the total left $5.4 billion to be analyzed. We determined that the results of uncertainty analysis performed on this limited portion of the total costs would not be meaningful.

Our analysis of the Bureau’s 2010 life cycle cost estimate was conducted prior to the redesign of the census and the subsequent revision of the FDCA program. Detailed information on revised assumptions, cost data, methods of calculation, or the process used to revise the estimate was not available for the Bureau’s proposed redesign in time to be analyzed as part of our scope. Because we did not analyze the Bureau’s updated range for the total cost estimate of from $13.7 billion to $14.5 billion, we cannot verify whether the revised estimate is accurate, valid or reliable.

To address the second objective, we reviewed policies and procedures for preparing annual budgets. We analyzed Bureau documents related to the budget preparation system and the process for preparing the decennial census life cycle cost estimate, including the Budget Formulation and Performance Planning Manual, and internal correspondence from the Budget Division to other Bureau offices concerning annual budget preparation. We also analyzed Bureau budget estimates as presented to Congress, life cycle cost estimate documents, and worksheets on life cycle costs prepared for budget formulation, and confirmed that appropriations figures for completed fiscal years did appear in subsequent life cycle cost estimate documents. Further, we interviewed officials from the Bureau’s Budget and Decennial Management Divisions about the relationship between the cost estimate and the annual budget process. We did not independently verify budget information provided by the Bureau because that would have been outside the scope of our review.

To address our third objective, we analyzed Bureau documents related to the life cycle cost estimate. We compiled information on the Bureau’s process for developing its decennial cost estimate and the credentials of the Bureau’s cost estimation staff. We requested the Bureau’s policies and procedures for cost estimation, but determined that the Bureau does not have specific policies and procedures for cost estimating. We evaluated Bureau information against the best practices criteria presented in our Cost Assessment Guide for developing a high-quality cost estimate and designating an experienced, well-trained cost estimation team. We also interviewed Bureau officials.

To determine what steps the Bureau is taking to improve its cost estimation practices, we attended a demonstration of the Decennial Budget Integration Tool, the new Bureau automated budget system currently being developed that should enable the Bureau to produce better cost estimates. We also interviewed Bureau officials.

We conducted our work from October 2006 to June 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Commerce