Decennial Census: Additional Actions Could Improve the Census Bureau's Ability to Control Costs for the 2020 Census
Released: 2012-02-23T00:00:00

Letter

Background

The cost of the census, in terms of cost for counting each housing unit, has been escalating over the last several censuses. The average cost for counting a housing unit increased from about $16 in 1970 to around $97 in 2010 constant dollars (see fig. 1). Meanwhile, the return of census questionnaires by mail (the primary mode of data collection) declined over this period from 78 percent in 1970 to 63 percent in 2010. Declining mail response rates are significant and have led to higher costs because the mail response rate directly dictates the number of housing units in the nonresponse follow-up (NRFU) universe. NRFU, where the Bureau attempts to contact households that did not mail back questionnaires, was the largest and most costly Bureau field operation in 2000 and 2010 and has had an impact on overall census costs.

Over the past several censuses, the Bureau has attempted to address the competing goals of containing costs and improving the quality of census information, but costs continued to rise in part because external factors, such as a growing and increasingly diverse population, required the Bureau to devote more resources in order to ensure a complete count. The Bureau is assessing various measures of the quality of the 2010 Census. This effort, combined with a better understanding of the specific sources of cost growth, could help managers make cost control decisions.

Within its financial management system, the Bureau classifies census costs into eight broad categories and hundreds of projects (see fig. 2). These broad categories are further subdivided into individual projects that may be discrete, such as the NRFU operation, which has its costs captured in a single project line, or several project lines may be combined, sometimes from multiple categories, to reflect the total cost of an operation, as is the case with the Local Update of Census Addresses (LUCA) operation. The 2010 Census costs were concentrated in few categories.

Planning for the 2020 Census is divided into five phases: (1) options analysis; (2) research and testing; (3) operational development and systems testing; (4) supplemental research and testing; and (5) readiness testing, execution, and closeout. The Bureau has identified a range of design alternatives for the 2020 Census and will narrow this range over the census life cycle. During fiscal year 2012, the Bureau will enter the research and testing phase and intends to develop a preliminary design that when adjusted for inflation will cost less than the $97 per housing unit cost of the 2010 Census but will also maintain quality. During the research and testing phase, the Bureau plans to execute at least 35 research projects to explore how design areas could be modified to control costs or improve quality. For example, the Bureau will examine the feasibility of using administrative records, such as Internal Revenue Service tax records, to collect information from nonresponders and thus reduce the fieldwork. Other research areas include new response options, such as the Internet and social networking sites.

The Bureau uses life cycle cost estimates as a starting point for annual budget formulation and revises the estimates based on appropriations and updated budget information. As noted in our Cost Estimating and Assessment Guide, a life cycle cost estimate can be thought of as a “cradle to grave” approach to managing a program throughout its duration. However, in our past work, we found that the Bureau’s 2010 Census life cycle cost estimate was not reliable because it lacked adequate documentation and was not comprehensive, accurate, or credible. The Bureau may continue to be challenged in developing reliable life cycle cost estimates for a program as large, costly, and complex as the census. As part of its planning for 2020, the Bureau has developed an early life cycle cost estimate based on existing information and plans to release a full range of life cycle cost estimates in the budget submission for fiscal year 2015.

The Bureau Cannot Identify Specific Sources of Cost Growth from 2000 to 2010

Field Data Collection Drove Cost Increases, but More Details Are Needed to Help Control Future Costs

Of the Bureau’s eight broad budget categories, field data collection and its associated support systems accounted for $3.5 billion of the $4.6 billion life cycle cost increase, or 77 percent of the overall cost growth from 2000 to 2010 (see table 1). This represents a 64 percent growth in the field data collection category from its 2000 totals, which was the largest percentage increase of all budget categories. Field data collection costs include training, labor, and mileage for temporary workers, as well as the support systems needed to run operations, including rental space and office equipment for local census offices (LCO). We previously reported that the field data collection budget category was also the largest contributor to cost growth from the 1990 Census to the 2000 Census.

The remaining seven budget categories accounted for less than 25 percent of overall cost growth. The automated data collection category experienced the second largest growth, accounting for 12 percent of overall cost growth from 2000 to 2010. Expenses in this category were $547 million more than in 2000, a 42 percent increase. This category includes data processing activities and related information technology (IT) system costs. Smaller categories experienced cost growth as well, including content, questionnaires, and products; census design, methodology, and evaluation; and census test and dress rehearsal. Other categories actually experienced cost decreases, including program development and management.

According to the Bureau, an increased workload—a larger number of housing units to count—is one of the factors driving up census costs. This, however, does not fully explain (1) why the cost to count each housing unit grew at a faster pace than the workload (39 percent increase to count each housing unit compared to 12 percent increase in workload) or (2) why component costs, such as data capture systems, experienced cost increases (see fig. 3).

To more fully understand what is driving up census costs aside from an increase in workload, it will be important for the Bureau to analyze cost growth below the category level to determine the specific reasons why cost per housing unit continues to grow at a faster pace than workload. Key questions in this regard include, for example, (1) to what extent did increased labor and gasoline costs contribute to overall increases in field data collection costs, (2) how did additional use of technology contribute to field data collection costs, (3) how did increased investments in non- field-related IT systems affect cost growth, and (4) to what extent did the weak economy in 2010 help the Bureau reduce costs for field operations. While some cost increases, such as rising gasoline prices, might have been outside of the Bureau’s direct control, better information on the sources of census cost growth could enable the Bureau to develop work- arounds and alternatives that could mitigate their impact.

Best practices in GAO’s Cost Estimating and Assessment Guide illustrate how an agency can strengthen its ability to control costs by using available cost data to make comparisons over time and identify and quantify trends. However, the Bureau cannot identify specific sources of cost growth below broad budget categories from 2000 to 2010 because the Bureau changed the way it defines projects without creating a crosswalk that documents the changes over time. As a result, the Bureau cannot specifically determine where costs are growing.

While it is reasonable for the Bureau to modify its budget structure to accommodate changes from one decennial to the next, a crosswalk would have enabled officials to compare costs for specific projects. For example, for the 2000 Census, 236 projects were identified in the budget. For the 2010 Census, the Bureau changed its budget structure to more precisely capture costs, and as a result, the number of projects listed in the budget increased by almost 400 percent to 1,175 projects. However, the Bureau created no documentation to facilitate comparison for most projects in the budget from 2000 to 2010. For example, costs for LUCA are combined into one project in 2000 data while in 2010 data, LUCA activities were identified in 11 separate projects (for example, LUCA processing and LUCA testing). Without documentation explaining what costs were included in LUCA for the 2000 Census, it is impossible to accurately compare costs for LUCA between the two decennials and determine where any cost growth might have occurred.

Further, the Bureau cannot accurately calculate the growth in field infrastructure costs, if any, from 2000 through 2010 because of a similar lack of documentation. Although the $2 billion the Bureau spent on its field infrastructure in 2010—including 12 regional census centers and almost 500 LCOs used to support field activities—represented a major investment, the Bureau lacks the information needed to accurately compare the costs of specific components from one decennial to the next. Such information would enable the Bureau to more accurately determine where any significant cost increases occurred and thus better focus its cost control efforts for the 2020 Census, as well as allow the Bureau to more precisely determine the potential cost savings of any operational changes.

Although these structural changes are recent, the absence of documentation has been a challenge in the past as well. In a prior report comparing costs for the 1990 and 2000 censuses, we were unable to compare costs at the project level because of limitations in the available data and documentation. For the 1990 Census, the Bureau provided limited cost data by activity and project, so we were not able to attempt detailed cost comparisons. Moving forward, it will be important for the Bureau to put a process in place to enhance its ability to identify potential factors affecting cost growth and, if necessary, target cost control efforts appropriately.

Cost Factors Identified by the Bureau Are of Limited Use

Although the Bureau identified five broad factors affecting cost growth, their ability to help the agency pinpoint and control future costs is limited because they mainly focus on high-level, generic management challenges rather than specific census-taking activities on which the Bureau can assess and take action as appropriate. Additionally, the Bureau has no data to support how much these factors contributed to cost growth. The five factors include 1. the increasing diversity of the population; 2. the demand for the Census Bureau to strive for improving accuracy 3. the lack of full public participation in the self-response phase of the census, requiring the hiring of a large field staff for NRFU; 4. the failure or challenges with linking major acquisitions, the schedule, and the budget; and 5. substantial investments in major national updating of the address frame just prior to enumeration (2009).

The Bureau plans to use these factors to guide 2020 Census planning and research efforts. For example, the forthcoming research and testing phase will focus on the decreasing self-response rate; the linkage of acquisitions, schedule, and budget; and updates to the address frame. While these factors, which the Bureau developed through management experience, likely affected the cost of the census, evaluating the extent to which specific operations and activities drove up census costs would provide the Bureau with more actionable information. As one example, the Bureau identified the demand for improved accuracy as a factor, but this effort to improve accuracy involved a number of operations aimed at producing a more complete count, ranging from advertising in different languages to special enumeration programs aimed at hard-to-count populations. What is not clear, and will be important for the Bureau to determine, is how the cost of the special enumeration programs compared to those for 2000, the extent to which they contributed to the cost of the 2010 Census, and whether they produced the desired results.

The Bureau Is Assessing Options for Controlling Field Costs, but Additional Steps Could Be Taken

The Bureau Is Considering Design Alternatives to Potentially Reduce Costs for the 2020 Census

The Bureau has developed a range of design alternatives for the 2020 Census aimed at counting each housing unit at a lower cost than in 2010. The Bureau estimated that if it repeated the design of the 2010 Census, and assuming real costs grow at the same rate they did between 1990 and 2010, it would cost $151 to count each housing unit—more than a 55 percent increase, compared to 2010. The challenge for the Bureau, as recognized in its 2020 Census business plan, is striking a balance between an accurate census, on the one hand, and reducing costs and managing risks, on the other.

The Bureau’s 2020 design alternatives have potential for containing costs but at varying degrees of risk for meeting cost, schedule, and performance goals. The design alternatives focus on options to target address canvassing, using the Internet and other social media to increase response rates, and reengineer field and IT infrastructures.shows the current range of 2020 Census design alternatives. According to the Bureau, the final 2020 design is likely to incorporate both existing approaches as well as activities that have never been used in the decennial census, such as a near paperless NRFU.

According to the Bureau, the greater the change to the overall design, the greater the potential for cost savings. However, greater design changes also incur greater risk, and further testing will be needed to identify the risks, costs, and benefits of any new approaches. According to the Bureau, alternative one has the lowest risk, as it most closely mirrors the 2010 Census design and is not dependent on implementing innovations such as administrative records and targeted address canvassing. The remaining alternatives incorporate varying degrees of centralized infrastructure; address canvassing; and use of administrative records, the Internet, and social networks. For example, most of the new design options use administrative records, which could save money by reducing labor-intensive and costly field operations. Yet, the Bureau has not previously used administrative records to supplement respondent data on a national level, so the new approach will present a certain degree of risk as such things as data quality and access to records must first be resolved.

Additional Analysis of 2010 Data Could Improve the Bureau’s Ability to Make Cost Decisions about 2020

The Bureau collects data on the costs of its field operations that are a potentially valuable source of information to help guide future cost quality trade-off decisions during the planning process. However, it could make better use of this information in gaining an understanding of return on investment for costly census-taking activities, such as address building and NRFU. According to Office of Management and Budget (OMB) guidance on benefit-cost analysis, agencies should have a plan for periodic, results-oriented evaluation of the effectiveness of federal programs. The guidance also notes that retrospective studies can be valuable in determining if any corrections need to be made to existing programs and to improve future estimates of other federal programs. In addition, our Cost Estimating and Assessment Guide suggests that agencies should seek the best value solution by gathering data on alternatives that inform agencies on cost and performance trade-offs.One way agencies can improve their ability to evaluate benefits and costs is to examine the marginal cost of activities, or the incremental cost of producing one more unit of output. For the Bureau, this means mining its performance and cost data to evaluate the effectiveness of its operations and to identify potential opportunities for improvement.

Although the Bureau has a number of efforts under way within two initiatives to help guide 2020 planning, only a handful are aimed at producing return on investment information that enhances its ability to make decisions on cost quality trade-offs. These initiatives are the 2010 Census Program for Evaluations and Experiments (CPEX) looks back at 2010 operations, and the research and testing phase looks ahead at potential design alternatives for 2020. According to Bureau officials, of the more than 100 planned evaluations, assessments, experiments, and quality profiles in CPEX, a few are designed to produce information describing the return on investment of census-taking activities, which can help the Bureau make decisions about cost-quality trade-offs. For example, 2 planned evaluations will examine potential cost savings for address canvassing—one looks at the potential cost reduction associated with targeted address canvassing and the other looks at potential cost savings associated with automated field data collection of address canvassing results. Moreover, of planned 2010 CPEX evaluations for which we have a description, the vast majority will measure aspects of accuracy or coverage.

The Bureau may be missing opportunities to mine performance data for information that could help officials increase the efficiency of costly field operations and could help inform difficult decisions for controlling costs and maintaining quality. As part of CPEX, the Bureau has planned about 50 assessments of specific enumeration activities and operations, such as address canvassing and NRFU. These assessments include an analysis of cost that would be of limited usefulness for informing return on investment decisions. For example, the assessments will compare budgeted and actual costs and indicate why an operation was over or under budget, but will not determine the marginal return for different enumeration or address-building operations. Information on the marginal returns on investments could, for example, help the Bureau determine where to focus cost control efforts. As one example, based on our analysis of operational data provided by the Bureau for NRFU, we determined that the marginal cost per questionnaire checked into LCOs was approximately $1,045 in the final weeks of the operation (see fig. 5). During this time, the Bureau completed a little over 2,300 questionnaires or roughly .005 percent of the entire NRFU universe of over 47 million housing units. This estimate is roughly a $1,000 increase per questionnaire compared to the first few weeks of the operation, which began on May 1, when the Bureau completed approximately 39 percent of the NRFU universe. Thus, it cost the Bureau approximately 17 times more per questionnaire in the final weeks of NRFU to attempt to obtain information from nonresponding housing units, units that may have been contacted as many as six times in person or by phone. More extensive analyses of these data could help the Bureau determine the extent to which specific activities contributed to cost growth and help it target control cost effects without compromising accuracy.

As the Bureau enters the research and testing phase, several planned projects will yield information that will improve its ability to make decisions balancing the competing goals of cost and quality. According to the Bureau, it is essential to conduct research and testing of multiple design alternatives prior to deciding upon a final census design and technical solution to ensure that the final census design is effective and works within the 2020 Census environment. Our review of Bureau planning documents identified 8 of 35 projects scheduled in the early part of the decade that will include analyses of costs and benefits. For example, a project on reducing and improving in-person follow-up operations is designed to examine the costs and benefits of different contact strategies and whether these will achieve the goals of the operation. However, most projects examine the accuracy and quality implications of conducting enumeration and not cost implications. Without gathering data on cost during this phase, specifically the potential cost savings that could be realized with certain alternatives, the Bureau could be making decisions based on incomplete information on the design alternatives.

The lack of emphasis on cost analyses is consistent with our previous reports that fundamental reforms will be needed to ensure that the Bureau’s management, culture, and business practices are aligned with cost-effective enumeration. According to Bureau officials, previous decisions about operational changes were based on a priority to improve quality and were sometimes made without much complete knowledge of cost implications. As we reported in 2009, the Bureau has not always used available information to determine the value added of the operation. For example, the Bureau has the information but has not determined which of its 11 operations for building its address list provide the best return on investment in terms of contributing to accuracy and coverage.

Decision Points Could Help the Bureau Control Costs and Avoid Delays

The Bureau’s planning documents have not clearly identified and defined decision points that can help avoid cost overruns and schedule delays. OMB guidance for large projects suggests that agencies develop a schedule with defined phases, decision points, and an identified decision authority to evaluate whether an agency should proceed to the next phase in the investment life cycle. In addition, our previous body of work on acquisition policies in high-performing organizations includes the best practice of identifying critical junctures, also known as knowledge or decision points, in the acquisition cycle and requiring executive-level oversight at critical junctures. Agencies can use decision points to determine whether a particular investment is ready to proceed to the next phase. For example, when moving out of an early phase agencies must determine if resources—that is, technology and funding—and needs are matched. The 2020 Census is a complex, costly project with immutable deadlines. Decision points at key phases of the planning process could improve the Bureau’s ability to manage risks as well as achieve desired cost, schedule, and performance outcomes for the decennial.

The Bureau’s 2020 business plan has a high-level preliminary schedule for the major phases of the decennial that includes, for example, a yearlong activity at the end of the research and testing phase to determine and refine initial operational designs. However, the schedule has no decision points at the end of research and testing or any phase, as best practices suggest, to determine whether progress was made and ensure that the agency’s needs for quality and accuracy match the available resources—that is, technology, design, time, and funding. In addition, there is no identified executive-level review at any point in the schedule. Since the research and testing phase is intended to develop a preliminary design from a range of alternatives, a decision point at the end of this phase could help the Bureau determine if it has enough information to support the increased investment necessary to move to the next phase of development and testing (see fig. 6). At subsequent stages in the process, decision points could be used to determine that the design was stable enough to meet operational requirements. Later decision points could also be used to determine whether a particular design alternative could be implemented within cost and schedule constraints while meeting quality targets and maintaining reliability. Absent such an approach at each phase, the Bureau lacks assurance that it has obtained the critical technological and design knowledge that best practices call for to avoid cost overruns, schedule slips, and performance shortfalls going forward.

According to one Bureau planning memo, cost is one of four categories of criteria that will be used to evaluate design options for the 2020 Census. However, the memo does not describe specifically how the cost criterion will be used to select among design alternatives. For example, the criterion for cost can be expressed by ranking costs (i.e., least costly to most costly), weighting costs for different elements, or specifying that costs fall within a range. We have previously reported that criteria should be clearly defined, well documented, transparent, and consistently applied. Neither the Bureau’s strategic plan nor its early business plan, which outlines and guides the early development of the 2020 Census, describes criteria or identifies when criteria would be used to select the design of the 2020 Census.

Bureau officials said they have not established when they will develop specific evaluation criteria for cost. Further, they acknowledged that selecting among design alternatives may take place during the research and testing phase, which begins in fiscal year 2012. In addition, Bureau officials told us that not all 2020 Census planning memos will be updated throughout the course of 2020 planning. Therefore, it is unclear how updates to criteria will be made to this planning memo. As a result, the Bureau may make decisions to eliminate design alternatives before clearly documenting how cost criteria will be applied, as well as how the alternatives will be considered along with the other criteria.

The Bureau’s early cost estimates range from $12.8 billion to $18 billion for four of the six design alternatives. Because of the wide range of 2020 cost estimates, documenting and consistently using cost as a criterion when deciding among design alternatives can help the Bureau control costs. It is important for the Bureau to apply cost in decision making because the Bureau has not achieved previous goals for containing costs and made late design changes that proved costly in previous censuses.

Without a Process in Place for Developing 2020 Census Life Cycle Cost Estimates, the Bureau Is at Risk of Not Following Best Practices

The Bureau has not yet established policies, procedures, or guidance for developing the 2020 Census life cycle cost estimate and is at risk of not following related best practices. The Bureau uses the life cycle cost estimate as the starting point for the annual budget formulation process and, according to our Cost Estimating and Assessment Guide, a reliable cost estimating process is necessary to ensure that cost estimates— particularly for large, complex projects like the 2020 Census—are comprehensive, well documented, accurate, and credible. Put another way, reliable cost estimates are essential for a successful census because they help ensure that the Bureau has adequate funds and that Congress, the administration, and the Bureau itself can have reliable information on which to base decisions.

Our guide identifies 12 steps of a high-quality cost estimation process, including, among other things, determining the estimate’s purpose; defining the program’s characteristics; clearly defining ground rules and assumptions; conducting sensitivity, risk, and uncertainty analyses; and documenting all steps used to develop the estimate. These best practices, if followed correctly, should produce reliable estimates that management can use for making informed decisions (see app. IV).

To date, the Bureau has developed a rough-order-of-magnitude estimate, which covers the four 2020 Census design alternatives that are the most similar to the 2010 Census design. Bureau officials stated that this was not an official estimate, but rather a starting point that will be revised and improved as the Bureau gathers more data in the research and testing phase. As the Bureau goes forward in its 2020 planning, it will be important for it to have reliable and accurate cost estimates as it narrows down design alternatives and moves closer to a final design.

The Bureau’s early 2020 planning documents note that the Bureau intends to use our cost guide as it develops cost estimates for 2020, and Bureau officials have stated that its cost estimators would follow best practices wherever practicable. Nevertheless, the Bureau has not yet documented how it plans to conduct its cost estimates; and, while officials stated that they plan on developing more detailed documentation in the future, they could not provide a specific time when such documents would be finalized.

Although the 2020 Census is still a number of years away, the timeline for the Bureau to develop a cost estimation process is growing short. The Bureau plans to begin work on an official life cycle cost estimate in fiscal year 2013, and plans to include its initial life cycle cost estimate in its fiscal year 2015 budget submission covering initiatives from 2015 through 2018. As a result, the Bureau has about a year to establish and finalize a process for preparing high-quality life cycle cost estimates.

The importance of reliable cost estimates is underscored by the Bureau’s experience leading up to the 2010 Census, where we found that the Bureau’s cost estimate lacked detailed documentation on data sources and significant assumptions and was not comprehensive because it did not include all costs. Among other weaknesses, we noted that the Bureau had insufficient policies and procedures for conducting high-quality cost estimation. Partly as a result, some operations had substantial variances between their initial cost estimates and their actual costs. Until the Bureau finalizes its cost estimating policies, procedures, and guidance, the Bureau runs the risk of again developing unreliable cost estimates for 2020.

Conclusions

For the Bureau to improve its ability to control the costs of future censuses without sacrificing accuracy, it will be critical for it to have a better understanding of the factors affecting cost increases from prior decennials, as well as how various census-taking activities contributed to the overall quality of the count. Although the Bureau will gain valuable insights from its evaluations of the 2010 Census as well as from research and testing for 2020, this information will only be of limited use in helping the Bureau develop a complete picture of the key drivers of census costs and the steps needed to control costs going forward. Therefore, to improve its capacity to identify cost drivers and effectively target cost control efforts, it will be important for the Bureau to develop a way to compare costs for key activities across censuses and assess the marginal returns of each.

The Bureau has set a clear goal for controlling costs while maintaining accuracy for the 2020 Census, and has developed a range of design alternatives aimed at achieving that goal. Given the number of design alternatives the Bureau is evaluating for the 2020 Census, it will be important for the Bureau to set explicit decision points for executive-level review at the end of individual phases to reduce the risk of cost, schedule, and performance shortfalls. Without clearly defined decision points in its 2020 planning phases, the Bureau may not be able to determine that it is on track or make the necessary adjustments in its design approach to achieve a more cost-effective census. Moreover, decision points would allow the Bureau to determine its readiness to move on to the next phase in 2020 planning. In conjunction with scheduled decision points, it will be critical for the Bureau to finalize evaluation criteria that are transparent, thoroughly documented, and consistently applied to maximize its ability to control costs for the 2020 Census. Without specifying explicit cost evaluation criteria for selecting among design alternatives, the Bureau and stakeholders, such as Congress, cannot accurately consider costs and may not have assurance that they are on the path to a more efficient census in 2020.

Cost estimates are necessary tools for major programs because they help in developing budget requests and efficiently allocating scarce resources. In a time of constrained budgets, these tools become even more important. However, cost estimates are technically complex and cost estimators face challenges in developing estimates for complex programs such as the 2020 Census. Previously, the Bureau had insufficient policies and procedures for developing reliable and high-quality cost estimates. Without clear guidance in place, there is no assurance that the Bureau will develop life cycle cost estimates for 2020 that are reliable and high- quality and follow best practices.

Recommendations for Executive Action

To improve the Bureau’s ability to control costs for the 2020 decennial and balance cost and quality, we recommend that the Secretary of Commerce direct the Under Secretary of the Economics and Statistics Administration, as well as the Director of the U.S. Census Bureau, to take the following four actions:

Develop and document a method to compare costs in 2010 to those in future decennials, for example, around major activities or investments, to allow the Bureau to identify and address factors that contribute to cost increases.

Analyze data from key census-taking activities to determine their marginal costs and benefits, and use this information to inform decisions on developing more cost-effective methods.

Identify decision points at the end of each planning phase and assign decision-making authority at the executive level, as well as consider adding decision points within phases to determine progress and readiness to proceed to the next phase.

Finalize how the Bureau will apply cost as an evaluation criterion for choosing among design alternatives for 2020 and ensure that all criteria are transparent, well documented, and consistently applied before alternatives are eliminated.

We have previously recommended that the Secretary of Commerce direct the Bureau to establish guidance, policies, and procedures for cost estimation that would meet best practice criteria. To help ensure that the Bureau produces a reliable and high-quality cost estimate for the 2020 Census, we recommend that the Bureau take the following action:

Finalize guidance, policies, and procedures for cost estimation in accordance with best practices prior to developing the Bureau’s initial 2020 life cycle cost estimate.

Agency Comments and Our Evaluation

The Secretary of Commerce provided written comments from the Bureau on a draft of this report on January 6, 2012. The comments are reprinted in appendix II. The Department of Commerce expressed broad agreement with the overall theme of the report but did not directly comment on the recommendations. It raised concerns about specific aspects of the summary of findings which GAO addressed as appropriate.

Specifically, the Bureau agrees with the importance of using analysis, such as assessing marginal returns to help with decision making on balancing the need to control costs while maintaining accuracy. Moving forward, it will be important for the Bureau to also recognize that more in- depth understanding of the growth in costs from prior censuses can, in fact, strengthen its decision-making ability and help it more effectively target cost control efforts in the future. The Bureau said understanding the growth in costs from 2000 through 2010 in depth has not been its highest-priority area for investment of scarce resources. We are sensitive to existing budget constraints. The fiscal issues facing federal agencies make it even more imperative for Bureau decision makers to develop and use actionable information, such as data on the extent to which specific operations and activities drove up costs, to pinpoint problem areas and target cost control efforts accordingly.

The Bureau expressed concern that the summary of findings and conclusions on the highlights page seemed premature and unsupported by discussions in the full report. In commenting on the first paragraph, the Bureau stated that it does not believe its inability to identify specific factors affecting past growth will make it difficult to control costs for the 2020 Census. However, our report concludes that the Bureau’s inability to identify specific actionable factors will make it difficult for the Bureau to focus its cost control efforts for the 2020 Census. To help pinpoint problem areas for controlling census costs, it is important for the Bureau to have a better understanding of the specific sources of cost growth. This requires analysis of costs below the broad category level, focusing on projects that tie directly to major operations and investments. We believe understanding how the cost of these programs compared to 2000 and the extent to which they contributed to the cost of the 2010 Census, and whether they produced the desired results can help with decision making on areas where there are trade-offs in cost and accuracy. We added language to the highlights page to reflect the need for analyses to more effectively target future cost control efforts.

In commenting on the second paragraph of the highlights page, the Bureau stated that it had not yet received any appropriated funds or had the opportunity to develop program management efforts for the 2020 Census that would allow the agency to establish formal guidance for developing cost estimates. However, the paragraph discusses practices for strengthening agency decision making for large projects rather than establishing formal guidance for developing cost estimates. OMB guidance for large projects and our previous body of work on acquisitions policies in high-performing organizations suggest setting explicit decision points for executive-level review at the end of individual planning phases can help reduce the risk of cost, schedule, and performance shortfalls. By clearly defining decision points in its 2020 planning phases, the Bureau could better ensure that it is on track or make adjustments in its design approach earlier in the 2020 planning process to achieve a more cost- effective census. On the issue of having appropriated funds for planning purposes, we agree that funding for the 2020 Census life cycle did not officially begin until fiscal year 2012. However, the Bureau includes costs of early planning for the next census in the final years of the previous census life cycle (i.e., 2010 appropriations pay for 2020 planning). During our audit, we interviewed individuals who were planning and developing the 2020 Census. We also reviewed informational memos, such as the strategic plan and the business plan issued in 2009 and 2010, respectively, as guiding documents for the 2020 Census planning effort. Moreover, during the audit, the Bureau released newly developed and revised planning documents, such as the updated business plan and rough-order-of-magnitude estimates. While we made changes to the second paragraph of the highlights page, these were not made in response to this comment.

The Bureau commented that it was unclear why the graphic in our highlights page focused on costs and mail response rate data over time. We selected mail response rates because, as the Bureau notes, declining mail response rates are significant and have led to higher costs. For example, the mail response rate directly dictates the number of housing units in the NRFU universe. NRFU is the largest and most costly Bureau field operation and has an impact on overall census costs. We agree with the Bureau that the declining mail response rate is only one factor leading to higher census costs. Our report acknowledges other factors that contribute to higher costs. As such, we made no change to the highlights page in response to this comment.

The Bureau made a number of technical comments on the body of the report. The Bureau commented that our report implies that the Bureau attributed all cost growth over the decades only to population growth. In fact, our draft report has a section dedicated to the five broad factors the Bureau identified as affecting cost growth. However, we added clarifying language to the discussion on workload and census costs to note workload is one of the factors driving up census costs.

The Bureau commented that the statement that the Bureau cannot determine areas of cost growth is a sweeping and premature conclusion given that the 2020 research and testing program just began. The Bureau stated that its primary focus is to study ways to reduce the cost of the next census while maintaining quality. While we acknowledge that the research and testing effort may help identify ways to reduce costs, coupling that information with specific factors of past cost growth could strengthen the Bureau’s ability to target cost reduction efforts in the future. We made no change to the report to address this comment.

The Bureau noted that statements in two areas seemed to be based on an assessment of how well the Bureau documented and analyzed costs relative to our Cost Estimating and Assessment Guide. The Bureau states that the guide was not issued until March 2009 and that the Bureau has not fully incorporated all those practices into a program as large, costly, and complex as the census. While it is true that we published the guide in March 2009, we issued an exposure draft in 2007 and shared a copy of our guide with the Bureau during our October 2006-June 2008 audit of Bureau cost estimating practices (GAO-08-554). In fact, the cost guide is based on long-standing industry and government best practices on cost estimation followed long before GAO published them in a concise form in 2009. Moreover, in its June 2008 action plan to address GAO recommendations, the Bureau noted its plan to use the guide, particularly the 12 steps of a high-quality cost estimating process. In less than 1 year from now, the Bureau plans to begin work on its official life cycle cost estimate for the 2020 Census. By not establishing policies, procedures, or guidance for developing life cycle cost estimates, the Bureau again runs the risk of developing cost estimates that are not comprehensive, accurate, or credible. We made no change to the report in response to this comment.

The Bureau commented that it was unsure why we presented the NRFU analysis of marginal costs as it was a small percentage of the entire budget. However, we used this as an example of how such an analysis may help point to areas for targeting cost reduction efforts or for modifying the Bureau’s approach to data collection. The analysis does not imply that the Bureau should ignore the remaining households at the end of NRFU as the Bureau’s comment states. Instead, it highlights the importance of considering alternative approaches in order to ensure a complete and cost-effective enumeration. The more important point is that it highlights the increasing marginal costs of contacting certain households at the tail end of the enumeration. We agree that the Bureau cannot ignore hard-to-contact households. By mining performance data on the NRFU operation, the Bureau may be in a better position to identify alternative approaches for the hardest-to-contact households that have the greatest potential to reduce costs without compromising accuracy. We made no change to the report to address this comment.

Finally, the Bureau commented that our reported costs for local census operations were incomplete, so we corrected the number based on information provided by the Bureau.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to other interested congressional committees, the Secretary of Commerce, and the Director of the U.S. Census Bureau. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V.

Appendix I: Objective, Scope, and Methodology

To identify the key factors affecting cost growth from the 2000 Census to the 2010 Census, we reviewed U.S. Census Bureau (Bureau) strategic planning documents for 2000 and 2010, Bureau operational and systems plans for 2000 and 2010, Bureau assessments and evaluations of past census operations, National Academy of Sciences work on decennial census costs, and our prior work on implementation of 2000 and 2010 census operations. We assessed the Bureau’s approach to determine trends in cost data using best practices for cost estimation in GAO’s Cost Estimating and Assessment Guide. The guide illustrates the importance of using cost data to understand trends and drivers. In addition, we interviewed Bureau officials and reviewed agency documentation on actions taken to determine sources of cost growth between decennials.

To identify sources of cost growth from the 2000 Census and the 2010 Census, we reviewed and analyzed expenditure data on 2000 Census and 2010 Census life cycle costs from the Bureau’s Commerce Business System (CBS). CBS is the Bureau’s financial management system and the official system of record for expenditures. information at two levels of aggregation for the census: budget categories, which are broad groupings of related items, and budget projects, which are the lowest level of cost information. To determine the level of cost growth from the 2000 Census and the 2010 Census, we developed life cycle totals for each census and life cycle totals for each of the budget categories within those censuses, comparing their absolute and percentage growth. We adjusted all monetary data for inflation using the gross domestic product implicit price deflator. All costs were adjusted to fiscal year 2010 dollars. In addition, we compared costs after adjusting for the number of housing units for each census.

We assessed the reliability of the Bureau’s CBS data by reviewing relevant documentation, interviewing knowledgeable agency officials, and conducting comparisons with other data sources. We reviewed previous GAO, Department of Commerce Inspector General, and other Department of Commerce reports covering the system. We conducted interviews with Bureau officials who maintain the system at the Bureau level and its primary users within the Decennial Management Division.

The system is referred to in older GAO reports as the Commerce Administrative Management System (CAMS). It is the same system, but the name changed over time.

After receiving the cost data covering the 2000 and 2010 censuses, we compared them to financial management reports provided by Bureau officials to determine data consistency. We determined that these data are sufficiently reliable for the purposes of this report.

Our review was subject to some limitations. The budget categories and budget projects for 2000 and 2010 varied from census to census. We requested and the Bureau provided recategorized cost data to facilitate comparison of the 2000 budget categories with 2010 budget categories. We requested and the Bureau was unable to provide recategorized cost data to facilitate comparison of 2000 budget projects with 2010 budget projects. We attempted to compare costs at the budget project level from 2000 to 2010 but were unable to do so for the following reasons: (1) the Bureau’s budget projects were not consistent from 2000 to 2010, making it impossible to match projects directly using project descriptions or project codes; (2) the number of projects increased substantially from 236 in 2000 to 1175 in 2010; and (3) the Bureau was unable to provide us with any documentation tracking similar projects from the 2000 Census to the 2010 Census. We attempted to group similar projects in 2000 and 2010 for comparison, but the available project descriptions did not provide enough information to group 2000 costs with the same precision as 2010. Therefore, we could not conduct a comparison of groups of projects.

To assess the Bureau’s plans for controlling costs for the 2020 Census and what additional steps, if any, could be taken, we reviewed available documentation on 2020 Census planning and 2010 Census evaluations and assessments, such as 2010 evaluation study plans. We consulted with GAO staff with expertise in economics to determine the potential for leveraging available Bureau cost data to better support the Bureau’s ability to make cost-quality trade-offs. We reviewed Office of Management and Budget guidance on major acquisitions as well as GAO work on acquisition best practices to determine whether the use of decision points could help the Bureau make more informed decisions about census design that could relate to cost control. Further, we reviewed Bureau documentation on criteria for selecting among 2020 design alternatives.

We analyzed the marginal cost of conducting nonresponse follow-up (NRFU)—the costliest field operation in the 2010 Census—to determine how the Bureau might be able to further use its cost data in planning for 2020. We used Bureau cost and progress data from the 2010 Census to identify the marginal costs of the NRFU operation in 3-week intervals. This analysis compared the cost of the operation for that period with the number of questionnaires checked in to identify return on investment. We assessed the reliability of the Bureau’s 2010 cost and progress data by consulting with the Bureau about variables we used and reviewing past GAO data reliability work that used cost and progress data. The cost and progress system is a daily management tool used by Bureau officials to track the work completed of various census operations. It includes measures of cost (such as field hours or mileage costs) and measures of work completed (such as questionnaires checked in). Our estimate of the marginal costs of checking in NRFU questionnaires in the early weeks of the operation may be somewhat overstated because, for instance, we included training costs as well as fieldwork costs because training costs were incurred in the early part of NRFU and those costs were not spread over the life of the operation. As a result, costs for the early weeks of the operation could be lower than presented in the graphic.

After developing the marginal costs methodology, we followed up with agency officials knowledgeable about the data when we had questions about potential errors or inconsistencies. In addition, we reviewed prior GAO data reliability work on cost and progress data that examined the accuracy and completeness of the entry and processing of data. Based on this work, we determined that the data were sufficiently reliable for gauging the approximate marginal cost increase per questionnaire checked in during the final weeks of the NRFU operation.

To assess the extent to which the Bureau’s plans for developing life cycle cost estimates for 2020 are consistent with best practices, we reviewed available Bureau documentation on the Bureau’s life cycle cost estimation processes and procedures. For example, we reviewed documentation from the Bureau’s rough-order-of-magnitude estimate—an early high- level estimate developed from limited data. We reviewed the guidance contained in our Cost Estimating and Assessment Guide and our previous work on census life cycle cost estimates. We also conducted interviews with knowledgeable Bureau officials and contractor staff and received a demonstration of new capabilities in the Bureau’s budgeting tool that will be used for 2020 Census cost estimation.

We conducted this performance audit from December 2010 through January 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Commerce

Appendix III: 2010 Census Costs Were Concentrated in Certain Budget Categories and Projects

Appendix IV: The 12 Steps of a High-Quality Cost Estimating Process

Step Description 1

Step Description 6 Obtain data

Step Description 10

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Signora May, Assistant Director; Tom Beall; Tim Carr; Eric Charles; Sara Daleski; Dewi Djunaidy; Ron Fecso; Robert Gebhart; Rich Hung; Kirsten Lauber; Jason Lee; Andrea Levine; Donna Miller; Stacey Steele; and Jack Wang made key contributions to this report.