2010 Census: Fundamental Building Blocks of a Successful Enumeration Face Challenges
Released: 2009-03-05T00:00:00

Letter

Background

As you know, Mr. Chairman, the decennial census is a critical national effort mandated by the Constitution. Census data are used to apportion seats in the Congress, redraw congressional districts, allocate billions of dollars in federal assistance to state and local governments, and for numerous other public and private sector purposes.

The Bureau estimates that the 2010 Census will cost more than $14 billion over its life-cycle, making it the most expensive census in our nation’s history. According to the Bureau, the increasing cost of the census is caused in part by various societal trends—such as increasing privacy concerns, more non-English speakers, and people residing in makeshift and other nontraditional living arrangements—making it harder to find people and get them to participate in the census.

In light of these challenges, it will be important for the Bureau to draw upon the lessons learned from previous decennials and apply them to the operational environment it faces today.

Some broad lessons learned that we have identified from our past work that directly affect the cost and accuracy of the census include the following: Sound risk management is critical to a successful census as the risks to a cost-effective enumeration are interrelated, and a shortcoming in one operation could trigger subsequent activities to spiral downward. Of course the reverse is also true, where a success in one operation could have a number of positive effects later in the process. Rigorous up- front preparations, testing, and where feasible, contingency planning, are the best ways to stave off problems. Likewise, management information systems capable of tracking key operations with real-time measures are essential because they enable the Bureau to quickly address trouble spots.

It is important for the Bureau to stay on schedule, as the census is conducted against a backdrop of immutable deadlines, and an elaborate chain of interrelated pre- and post-Census Day activities are predicated upon those dates. Specifically, the Secretary of Commerce is legally required to (1) conduct the census on April 1 of the decennial year, (2) report the state population counts to the President for purposes of congressional apportionment by December 31 of the decennial year, and (3) send population tabulations to the states for purposes of redistricting no later than April 1 of the year following Census Day. To meet these reporting requirements, it is absolutely critical for the Bureau to stay on schedule. The figure shows some planned dates for selected decennial events.

Finally, the decennial census is a shared national undertaking, where Congress; other federal agencies; state, local, and tribal governments; nonprofit and private organizations; and, ultimately, the American public, all play vital roles in securing a complete and accurate population tally. Recognizing this, the Bureau fosters partnerships with these various entities to help with such activities as recruiting census workers, boosting participation, and building the Bureau’s master address list. Mobilizing and coordinating these organizations requires an enormous effort on the Bureau’s part.

Providing Reliable Cost Estimates and Justifications for Spending as 2010 Approaches Presents a Major Challenge for the Bureau

Accurate cost estimates are essential to a successful census because they help ensure that the Bureau has adequate funds, and so that Congress, the administration, and the Bureau itself can have reliable information on which to base or advise decisions. However, as we have reported before, the Bureau has insufficient policies and procedures and inadequately trained staff for conducting high-quality cost estimation for the decennial census. The Bureau does not have cost estimation guidance and procedures in place or staff that is certified in cost estimation techniques. The Bureau is developing a new budget management tool that will support the cost estimation process beyond 2010. As part of that effort, the Bureau will need to establish rigorous cost estimation policies and procedures and use skilled estimators to ensure that future cost estimates are reliable and of high quality.

For example, to help manage the 2010 Census and contain costs, over 5 years ago we recommended that the Bureau develop a comprehensive, integrated project plan for the 2010 Census that should include the itemized, estimated costs of each component and a sensitivity analysis and an explanation of significant changes in the assumptions on which these costs were based. In response, the Bureau provided us with the 2010 Census Operations and Systems Plan, dated August 2007. This plan represented an important step forward by including operational inputs and outputs and describing linkages among operations and systems. However, that document did not include itemized cost estimates of each component or sensitivity analyses, and thus did not provide a valid baseline or range of estimates for the Bureau and Congress. The Bureau has provided annual cost updates as part of its budget submission process, but these too have lacked cost analyses to support them. As the Bureau approaches the final surge in the current decade-long decennial spending cycle, providing reliable cost estimates accompanied by sound justification, as we have recommended, will be important if Congress is to make informed decisions on the levels at which to fund the remainder of the 2010 Decennial Census.

Effective Address Canvassing Is Essential for a Complete and Accurate Count

A complete and accurate list of all addresses where people live in the country is the cornerstone of a successful census because it identifies all households that are to receive a census questionnaire and serves as the control mechanism for following up with households that fail to respond. The Bureau goes to great lengths to develop a quality address list and maps, working with the U.S. Postal Service; federal agencies; state, local, and tribal governments; local planning organizations; private sector; and nongovernmental entities. For example, under the Local Update of Census Addresses (LUCA) program, the Bureau is authorized to partner with state, local, and tribal governments, tapping into their knowledge of local populations and housing conditions in order to secure a more complete count. Between November 2007 and March 2008, over 8,000 state, local, and tribal governments provided approximately 8 million address updates through the LUCA program. The Bureau will send thousands of temporary census workers, known as listers, into the field to collect and verify address information and update maps on-site, including verifying address updates provided through the LUCA program.

Despite the Bureau’s efforts, an inherent challenge is locating unconventional and hidden housing units, such as converted basements and attics. For example, as shown in figure 2, what appears to be a small, single-family house could contain an apartment, as suggested by its two doorbells. The Bureau has trained listers to look for extra mailboxes, utility meters, and other signs of hidden housing units and is developing training guides for 2010 to help listers locate hidden housing. Nonetheless, decisions on what is a habitable dwelling are often difficult to make—what is habitable to one worker may seem uninhabitable to another. According to Bureau estimates, approximately 1.4 million housing units were missed in the 2000 Census. If an address is not in the Bureau’s address file, its residents are less likely to be included in the census.

Performance of Handheld Computers Have Improved in Field Testing, but More Information Is Needed to Evaluate Readiness for Address Canvassing

A nationwide address canvassing operation for the 2010 Census is scheduled to begin this spring, when listers will use handheld computers for the first time to collect address data. Listers will add addresses that do not already appear on the Bureau’s list and mark for deletion any that they cannot verify according to the rules and guidance developed by the Bureau.

When the handheld computers were tested during the dress rehearsal of the address canvassing operation, the devices experienced such problems as slow or inconsistent data transmission, freeze-ups, and difficulties collecting mapping coordinates. The software that managers used to review work productivity and assign work was also troublesome. For example, management reports were unreliable because they pulled data from incorrect sources, and Bureau staff had difficulty using the work management software to reassign work.

The Bureau took steps to fix these issues, and in December 2008 conducted a limited field test in Fayetteville, North Carolina, to test the functionality and usability of the handheld computer, including whether the handheld computer problems encountered earlier had been resolved. Although the Bureau’s final evaluation of the field test was due by the end of February 2009, we were not able to review it for this testimony. From observations of the December 2008 field test and interviews with Bureau officials, the Bureau appears to have addressed many of the handheld computer performance issues, as well as the problems with the work management software, observed during the dress rehearsal. This is an important and noteworthy development.

Nonetheless, more information is needed to determine the Bureau’s overall readiness for address canvassing as the field test was not an end- to-end systems evaluation, did not validate all address canvassing requirements, such as training and help desk support, and did not include urban areas. Additionally, the scale of the field test was a fraction of that of the address canvassing operation. The Bureau was to conduct a review of the readiness of the handheld computers in January 2009, but has not yet reported the results of that review. Finally, the Bureau’s actual workload for address canvassing—about 144.7 million addresses—is 11 million addresses more than the Bureau had planned for, leaving the Bureau with too few handheld computers to complete the workload in the time originally scheduled. In response, the Bureau will be extending the amount of time listers will be working in the field in affected areas, although not extending the end date of the operation, to compensate for the larger workload.

During dress rehearsal, listers experienced problems using handheld computers when collecting address data for large blocks having more than 1,000 housing units. According to the Bureau, the handheld computer did not have the capacity to efficiently collect data for large blocks. The Bureau has taken steps to mitigate this problem. In January 2009, the Bureau began using laptop computers and software already used in other operations to canvass the 2,086 blocks it identified as large blocks and by the end of February, the Bureau had completed approximately 80 percent of large block canvassing. In February 2009, we observed large-block canvassing in Atlanta, Georgia; Boston, Massachusetts; New York, New York; San Francisco, California; and Washington, D.C. Based on our preliminary observations, the laptops appear to work well and listers reported their training was satisfactory. We are in the process of discussing these and other observations with the Bureau.

Bureau Needs to Finalize Field Data Collection Plans

The Bureau’s largest and most costly field operation is nonresponse follow-up. The Bureau estimates that it will employ over 600,000 temporary workers to collect data from about 47 million nonrespondening households over the course of 10 weeks in 2010. On April 3, 2008, the Bureau announced that it would no longer use handheld computers for nonresponse follow-up and would instead change to a paper-based nonresponse follow-up operation. According to the Bureau, this change added between $2.2 billion to $3 billion to the total cost of the census.

In May 2008, the Bureau issued a plan that covered major components of the paper-based nonresponse follow-up. Bureau officials said that they are developing a more detailed plan that would describe 2010 nonresponse follow-up operations and systems, workflow, major milestones, and roles and responsibilities of different census divisions. Although the plan was due in January 2009, it has yet to be completed. Because this plan serves as a road map for monitoring the development and implementation of nonresponse follow-up, it will be important for the Bureau to complete this plan.

The Bureau has changed plans for many aspects of nonresponse follow-up, and officials are determining which activities and interfaces will be tested and when this testing will occur. Although the Bureau has carried out a paper-based follow-up operation in past decennials, the 2010 Census includes new procedures and system interfaces that have not been tested under census-like conditions because they were dropped from the dress rehearsal. Bureau officials acknowledged the importance of testing new and modified nonresponse follow-up activities and system interfaces to reduce risk, but have not yet developed detailed testing plans. Given the number of tasks at hand and the increasingly shorter time frame in which to accomplish them, it will be important for the Bureau to monitor the development of these testing plans, coordinate this testing with other activities, and ensure that testing occurs in time to take corrective actions, if needed.

In our previous work, we have highlighted the importance of sound risk management in planning for the decennial census. The Bureau has strengthened aspects of its risk management process. For example, in July 2008, the Bureau identified 31 nonresponse follow-up risks, such as lower than expected enumerator productivity. However, it has not developed mitigation plans for these risks. Officials said that they are reevaluating these risks and plan to develop mitigation plans for high- and medium- priority nonresponse follow-up risks starting in spring 2009. However, the Bureau has not yet determined when these plans will be completed.

Coverage Follow-up Operation Needs to Be Finalized

One of the Bureau’s long-standing challenges is resolving conflicting information respondents provide on census forms. This problem can occur, for example, when the number of household members reported on a completed form differs from the number of persons for whom information is provided. In such instances, the Bureau attempts to reconcile the data during the coverage follow-up operation. For 2010, the Bureau plans to expand the scope of this operation and include two questions—known as coverage probes— on the census form to identify households where someone may have been missed or counted incorrectly (see fig. 3).

However, after testing the probes earlier in the decade, the Bureau found that one of the probes was problematic in identifying persons potentially missing from the count. Although these probes were included on the forms mailed out during the dress rehearsal, the coverage follow-up operation did not include cases from nonresponse follow-up, which was canceled from the dress rehearsal. In the absence of a final test of the coverage probes in nonresponse follow-up, the effectiveness of the information generated by the probes is uncertain.

Fieldwork Management System for Most Operations Still Needs to Be Specified and Programmed

A successful census depends, in large part, on the work carried out in the local census offices. For the 2010 Census, this field work cannot be accomplished without a properly functioning Operations Control System (OCS). This system is intended to provide managers with essential real- time information, such as worker productivity and completion rates for field operations. It also allows managers to assign or reassign cases among workers. If the system does not work as intended, it could bog down or delay field operations and introduce errors into data collected.

Initially, the Bureau had planned to use a contractor to develop OCS to manage the workflow for those operations relying on paper-based processes, such as group quarters enumeration and nonresponse follow- up. However, in August 2008, the Bureau created an internal program to develop OCS and other related infrastructure that are needed to support these operations. The Bureau is still in the process of developing OCS for paper-based operations.

Although the Bureau has established a high-level schedule for testing of OCS, it has not yet finalized the requirements needed to begin its programming or developed a detailed schedule for conducting additional tests. Further, the Bureau has not yet fully defined how OCS will work together with other systems. Bureau officials said that the lack of detailed plans for operations, such as nonresponse follow-up, makes it difficult to finalize requirements for OCS or its testing plans. Our work on IT systems testing has shown that without adequate oversight and more comprehensive guidance, the Bureau cannot ensure that it is thoroughly testing its systems and properly prioritizing testing activities before the 2010 Census.

The Bureau Has Taken Steps to Improve Map Production, but Faces a Tight Schedule

The Bureau estimates that it will need to produce approximately 30 million different map files from which 80 million paper maps will be printed to assist census workers in locating addresses in major census operations. The quality of maps and the timing of map printing are critical to the success of the census. In addition, many map production and printing activities must be conducted in sequence with no time to spare, putting at risk the Bureau’s ability to print its maps on time. The Bureau has taken positive steps to meet its requirements for map production and printing for 2010. For example, in June 2008, the Bureau decided to produce a generic map type in lieu of several operation-specific versions to reduce the number of map files to be produced. Furthermore, the Bureau is preparing to print most of its maps at the local census offices rather than at the regional offices, reducing the need to coordinate map delivery to the local census offices. In addition, the Bureau has replaced its labor-intensive quality assurance process with integrated, automated processes. These steps taken to improve workflow will become particularly important as the Bureau works to produce and print maps on an already compressed schedule.

The Bureau’s schedule for producing and printing maps does not allow for any delays in receiving data from other operations or from the contractor delivering map files. For example, the Bureau intends to include map information from address canvassing, which ends in July 2009, in maps that will be used to validate locations of group quarters, which begins in September 2009. Bureau officials have stated that the turnaround time between these operations allows no slippage, and if these data are received late, an entire chain of subsequent map production steps would be thrown off schedule. Furthermore, according to the Bureau, local census offices need to receive map files from the contractor in time to print maps for certain field operations by January 8, 2010. However, the contractor is not scheduled to finish delivering the map files until January 19, 2010. Bureau officials said that they have taken steps to ensure that the necessary map files are delivered in time for printing but are still working to resolve the discrepancy.

Census Marketing Programs Will Need to Improve Response Rates of Historically Undercounted Groups

The Bureau goes to great lengths to reduce the undercount, especially among those groups likely to be undercounted at a higher rate than others, such as minorities and renters. For example, the Bureau plans to provide language assistance guides in 59 languages for completing the census, an increase from 49 languages in 2000. For the first time in 2010, the Bureau plans to send bilingual questionnaires to approximately 13 million households that are currently likely to need Spanish language assistance, as determined by analyzing recent data from a related Bureau survey program.

The Bureau also plans to deploy a multifaceted communications campaign consisting of, among other efforts, paid advertising and the hiring of as many as 680 partnership staff who will be tasked with reaching out to local governments, community groups, and other organizations in an effort to secure a more complete count. Overall, the Bureau estimates it will spend around $410 million on its communication efforts for the 2010 Census;. However, in constant 2010 dollars, this amount is somewhat less than the approximately $480 million that the Bureau spent marketing the 2000 Census.

Although the effects of the Bureau’s communication efforts are difficult to measure, the Bureau reported some positive results from its 2000 Census marketing efforts with respect to raising awareness of the census. For example, four population groups—non-Hispanic Blacks, non-Hispanic Whites, Asians, and Native Hawaiians—indicated they were more likely to return the census form after the 2000 Census partnership and marketing program than before its onset. However, a Bureau evaluation demonstrated only a limited linkage between the partnership and marketing effort and improvements in actual census mail return behavior for these or other groups. Put another way, while the Bureau’s marketing activities might raise awareness of the census, a remaining challenge is converting that awareness into an actual response. Other marketing challenges include long-standing issues such as the nation’s linguistic diversity and privacy concerns, as well as a number of newly emerging concerns such as local campaigns against illegal immigration and a post- September 11 environment that could heighten some groups’ fears of government agencies.

Managing and Testing of Information Technology Systems Remain a Concern

Since 2005, we have reported on weaknesses in the Bureau’s management of its IT acquisitions, and we remain concerned about the Bureau’s IT management and testing of key 2010 Census systems. For example, in October 2007, we reported on the status of and plans for key 2010 Census IT acquisitions and whether the Bureau was adequately managing associated risks. We found critical weaknesses in the Bureau’s risk management practices, including those associated with risk identification, mitigation, and oversight. We later presented multiple testimonies on the Bureau’s progress in addressing significant risks facing the 2010 Census. In particular, the Field Data Collection Automation (FDCA) program, which includes the development of handheld computers for the address canvassing operation and the systems, equipment, and infrastructure that field staff will use to collect data, has experienced significant problems.

For example, in March 2008, we testified that the FDCA program was experiencing schedule delays and cost increases, and was contributing significant risk to the 2010 Census. At that time, we highlighted our previous recommendations to better manage FDCA and the other IT acquisitions.

In response to our findings and recommendations, the Bureau has taken several steps to improve its management of IT for the 2010 Census. For example, the Bureau has sought external assessments of its activities from independent research organizations, implemented a new management structure and management processes and brought in experienced personnel to key positions, and improved several reporting processes and metrics. In part, due to our review of the FDCA program, the Bureau requested a revised cost proposal for the FDCA program which resulted in a cost reduction of about $318 million for the remaining 5-year life-cycle of the program.

As we have previously reported, operational testing planned during the census dress rehearsal would take place without the full complement of systems and functionality that was originally planned, and it was unclear whether the Bureau was developing plans to test all interrelated systems and functionality. At your request, we reviewed the status and plans of testing of key 2010 Census systems. As stated in our report, which we are releasing today, we found that the Bureau has made progress in conducting systems, integration, and end-to-end testing, but critical testing still remains to be performed before systems will be ready to support the 2010 Census, and the planning, execution, and monitoring of its testing needs much improvement. We are making 10 recommendations for strengthening the Bureau’s testing of 2010 Census systems. Those recommendations address improvements needed in test planning, management, and monitoring. In response to our report, the Department of Commerce and Bureau stated they had no significant disagreements with our recommendations.

Concluding Observations

In summary, little more than a year remains until Census Day. At a time when major testing should be completed and there should be confidence in the functionality of key operations, the Bureau instead finds itself managing late design changes and developing testing plans. The Bureau has taken some important steps toward mitigating some of the challenges that it has faced to date, yet much remains uncertain, and the risks to a successful decennial census remain.

Addressing these risks and challenges will be critical to the timely completion of a cost-effective census, and it will be essential for the Bureau to develop plans for testing systems and procedures not included in the dress rehearsal, and for Congress to monitor the Bureau’s progress.

As always, we look forward to working with Congress in assessing the Bureau’s efforts to overcome these hurdles to a successful census, and providing regular updates on the rollout of the decennial in the critical months that lie ahead.

Mr. Chairman and members of this Subcommittee, this concludes our statement. We would be happy to respond to any questions that you or members of the Subcommittee may have at this time.

If you have any questions on matters discussed in this testimony, please contact Robert Goldenkoff at (202) 512-2757 or David A. Powner at (202) 512-9286 or by e-mail at goldenkoffr@gao.gov or pownerd@gao.gov. Other key contributors to this testimony include Sher’rie Bacon, Thomas Beall, Steven Berke, Vijay D’Souza, Elizabeth Fan, Richard Hung, Andrea Levine, Signora May, Ty Mitchell, Catherine Myrick, Lisa Pearson, Kathleen Padulchick, Crystal Robinson, Melissa Schermerhorn, Cynthia Scott, Karl Seifert, Jonathan Ticehurst, Timothy Wexler, and Katherine Wulff.

Related GAO Products

High-Risk Series: An Update. GAO-09-27. Washington, D.C.: January 2009. 2010 Census: The Bureau’s Plans for Reducing the Undercount Show Promise, but Key Uncertainties Remain. GAO-08-1167T. Washington, D.C.: September 23, 2008. 2010 Census: Census Bureau’s Decision to Continue with Handheld Computers for Address Canvassing Makes Planning and Testing Critical. GAO-08-936. Washington, D.C.: July 31, 2008. 2010 Census: Census Bureau Should Take Action to Improve the Credibility and Accuracy of Its Cost Estimate for the Decennial Census. GAO-08-554. Washington, D.C.: June 16, 2008. 2010 Census: Plans for Decennial Census Operations and Technology Have Progressed, But Much Uncertainty Remains. GAO-08-886T. Washington, D.C.: June 11, 2008. 2010 Census: Bureau Needs to Specify How It Will Assess Coverage Follow-up Techniques and When It Will Produce Coverage Measurement Results. GAO-08-414. Washington, D.C.: April 15, 2008. 2010 Census: Census at Critical Juncture for Implementing Risk Reduction Strategies. GAO-08-659T. Washington, D.C.: April 9, 2008.

Information Technology: Significant Problems of Critical Automation Program Contribute to Risks Facing 2010 Census. GAO-08-550T. Washington, D.C.: March 5, 2008.

Information Technology: Census Bureau Needs to Improve Its Risk Management of Decennial Systems. GAO-08-259T. Washington, D.C.: December 11, 2007. 2010 Census: Planning and Testing Activities Are Making Progress. GAO-06-465T. Washington, D.C.: March 1, 2006.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.