2010 Census: Plans for Census Coverage Measurement Are on Track, but Additional Steps Will Improve Its Usefulness
Released: 2010-04-23T00:00:00

Letter

Background

The Bureau puts forth tremendous effort to conduct a complete and accurate count of the nation’s population; nonetheless, some degree of coverage error is inevitable because of the inherent complexity of counting the nation’s large and diverse population and limitations in census-taking methods. These census coverage errors can take a variety of forms, including a person missed (an undercount), a person counted more than once (an overcount), or a person who should not have been counted, such as a child born after Census Day (another type of overcount). To further understand and to inform users about the quality of the census, the Bureau has been evaluating coverage measurement for more than 50 years. While initial evaluations relied solely on demographic analysis— population estimates based on birth and death rates as well as immigration estimates—modern coverage measurement began with the 1980 Census when the Bureau began also comparing census counts to survey results from an independent coverage measurement sample of the population. Using statistical methods, the Bureau generated detailed measures of the differences among undercounts of particular ethnic, racial, and other groups, which have been referred to as “differential undercounts.” These measures were also generated for the 1990 and 2000 censuses.

Although the Bureau considered doing so in earlier decades, it has never used its estimates of coverage error to adjust census data. In 1980, the Director of the Census Bureau decided that potential adjustments would be flawed due to missing and inaccurate data. In 1990, the Bureau recommended statistically adjusting census data; however, the Secretary of Commerce determined that the evidence to support an adjustment was inconclusive and decided not to adjust. For the 2000 Census, a 1999 Supreme Court ruling held that the Census Act prohibited the use of statistical sampling to generate population data for apportioning the House of Representatives. The Bureau had planned to produce apportionment numbers using traditional census-taking methods, and provide statistically adjusted numbers for non-apportionment uses of the data such as congressional redistricting and allocating federal funds. The Bureau later determined that its statistical estimates did not provide a reliable measure of census accuracy and could not be used to adjust the non-apportionment census data.

The Bureau is not planning to use CCM to adjust the 2010 Census. Instead, CCM will be used to evaluate coverage error to improve the 2020 and future censuses, and will focus on estimating various components of census coverage in addition to net coverage errors—the net effect on coverage after undercounts and overcounts are considered. These components of coverage include correct enumerations, erroneous enumerations (people or housing units that were counted but should not have been), and omissions (people or housing units that were not counted but should have been). The Bureau also plans to include imputations (counts of people and their characteristics that are provided for nonresponding households, usually based on responses from others under similar circumstances, such as from surrounding households).

Statistical measurements of census coverage are obtained by comparing and matching the housing units and people counted by the independent coverage measurement sample to those counted by the census in and around the sample areas. The Bureau has developed separate address lists—one for the entire nation of over 134 million housing units that it will use to conduct the census and one for coverage measurement sample areas—and will collect each set of data through independent operations. For the 2010 Census, census operations began collecting population data from households in January 2010 and will continue through the end of July, while CCM operations will collect data by visiting each of the housing units in the coverage measurement sample during an operation called Person Interviewing from August through October.

The statistical methodology the Bureau uses to estimate net coverage errors relies on an assumption that the chance that a person is counted by the census is not affected by whether he or she is counted in the independent coverage measurement sample, or vice versa. Because violating this “independence” assumption can bias coverage estimates, the Bureau takes special measures to maintain CCM’s separation from the census, such as developing a separate address list for the coverage measurement sample discussed above.

The Bureau Has Finalized Decisions in Some Key Areas Since Our 2008 Report

Since our April 2008 report, the Bureau has finalized its plans in key areas of the CCM program including CCM’s goals, the timing of operations, and the timing and types of results to be produced. Planning continues in other areas, such as developing estimation methods, evaluating the CCM program, and implementing its Master Trace Project. Continued progress and adherence to schedule will be important to ensure that the Bureau carries out CCM in order to meet its goal of improving the 2020 Census.

For example, in our 2008 report, we recommended that the Bureau provide decision points and plans for evaluating CCM. In September 2009, the Bureau finalized its list of 22 planned evaluations for the 2010 Census, which included five that address specific methodological or procedural topics within the CCM program. However, all study plans are not due to be completed until April 2010. In addition, while the deadlines for finalizing CCM estimation methods have not yet passed, the Bureau has many of its default plans already in place. Default plans allow the Bureau to move forward on schedule even if new plans have not been developed. Table 1 shows the status of the Bureau’s plans for the design of CCM in each of these areas.

Recent Changes to CCM Data Collection Plan Could Improve Quality of CCM Data

In September 2009, shortly after taking office, the Director of the Census Bureau asked the staff responsible for CCM to review its CCM design and propose specific changes that would reduce the introduction of nonsampling error—such as human errors made when recording data during interviews— into CCM and its resulting estimates. The staff proposed numerous changes intended to reduce error in collected data. They also proposed an additional research study. The Director approved all of these proposals in mid-December 2009. Key changes included: increasing the reinterview rates for CCM field work to improve quality assurance; increasing training time for short-term workers hired to conduct door-to- door visits during the Person Interviewing operation to improve interview techniques for local or other special situations due to current economic conditions (such as people who became homeless or have had to move frequently during the housing crisis); increasing supervisor-to-employee field staffing ratios to improve quality and monitoring of field work at each level; and adding a telephone-based study to collect information about how well respondents recall information about their residence and possible movement since Census Day.

In addition, the decision authorized a nearly 45 percent reduction in the CCM sample size that the Bureau believes would generate the cost savings to pay for the other changes. Our understanding of the issues suggests that these are reasonable efforts to improve survey quality. The Bureau’s reduction in sample size will reduce precision of the estimates, yet the proposed changes should reduce nonsampling errors and thus provide users with more reliable estimates. For example, the Bureau expects short-term CCM workers to make fewer mistakes in identifying temporary or unconventional housing units when they have received additional training specific to their local circumstances, such as in areas with large numbers of seasonal or displaced workers.

The Bureau Needs to Take Additional Actions to Improve the Usefulness of CCM

The Bureau’s actions to finalize some areas of CCM program planning are important steps in the right direction. Still, in some cases, it will be important for the Bureau to take additional actions to help ensure the results of CCM are as useful as they could be to inform Bureau decisions on improving future censuses and coverage measurement efforts.

For example, the Bureau could better document how CCM results will be used as part of the planning process for the 2020 Census. Indeed, the Bureau has already begun laying the foundation for its 2020 planning efforts. These early planning efforts are described in a series of decision memorandums issued in the summer of 2009, and include milestones leading up to a census test in April 2014, descriptions of planning phases, and a list of the various organizational components that conduct the census. Although these planning documents explicitly state the importance of relying on the 2010 Census Evaluation and Testing program—an ongoing assessment effort separate from CCM that, like CCM, is designed to improve future operations—the Bureau has not yet taken similar steps to integrate the CCM program with 2020 planning. In addition, the Bureau does not have specific plans in its CCM program goals to do anything beyond producing CCM results.

Bureau officials have maintained that until it produces CCM results, it is difficult to determine how to use CCM data to improve the design of future decennials. While we agree with the Bureau that the results will determine the specifics of any potential design improvements, it is not premature to consider how the earliest results from CCM—scheduled for early 2012— could help inform early planning and decisions. Importantly, by creating a “roadmap” that describes, for example, what the Bureau might learn from CCM or how the results might feed into early 2020 Census planning, the Bureau will better ensure that there are no gaps or overlaps in the use of CCM in early 2020 planning.

The Bureau’s Master Trace Project is another area where additional efforts are needed to ensure useful CCM results. The Bureau initiated the Master Trace Project in September 2009, to facilitate the use of census and CCM data for future research. Currently, Bureau data are collected and archived in different types of datasets and systems. The Master Trace Project is intended to ensure that these datasets and systems can be used together, or linked, to support detailed research into the causes of census coverage problems and facilitate research on the possible interactions of future operations. For example, a researcher might want to see if there is a relationship between the Bureau’s employment practices and the magnitude of an undercount in a particular area. In so doing, the researcher may want to compare census payroll, overtime, and other human capital data to the data from that region collected and processed by census and CCM. Such datasets would not ordinarily be linked during the census.

The Bureau has not yet taken the steps needed to ensure that such research across different data systems would be possible. The Bureau held a meeting in December 2009 with staff responsible for many major decennial systems and obtained agreement about the importance of data retention for this project; however, the Bureau has not yet resolved how it would make the project happen. In particular, the Bureau has not yet completed an inventory of the census databases that might be of potential interest for future research, identified which archived versions might be most useful, or mapped out how they might be archived and linked. Until this is done, it is unclear that Bureau or other researchers will have access to census operational data that they need to fully analyze the census coverage errors that CCM may uncover. Moving forward, it will be important for the Bureau to perform the initial assessment of its data systems, identify gaps in data collection, and identify any other related steps to ensure that key data can be linked. Doing this quickly will also be important as Census 2010 is underway and it could become increasingly difficult to make changes to database structures or archival and data storage plans if the Bureau’s assessments determine that changes are necessary.

A third area where the Bureau needs to do additional work is in assessing how the timing of CCM data collection might adversely affect CCM findings. When planning CCM, the Bureau faced the challenge of determining the optimal time to launch the CCM data collection operation, known as Person Interviewing (PI). If the Bureau starts PI too early, it increases the chance that it overlaps with census data collection, possibly compromising the independence of the two different operations and introducing a “contamination bias” error into CCM data. If the Bureau starts PI too late, it increases the chance that respondents will not accurately remember household information from Census Day, April 1, introducing error (known as “recall bias”) in the CCM count. Both types of errors—contamination bias and recall bias—could affect the Bureau’s conclusions about the accuracy of the census. An understanding of the trade-offs between these two types of biases would be important in future decisions regarding the optimal timing of PI.

In early 2009, based on concerns by the National Academy of Sciences (NAS) and other stakeholders about the relative lateness in the start date of PI and its possible impact on the quality of CCM findings, the Bureau considered whether to start PI 6 weeks earlier than planned. In June 2009, the Bureau decided to keep the originally scheduled start on August 14, 2010. Bureau memorandums and officials justified the decision largely because of concern that it was too late in the planning process to make a change in the complex CCM schedule. The memorandums cited gaps in knowledge about the impact of timing on recall bias, presented research with differing conclusions about the extent of contamination in prior census tests, and justified the recommendation to not change the start date by the operational challenges faced to make the change.

Bureau officials have also explained that the goal of using coverage measurement in 2000 to possibly adjust the census-created time pressures in 2000 that forced an early PI, and because such time pressures do not exist for PI in 2010, it is scheduled to begin more than 4 months after Census Day.

By comparison, during the 2000 Census, the Bureau launched PI in April 2000 and had completed about 99 percent of its data collection by the end of the first week of August 2000, a week earlier than the scheduled 2010 PI start date. An extensive 2000 Census evaluation found no evidence of contamination bias caused by the earlier start of PI in 2000. Related Bureau research since then has also found no significant evidence of contamination bias during census tests, although one test found that census results could be affected. Yet Bureau officials remained concerned about the possibility, since the CCM questions are similar to follow-up questions used in one of the 2010 census follow-up operations. Furthermore, parts of this census operation are new in 2010, and end later than similar operations did in 2000.

Moving forward, additional research on the trade-offs between recall bias and contamination errors could help the Bureau more fully understand the implications of choosing various start times for PI on the resulting estimates of coverage error and better determine the optimal timing of PI in future censuses. Currently, the Bureau has a telephone-based study planned in order to measure recall errors, which could provide additional information about when recall errors are more likely to occur. However, this study is limited to certain types of recall error, and the Bureau does not have an evaluation planned to measure possible contamination between the new, much later, parts of census follow-up and CCM data collection or to assess the trade-offs between the biases from starting earlier compared to starting later. Such additional study after the 2010 Census could provide the Bureau better information about the trade-offs in data quality from potential contamination and recall biases and provide a better basis for determining the optimal scheduling of coverage measurement operations.

Conclusions

Assessing the accuracy of the census is an essential step in improving current and future censuses. The Bureau has made progress on designing and planning for its CCM program and continues activity to complete the plan. Additional actions in three CCM planning areas may further improve CCM or its usefulness to the 2020 Census.

Specifically, the Bureau has stated the importance of using 2010 evaluation data such as CCM’s for 2020 Census design, but has not yet taken steps to link CCM data to an improved 2020 design. If the Bureau is to best achieve its goal of using CCM to help improve the 2020 Census, it will need to integrate planning for any follow-up work on CCM results or data with the other early planning already underway for Census 2020.

Second, the Bureau has many different processes that come together in the conduct of a decennial census, and archived data on those processes could provide useful information to researchers trying to figure out what worked well and what did not. The Master Trace Project can help researchers link CCM results and data to potential design changes for Census 2020. Determining which data need to be linked or archived to enable future linkage within the project can help prevent gaps in 2010 data that might hinder the project’s ability to help identify improvements for the 2020 Census.

Third, the timing of CCM’s primary data collection operation—Person Interviewing—involves trade-offs between reducing contamination bias and reducing recall error that the Bureau did not have conclusive information on. Since 2010 Person Interviewing is starting 1 week after a similar operation ended in 2000, the chance of introducing recall bias errors into CCM data is higher in 2010 than it was in 2000. Although the Bureau has a study planned to measure some recall errors, there is no study planned to measure contamination between the new parts of census follow-up—which use questions similar to those asked by CCM and finish much later than follow-up did in 2000—and CCM or to assess the trade- offs between the two types of biases in timing decisions. Targeted research after the 2010 Census on the relationship between the timing of data collection and the trade-offs between these types of errors before the 2020 Census and its coverage measurement efforts could help the Bureau better determine the optimal timing of future data collection operations.

Recommendations for Executive Action

We recommend that the Secretary of Commerce require the Director of the U.S. Census Bureau to take the following three actions to improve the usefulness of CCM for 2020: To help the Bureau achieve its goal of using CCM to improve the 2020 Census, better document links between the 2010 CCM program and 2020 Census planning, integrating the goal of using the CCM program to improve Census 2020, such as with CCM results and data, into those broader plans for 2020.

To ensure that Bureau datasets from the 2010 Census can be used with other Bureau datasets to support research that could improve the census and CCM, complete the Master Trace Project’s assessment of how key census and CCM data systems are, or can be, linked to each other; identify any potential data gaps; and identify other related steps for future action.

To help the Bureau better determine the optimal timing of future coverage measurement data collection, fully assess the trade-offs between starting the data collection earlier, with the possibility of introducing contamination errors, and starting later, with the possibility of introducing recall errors.

Agency Comments and Our Evaluation

The Secretary of Commerce provided written comments on a draft of this report on April 5, 2010. The comments are reprinted in appendix I. Commerce generally agreed with the overall findings and recommendations and appreciated our efforts in helping the Census Bureau develop a successful evaluation plan for the 2020 Census. Commerce also provided additional information and comments on certain statements and conclusions in the report.

With respect to our second recommendation to complete the Master Trace Project’s assessment of linking key census and CCM data systems, to identify any potential data gaps, and to identify other related steps for future action, Commerce maintained that it would be taking action to preserve adequate documentation and maximize the amount of data retained from each major decennial system. We commend the Bureau for committing to these steps and encourage its follow-through on them and its identification of remaining data gaps and additional steps needed.

With respect to our third recommendation to fully assess the trade-offs between two types of error related to starting CCM data collection either earlier or later relative to Census Day, Commerce responded that (1) it is too late to create a new study for 2010 Census; (2) it considers a Bureau contamination study from 2000 to be definitive; and (3) it has recently developed a study on recall bias to try to measure some of the effects of scheduling CCM data collection at various periods of time following the census enumeration. We agree that it is too late to attempt any additional unplanned data collection during the 2010 Census, and we revised our discussion to clarify our intent that the recommended research be conducted after the 2010 Census. We also recognize the thoroughness of the 2000 contamination study the Bureau cites, commend the Bureau on undertaking additional study of recall bias, and look forward to reviewing its study plans when they are available. However, we recommended research comparing trade-offs between the two types of errors at a variety of start dates for CCM data collection—something the 2000 study did not discuss and something it is unclear that a study of only recall bias will achieve. Furthermore as we discussed in our draft report, the Bureau expressed concerns over possible contamination between CCM and new parts of census follow-up in 2010—parts that were introduced after the 2000 study and that were not included in the scope of the 2000 study. We clarified our discussion of this in the report to better focus on the need for research that relates the trade-offs between the two types of error at different timing of data collection.

Commerce provided additional information that in response to advice from various advisory panels and after additional research, it would soon make public its proposed geographic levels for CCM estimates. We reflected this decision in table 1 of our report.

Finally, Commerce provided additional information about its plans to produce highly technical documentation of the results of CCM estimation including modeling, missing data, and errors in the estimates in a series of memorandums as it did for Census 2000. We reflected this decision in table 1 of this report.

We are sending copies of this report to the Secretary of Commerce, the Director of the U.S. Census Bureau, and interested congressional committees. The report also is available at no charge on GAO’s Web site at http://www.gao.gov.

If you have any questions about this report please contact me at (202) 512- 2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report were Jeff Dawson, Dewi Djunaidy, Ron Fecso (Chief Statistician), Andrea Levine, Ty Mitchell, Melanie Papasian, and Tamara F. Stenzel.

Appendix I: Comments from the Department of Commerce