2010 Census: Plans for Census Coverage Measurement Are on Track, but Additional Steps Will Improve Its Usefulness
Published: 2010-04-23T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Assessing the accuracy of the census is essential given that census data are used to apportion seats in Congress, to redraw congressional districts, and for many other public and private purposes. The U.S. Census Bureau's (Bureau) Census Coverage Measurement program (CCM) is to assess the accuracy of the 2010 Census and improve the design of operations for the 2020 Census. In April 2008, GAO recommended that the Bureau identify how it would relate CCM results--where the 2010 Census was accurate and inaccurate--to census operations to improve future censuses. Knowing where the 2010 Census was inaccurate can help inform research to improve the 2020 Census. GAO was asked to examine (1) the status of CCM planning and (2) the effects of design decisions since GAO issued its April 2008 report. GAO reviewed Bureau documents related to CCM design and National Academy of Sciences reports, and interviewed responsible Bureau officials.
section_title: What GAO Found
paragraphs: Since GAO's April 2008 report, the Bureau has finalized plans for 2010 CCM goals, the timing of operations, and the types of results to be produced. Planning continues in other areas, such as developing estimation methods, evaluating the CCM program, and implementing its Master Trace Project, which would enable the Bureau to link its datasets and systems to support a broad range of research. The deadlines for some of these plans have not yet passed, but the Bureau already has default plans in place in case further changes do not occur. In mid-December, the Director decided to make some additional changes to the CCM program to improve the quality of CCM results. GAO found that additional actions on Bureau decisions may make CCM more useful in informing Bureau decisions on future census and coverage measurement efforts: (1) The Bureau's 2020 planning efforts are described in a series of decision memoranda issued in the summer of 2009. However, the Bureau has not yet taken steps to integrate CCM results with early 2020 planning to prepare for a census test in 2014. By describing, for example, what the Bureau might learn from CCM or how the results might feed into 2020 Census planning, the Bureau will better ensure that there are no gaps or overlaps in the use of CCM for early 2020 planning. (2) In September 2009, the Bureau began its Master Trace Project, which is intended to ensure that its datasets and systems can be used together to support detailed research into the causes of census coverage problems and facilitate research on the possible interactions of future operations. At the time of this review, the Bureau had not yet completed an inventory of the census databases that might be of potential interest for future research, identified which archived versions might be most useful, or mapped out how they might be archived and linked. Doing this quickly will be important as the census is already underway and it will be difficult to make changes to database structures or archival and data storage plans if the Bureau's assessments determine that changes are necessary. (3) The Bureau reviewed its previous decision to start CCM's Person Interviewing operation later than it did in 2000, and decided in June 2009 not to change it. However, the Bureau does not have a plan to assess the trade-offs in error between earlier and later start dates. Additional research on the trade-offs of different start dates could help the Bureau more fully understand the implications of CCM timing decisions on the resulting estimates of coverage error and better determine the optimal timing of Person Interviewing in future censuses.

section_title: Letter
paragraphs: Data from the decennial census—a constitutionally mandated count of the  national population—are used to apportion seats in Congress, redraw  congressional districts, allocate billions of dollars in federal assistance to  state and local governments each year, and inform the planning and  investment decisions of numerous public and private sector entities. The  census aims to locate and count people—only once—in the right place,  and collect complete and correct information about them. Because census  data are central to so many critical functions, it is essential to assess  census accuracy and improve the process when needed.
In April 2008, we reported that the U.S. Census Bureau (Bureau) needed to  finalize plans for its Census Coverage Measurement (CCM) program—the  effort intended to assess the accuracy of the 2010 Census and improve the  design of operations for the 2020 Census—and made related  recommendations to the Bureau’s parent agency, the Department of  Commerce. In particular, we noted that the Bureau should produce plans  that include (1) a description of when it will provide CCM results, (2) how  it plans to report its CCM results in relation to census operations, and (3)  key decision points and plans for evaluating aspects of the CCM.  Commerce has taken steps to implement actions for each  recommendation. Since we issued our report, the National Academy of  Sciences panel on Correlation Bias and Coverage Measurement in the 2010  Decennial Census released its final report on the Bureau’s plans for CCM,  recommending numerous steps to enhance the value of the program.
You asked us to examine the current plan for the CCM program and to  provide (1) an update on the status of the planning since our April 2008  report, and (2) the potential effects of major CCM decisions on the quality  and usefulness of CCM data. To meet these objectives, we reviewed  Bureau documents related to CCM design and National Academy of  Sciences (NAS) reports, and we interviewed Bureau officials responsible  for CCM. We focused primarily on the Bureau’s decisions in the following  areas: 2010 CCM goals, timing of operations and reporting results,  estimation methods, results to be produced, program evaluation, and  implementing its Master Trace Project. Specifically, to update the status of  CCM, we reviewed scheduling documents and decision memorandums. To  identify the potential effects we reviewed decision memorandums and  related justifications, prior Bureau and NAS research and our reports  related to CCM and evaluation. We conducted our work from June 2009 to  February 2010 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The Bureau puts forth tremendous effort to conduct a complete and  accurate count of the nation’s population; nonetheless, some degree of  coverage error is inevitable because of the inherent complexity of  counting the nation’s large and diverse population and limitations in  census-taking methods. These census coverage errors can take a variety of  forms, including a person missed (an undercount), a person counted more  than once (an overcount), or a person who should not have been counted,  such as a child born after Census Day (another type of overcount). To  further understand and to inform users about the quality of the census, the  Bureau has been evaluating coverage measurement for more than 50  years. While initial evaluations relied solely on demographic analysis— population estimates based on birth and death rates as well as immigration  estimates—modern coverage measurement began with the 1980 Census  when the Bureau began also comparing census counts to survey results  from an independent coverage measurement sample of the population.  Using statistical methods, the Bureau generated detailed measures of the  differences among undercounts of particular ethnic, racial, and other  groups, which have been referred to as “differential undercounts.” These  measures were also generated for the 1990 and 2000 censuses.
Although the Bureau considered doing so in earlier decades, it has never  used its estimates of coverage error to adjust census data. In 1980, the  Director of the Census Bureau decided that potential adjustments would  be flawed due to missing and inaccurate data. In 1990, the Bureau  recommended statistically adjusting census data; however, the Secretary  of Commerce determined that the evidence to support an adjustment was  inconclusive and decided not to adjust. For the 2000 Census, a 1999  Supreme Court ruling held that the Census Act prohibited the use of  statistical sampling to generate population data for apportioning the  House of Representatives. The Bureau had planned to produce  apportionment numbers using traditional census-taking methods, and  provide statistically adjusted numbers for non-apportionment uses of the  data such as congressional redistricting and allocating federal funds. The  Bureau later determined that its statistical estimates did not provide a  reliable measure of census accuracy and could not be used to adjust the  non-apportionment census data.
The Bureau is not planning to use CCM to adjust the 2010 Census. Instead,  CCM will be used to evaluate coverage error to improve the 2020 and  future censuses, and will focus on estimating various components of  census coverage in addition to net coverage errors—the net effect on  coverage after undercounts and overcounts are considered. These  components of coverage include correct enumerations, erroneous  enumerations (people or housing units that were counted but should not  have been), and omissions (people or housing units that were not counted  but should have been). The Bureau also plans to include imputations  (counts of people and their characteristics that are provided for  nonresponding households, usually based on responses from others under  similar circumstances, such as from surrounding households).
Statistical measurements of census coverage are obtained by comparing  and matching the housing units and people counted by the independent  coverage measurement sample to those counted by the census in and  around the sample areas. The Bureau has developed separate address  lists—one for the entire nation of over 134 million housing units that it will  use to conduct the census and one for coverage measurement sample  areas—and will collect each set of data through independent operations.  For the 2010 Census, census operations began collecting population data  from households in January 2010 and will continue through the end of  July, while CCM operations will collect data by visiting each of the housing  units in the coverage measurement sample during an operation called  Person Interviewing from August through October.
The statistical methodology the Bureau uses to estimate net coverage  errors relies on an assumption that the chance that a person is counted by  the census is not affected by whether he or she is counted in the  independent coverage measurement sample, or vice versa. Because  violating this “independence” assumption can bias coverage estimates, the  Bureau takes special measures to maintain CCM’s separation from the  census, such as developing a separate address list for the coverage  measurement sample discussed above.
section_title: The Bureau Has Finalized Decisions in Some Key Areas Since Our 2008 Report
paragraphs: Since our April 2008 report, the Bureau has finalized its plans in key areas  of the CCM program including CCM’s goals, the timing of operations, and  the timing and types of results to be produced. Planning continues in other  areas, such as developing estimation methods, evaluating the CCM  program, and implementing its Master Trace Project. Continued progress  and adherence to schedule will be important to ensure that the Bureau  carries out CCM in order to meet its goal of improving the 2020 Census.
For example, in our 2008 report, we recommended that the Bureau  provide decision points and plans for evaluating CCM. In September 2009,  the Bureau finalized its list of 22 planned evaluations for the 2010 Census,  which included five that address specific methodological or procedural  topics within the CCM program. However, all study plans are not due to be  completed until April 2010. In addition, while the deadlines for finalizing  CCM estimation methods have not yet passed, the Bureau has many of its  default plans already in place. Default plans allow the Bureau to move  forward on schedule even if new plans have not been developed. Table 1  shows the status of the Bureau’s plans for the design of CCM in each of  these areas.
subsections: section_title: Recent Changes to CCM Data Collection Plan Could Improve Quality of CCM Data
paragraphs: In September 2009, shortly after taking office, the Director of the Census  Bureau asked the staff responsible for CCM to review its CCM design and  propose specific changes that would reduce the introduction of  nonsampling error—such as human errors made when recording data  during interviews— into CCM and its resulting estimates. The staff  proposed numerous changes intended to reduce error in collected data.  They also proposed an additional research study. The Director approved  all of these proposals in mid-December 2009. Key changes included:  increasing the reinterview rates for CCM field work to improve quality  assurance;  increasing training time for short-term workers hired to conduct door-to- door visits during the Person Interviewing operation to improve interview  techniques for local or other special situations due to current economic  conditions (such as people who became homeless or have had to move  frequently during the housing crisis);  increasing supervisor-to-employee field staffing ratios to improve quality  and monitoring of field work at each level; and    adding a telephone-based study to collect information about how well  respondents recall information about their residence and possible  movement since Census Day.
In addition, the decision authorized a nearly 45 percent reduction in the  CCM sample size that the Bureau believes would generate the cost savings  to pay for the other changes. Our understanding of the issues suggests that  these are reasonable efforts to improve survey quality. The Bureau’s  reduction in sample size will reduce precision of the estimates, yet the  proposed changes should reduce nonsampling errors and thus provide  users with more reliable estimates. For example, the Bureau expects  short-term CCM workers to make fewer mistakes in identifying temporary  or unconventional housing units when they have received additional  training specific to their local circumstances, such as in areas with large  numbers of seasonal or displaced workers.
section_title: The Bureau Needs to Take Additional Actions to Improve the Usefulness of CCM
paragraphs: The Bureau’s actions to finalize some areas of CCM program planning are  important steps in the right direction. Still, in some cases, it will be  important for the Bureau to take additional actions to help ensure the  results of CCM are as useful as they could be to inform Bureau decisions  on improving future censuses and coverage measurement efforts.
For example, the Bureau could better document how CCM results will be  used as part of the planning process for the 2020 Census. Indeed, the  Bureau has already begun laying the foundation for its 2020 planning  efforts. These early planning efforts are described in a series of decision  memorandums issued in the summer of 2009, and include milestones  leading up to a census test in April 2014, descriptions of planning phases,  and a list of the various organizational components that conduct the  census. Although these planning documents explicitly state the  importance of relying on the 2010 Census Evaluation and Testing  program—an ongoing assessment effort separate from CCM that, like  CCM, is designed to improve future operations—the Bureau has not yet  taken similar steps to integrate the CCM program with 2020 planning. In  addition, the Bureau does not have specific plans in its CCM program  goals to do anything beyond producing CCM results.
Bureau officials have maintained that until it produces CCM results, it is  difficult to determine how to use CCM data to improve the design of future  decennials. While we agree with the Bureau that the results will determine  the specifics of any potential design improvements, it is not premature to  consider how the earliest results from CCM—scheduled for early 2012— could help inform early planning and decisions. Importantly, by creating a  “roadmap” that describes, for example, what the Bureau might learn from  CCM or how the results might feed into early 2020 Census planning, the  Bureau will better ensure that there are no gaps or overlaps in the use of  CCM in early 2020 planning.
The Bureau’s Master Trace Project is another area where additional efforts  are needed to ensure useful CCM results. The Bureau initiated the Master  Trace Project in September 2009, to facilitate the use of census and CCM  data for future research. Currently, Bureau data are collected and archived  in different types of datasets and systems. The Master Trace Project is  intended to ensure that these datasets and systems can be used together,  or linked, to support detailed research into the causes of census coverage  problems and facilitate research on the possible interactions of future  operations. For example, a researcher might want to see if there is a  relationship between the Bureau’s employment practices and the  magnitude of an undercount in a particular area. In so doing, the  researcher may want to compare census payroll, overtime, and other  human capital data to the data from that region collected and processed  by census and CCM. Such datasets would not ordinarily be linked during  the census.
The Bureau has not yet taken the steps needed to ensure that such  research across different data systems would be possible. The Bureau held  a meeting in December 2009 with staff responsible for many major  decennial systems and obtained agreement about the importance of data  retention for this project; however, the Bureau has not yet resolved how it  would make the project happen. In particular, the Bureau has not yet  completed an inventory of the census databases that might be of potential  interest for future research, identified which archived versions might be  most useful, or mapped out how they might be archived and linked. Until  this is done, it is unclear that Bureau or other researchers will have access  to census operational data that they need to fully analyze the census  coverage errors that CCM may uncover. Moving forward, it will be  important for the Bureau to perform the initial assessment of its data  systems, identify gaps in data collection, and identify any other related  steps to ensure that key data can be linked. Doing this quickly will also be  important as Census 2010 is underway and it could become increasingly  difficult to make changes to database structures or archival and data  storage plans if the Bureau’s assessments determine that changes are  necessary.
A third area where the Bureau needs to do additional work is in assessing  how the timing of CCM data collection might adversely affect CCM  findings. When planning CCM, the Bureau faced the challenge of  determining the optimal time to launch the CCM data collection operation,  known as Person Interviewing (PI). If the Bureau starts PI too early, it  increases the chance that it overlaps with census data collection, possibly  compromising the independence of the two different operations and  introducing a “contamination bias” error into CCM data. If the Bureau  starts PI too late, it increases the chance that respondents will not  accurately remember household information from Census Day, April 1,  introducing error (known as “recall bias”) in the CCM count. Both types of  errors—contamination bias and recall bias—could affect the Bureau’s  conclusions about the accuracy of the census. An understanding of the  trade-offs between these two types of biases would be important in future  decisions regarding the optimal timing of PI.
In early 2009, based on concerns by the National Academy of Sciences  (NAS) and other stakeholders about the relative lateness in the start date  of PI and its possible impact on the quality of CCM findings, the Bureau  considered whether to start PI 6 weeks earlier than planned. In June 2009,  the Bureau decided to keep the originally scheduled start on August 14,  2010. Bureau memorandums and officials justified the decision largely  because of concern that it was too late in the planning process to make a  change in the complex CCM schedule. The memorandums cited gaps in  knowledge about the impact of timing on recall bias, presented research  with differing conclusions about the extent of contamination in prior  census tests, and justified the recommendation to not change the start  date by the operational challenges faced to make the change.
Bureau officials have also explained that the goal of using coverage  measurement in 2000 to possibly adjust the census-created time pressures  in 2000 that forced an early PI, and because such time pressures do not  exist for PI in 2010, it is scheduled to begin more than 4 months after  Census Day.
By comparison, during the 2000 Census, the Bureau launched PI in April  2000 and had completed about 99 percent of its data collection by the end  of the first week of August 2000, a week earlier than the scheduled 2010 PI  start date. An extensive 2000 Census evaluation found no evidence of  contamination bias caused by the earlier start of PI in 2000. Related  Bureau research since then has also found no significant evidence of  contamination bias during census tests, although one test found that  census results could be affected. Yet Bureau officials remained concerned  about the possibility, since the CCM questions are similar to follow-up  questions used in one of the 2010 census follow-up operations.  Furthermore, parts of this census operation are new in 2010, and end later  than similar operations did in 2000.
Moving forward, additional research on the trade-offs between recall bias  and contamination errors could help the Bureau more fully understand the  implications of choosing various start times for PI on the resulting  estimates of coverage error and better determine the optimal timing of PI  in future censuses. Currently, the Bureau has a telephone-based study  planned in order to measure recall errors, which could provide additional  information about when recall errors are more likely to occur. However,  this study is limited to certain types of recall error, and the Bureau does  not have an evaluation planned to measure possible contamination  between the new, much later, parts of census follow-up and CCM data  collection or to assess the trade-offs between the biases from starting  earlier compared to starting later. Such additional study after the 2010  Census could provide the Bureau better information about the trade-offs  in data quality from potential contamination and recall biases and provide  a better basis for determining the optimal scheduling of coverage  measurement operations.
section_title: Conclusions
paragraphs: Assessing the accuracy of the census is an essential step in improving  current and future censuses. The Bureau has made progress on designing  and planning for its CCM program and continues activity to complete the  plan. Additional actions in three CCM planning areas may further improve  CCM or its usefulness to the 2020 Census.
Specifically, the Bureau has stated the importance of using 2010 evaluation  data such as CCM’s for 2020 Census design, but has not yet taken steps to  link CCM data to an improved 2020 design. If the Bureau is to best achieve  its goal of using CCM to help improve the 2020 Census, it will need to  integrate planning for any follow-up work on CCM results or data with the  other early planning already underway for Census 2020.
Second, the Bureau has many different processes that come together in  the conduct of a decennial census, and archived data on those processes  could provide useful information to researchers trying to figure out what  worked well and what did not. The Master Trace Project can help  researchers link CCM results and data to potential design changes for  Census 2020. Determining which data need to be linked or archived to  enable future linkage within the project can help prevent gaps in 2010 data  that might hinder the project’s ability to help identify improvements for  the 2020 Census.
Third, the timing of CCM’s primary data collection operation—Person  Interviewing—involves trade-offs between reducing contamination bias  and reducing recall error that the Bureau did not have conclusive  information on. Since 2010 Person Interviewing is starting 1 week after a  similar operation ended in 2000, the chance of introducing recall bias  errors into CCM data is higher in 2010 than it was in 2000. Although the  Bureau has a study planned to measure some recall errors, there is no  study planned to measure contamination between the new parts of census  follow-up—which use questions similar to those asked by CCM and finish  much later than follow-up did in 2000—and CCM or to assess the trade- offs between the two types of biases in timing decisions. Targeted  research after the 2010 Census on the relationship between the timing of  data collection and the trade-offs between these types of errors before the  2020 Census and its coverage measurement efforts could help the Bureau  better determine the optimal timing of future data collection operations.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Commerce require the Director of  the U.S. Census Bureau to take the following three actions to improve the  usefulness of CCM for 2020:  To help the Bureau achieve its goal of using CCM to improve the 2020  Census, better document links between the 2010 CCM program and 2020  Census planning, integrating the goal of using the CCM program to  improve Census 2020, such as with CCM results and data, into those  broader plans for 2020.
To ensure that Bureau datasets from the 2010 Census can be used with  other Bureau datasets to support research that could improve the census  and CCM, complete the Master Trace Project’s assessment of how key  census and CCM data systems are, or can be, linked to each other; identify  any potential data gaps; and identify other related steps for future action.
To help the Bureau better determine the optimal timing of future coverage  measurement data collection, fully assess the trade-offs between starting  the data collection earlier, with the possibility of introducing  contamination errors, and starting later, with the possibility of introducing  recall errors.
section_title: Agency Comments and Our Evaluation
paragraphs: The Secretary of Commerce provided written comments on a draft of this  report on April 5, 2010. The comments are reprinted in appendix I.  Commerce generally agreed with the overall findings and  recommendations and appreciated our efforts in helping the Census  Bureau develop a successful evaluation plan for the 2020 Census.  Commerce also provided additional information and comments on certain  statements and conclusions in the report.
With respect to our second recommendation to complete the Master Trace  Project’s assessment of linking key census and CCM data systems, to  identify any potential data gaps, and to identify other related steps for  future action, Commerce maintained that it would be taking action to  preserve adequate documentation and maximize the amount of data  retained from each major decennial system. We commend the Bureau for  committing to these steps and encourage its follow-through on them and  its identification of remaining data gaps and additional steps needed.
With respect to our third recommendation to fully assess the trade-offs  between two types of error related to starting CCM data collection either  earlier or later relative to Census Day, Commerce responded that (1) it is  too late to create a new study for 2010 Census; (2) it considers a Bureau  contamination study from 2000 to be definitive; and (3) it has recently  developed a study on recall bias to try to measure some of the effects of  scheduling CCM data collection at various periods of time following the  census enumeration. We agree that it is too late to attempt any additional  unplanned data collection during the 2010 Census, and we revised our  discussion to clarify our intent that the recommended research be  conducted after the 2010 Census. We also recognize the thoroughness of  the 2000 contamination study the Bureau cites, commend the Bureau on  undertaking additional study of recall bias, and look forward to reviewing  its study plans when they are available. However, we recommended  research comparing trade-offs between the two types of errors at a variety  of start dates for CCM data collection—something the 2000 study did not  discuss and something it is unclear that a study of only recall bias will  achieve. Furthermore as we discussed in our draft report, the Bureau  expressed concerns over possible contamination between CCM and new  parts of census follow-up in 2010—parts that were introduced after the  2000 study and that were not included in the scope of the 2000 study. We  clarified our discussion of this in the report to better focus on the need for  research that relates the trade-offs between the two types of error at  different timing of data collection.
Commerce provided additional information that in response to advice  from various advisory panels and after additional research, it would soon  make public its proposed geographic levels for CCM estimates. We  reflected this decision in table 1 of our report.
Finally, Commerce provided additional information about its plans to  produce highly technical documentation of the results of CCM estimation  including modeling, missing data, and errors in the estimates in a series of  memorandums as it did for Census 2000. We reflected this decision in  table 1 of this report.
We are sending copies of this report to the Secretary of Commerce, the  Director of the U.S. Census Bureau, and interested congressional  committees. The report also is available at no charge on GAO’s Web site at  http://www.gao.gov.
If you have any questions about this report please contact me at (202) 512- 2757 or goldenkoffr@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page   of this report. Key contributors to this report were Jeff Dawson, Dewi  Djunaidy, Ron Fecso (Chief Statistician), Andrea Levine, Ty Mitchell,  Melanie Papasian, and Tamara F. Stenzel.
section_title: Appendix I: Comments from the Department of Commerce