Data Quality: Improvements to Count Correction Efforts Could Produce More Accurate Census Data
Released: 2005-07-20T00:00:00

Letter

Background

The Bureau launched the CQR program on June 30, 2001, as the last in a series of quality assurance initiatives aimed at improving the accuracy of 2000 Census data (see fig. 1). Specifically, the CQR program provided a mechanism for state, local, and tribal governments to have the Bureau correct errors in certain types of census data. The Bureau referred to these challenges as “external cases.” Bureau personnel could also initiate reviews of suspected count errors, independent of these challenges, for further review. These were known as “internal cases.” Many of the internal cases were unresolved issues inherited from Full Count Review. Indeed, when the Full Count Review program began, the Bureau planned to fold unresolved issues from that program into CQR. The Bureau accepted no new submissions after the program officially ended on September 30, 2003, although it continued to review challenges submitted before the deadline and completed the final revisions in the summer of 2004.

Three types of corrections were permissible under the CQR program: (1) boundary corrections, where a jurisdictional boundary of a functioning governmental unit was in the wrong location; (2) geocoding corrections, where the Bureau placed the living quarters and their associated population in the wrong location; and (3) coverage corrections, where the Bureau properly enumerated specific living quarters and their corresponding population during the census but incorrectly added or deleted the information during data processing. Bureau officials were to research cases using existing Bureau data gathered during the 2000 Census; they could not conduct any new fieldwork to resolve count questions. The Bureau required governmental entities to accompany their challenges with specific documentation before it would investigate their claims.

Importantly, under the design of CQR, if a governmental unit had evidence that the Bureau missed housing units or group quarters that existed on Census Day 2000 (April 1), but the Bureau’s records indicated that all of the Bureau’s boundary information, geocoding, and processing were properly implemented, the Bureau would not change the data. Rather, the Bureau was to address this as part of the planning process for the 2010 Census.

If the CQR program corrected the population or housing unit counts of a particular entity, the Bureau was to issue revised, official figures for that jurisdiction. The governmental unit could then use the updated numbers for future programs requiring 2000 Census data. CQR corrections were also used to modify annual post-censal estimates beginning December 2002 and were publicized on the Bureau’s Census 2000 and American FactFinder Web sites (www.census.gov and www.factfinder.census.gov, respectively), as part of the 2000 Census notes and errata. However, CQR was not designed or publicized as a mechanism to correct the census results for purposes of apportionment and redistricting.

In compliance with legal requirements, the Bureau produced apportionment data by December 31, 2000, and redistricting data by April 1, 2001 (this information is known collectively as public law data). Although the law does not require that states use census data to redraw the boundaries of congressional districts, most states have always done so. Nothing would preclude the states from using the corrected data for redistricting. The general perception of the impartiality of the Bureau and the great cost and administrative effort required to take a census have been strong arguments in favor of using the Bureau’s data.

Scope and Methodology

As agreed with your offices, we assessed whether the program was consistently implemented across the country, paying particular attention to the extent to which the Bureau reviewed the census data for errors that could have been caused by broader, more systemic problems, such as shortcomings with a particular census-taking procedure. We also evaluated how well the Bureau transitioned from an earlier quality assurance program used in the 2000 Census, Full Count Review.

To assess the implementation of the CQR program, we obtained a headquarters perspective by reviewing program documents and case files at the Bureau’s offices in Suitland, Maryland, as well as program results reported on the Bureau’s Web site. As part of this assessment, we reviewed the program’s internal controls, especially those controls related to ensuring data quality. We also interviewed Bureau officials responsible for administering the program.

To determine how CQR was implemented in the field, we visited 4 of the Bureau’s 12 regional offices—Charlotte, North Carolina; Denver, Colorado; Kansas City, Missouri; and Los Angeles, California (see fig. 2).

We selected these regions because they included the six states and 10 governmental units within those states where the largest CQR count revisions occurred. We supplemented these cases by selecting an additional seven states and 61 places within the four regions for further examination. The 61 localities were selected because they represented the full spectrum of CQR cases and were geographically diverse.

At each of the four regions, we reviewed regional case file information and interviewed Bureau personnel responsible for implementing CQR, such as program managers and geographers. We also made a site visit to at least one type of facility—including prisons, apartment buildings, and dormitories—in each region to understand firsthand the nature of the errors and the corrections made. To augment these four regional visits and obtain a more complete picture of how CQR was implemented, we used a structured telephone interview to elicit information from program officials at the Bureau’s eight remaining regional offices that we did not visit in person.

To determine the extent to which the Bureau reviewed census data for systemic errors, and its procedures for folding unresolved cases from the Full Count Review program into CQR, we examined program manuals, memoranda, and other documents, and interviewed officials in the Bureau’s headquarters and all of its regional offices. As part of this effort, we also analyzed the CQR case-tracking data in an attempt to determine the number of unresolved Full Count Review cases that were rolled into the CQR program. However, we were unable to do this because the tracking system did not contain information on which CQR cases originated as full count issues.

We requested comments on a draft of this report from the Secretary of Commerce. On May 20, 2005, we received the Acting Deputy Secretary’s written comments and have reprinted them in appendix III; we address them in the Agency Comments and Our Evaluation section of this report.

CQR Program Corrected Numerous Data Errors, but More Consistent Implementation and Other Improvements Are Needed

Overall, the CQR program corrected data affecting over 1,180 of the nation’s more than 39,000 governmental units. The revisions impacted a range of housing types including private homes with only a handful of residents, to college dormitories and prison cell blocks with populations in the thousands. At the same time, however, we identified several shortcomings with the CQR program, including inconsistent handling of internal cases by the Bureau’s regional offices and inaccurate data being posted to the Bureau’s public Web site. Moreover, while CQR found the counting of group quarters in their correct location—a problem known as geocoding error—to be particularly challenging, the Bureau did not perform a nationwide review of these known trouble spots, and thus missed an opportunity to improve the accuracy of the data for these dwellings.

CQR Program Corrected Errors in Hundreds of Governmental Units

Nationwide, the CQR program corrected count errors involving governmental units in 47 states, Puerto Rico, and the District of Columbia (see fig. 3). Three states—Maine, New Hampshire, and Rhode Island—had no CQR corrections.

The corrections affected over 1,180 governmental units in the United States. Although this is a small percentage of the nation’s more than 39,000 governmental units, the impact of those changes on local governments was, in some cases, substantial, and could have implications for federal assistance and state funding programs that use census numbers in their allocation formulas, as well as other applications of census data.

For example, officials in one Kentucky county challenged the geocoding of a housing unit located near new precinct and congressional district boundaries. They told the Bureau that the new boundaries split the county, and they were concerned that the geocoding error would affect where the housing unit’s few occupants registered to vote. Because the housing unit was improperly geocoded, the Bureau corrected the data.

With respect to fiscal effects, the Controller of the State of California uses population figures as the basis for refunding a portion of state taxpayer fees—including automobile licensing fees—to cities and counties. Because of an error in the 2000 Census, Soledad, California, officials estimated it lost more than $140,000 in state refunds when over 11,000 residents were incorrectly counted in two nearby cities’ populations, according to city and state officials. Although CQR eventually corrected the error, Soledad did not recover the funds that went to the other cities.

Other examples of large CQR corrections include the following (See app. I for a complete list of state-level population changes): North Carolina’s population count was reduced by 2,828 people, largely because the Bureau had to delete duplicate data on almost 2,700 students in 26 dormitories (see fig. 4) at the University of North Carolina (UNC) at Chapel Hill. The erroneous enumerations occurred, in large part, because of mistakes that occurred in various preparatory activities leading up to the 2000 Census. (See app. II for a more detailed discussion of this incident.)

The population count of Morehead, Kentucky, increased by more than 1,600 when CQR found that a large number of students from Morehead State University’s dormitories were erroneously excluded from the city’s population. During the 2000 Census, the Bureau had incorrectly identified the dormitories as being outside city limits and in an unincorporated area of Rowan County.

The population count of Cameron, Missouri, was off by nearly 1,500 people when the Bureau found that the prison population of the state’s Crossroads Correctional Center was inadvertently omitted from the town’s headcount (see fig. 5). The correction to the town’s population accounted for the entire 1,472 person increase in Missouri’s total population under the CQR program.

The population of the city of Waseca, Minnesota, increased by more than 1,100. The 2000 Census had mistakenly included the prison population for the Waseca Federal Correctional Institute in two surrounding townships in Waseca County. The CQR program resulted in the population being shifted to the city.

The population of Colorado increased by more than 750 in large part because a processing error in counting housing units in Grand Junction initially excluded almost 700 people from the city’s population total.

The population of Denver and Arapahoe Counties in Colorado shifted by more than 900 because the Bureau had incorrectly assigned the location of two apartment complexes. As a result of CQR, the apartment complexes were incorrectly identified and counted as being in Denver but under CQR were later found to be in adjoining Arapahoe County.

CQR Program Was Unevenly Executed

The Bureau’s 12 regional offices did not always adhere to the same set of procedures when developing internal cases, and this, in turn, produced uneven results. Importantly, the procedures used to execute public programs need to be well documented, transparent, and consistently applied in order to ensure fairness, accountability, defensible decisions, and reliable outcomes. To do otherwise could raise equity questions.

One variation in the way internal cases were handled was evident at the Bureau’s Los Angeles Regional Office, which appeared to be the only region to do comprehensive methodical research to actively identify possible count errors beyond those that were submitted by governmental entities. According to the office’s senior geographer, the geography staff developed a structured approach to systematically examine census data from all the prisons and colleges within the office’s jurisdiction, because the data on both types of group quarters were known to be problematic. He added that the more problems they found, the more they were motivated to keep digging. The geographer noted that the in-depth review was possible because the Los Angeles region covers only the southern half of California and the state of Hawaii, and thus has fewer governmental units compared to the Bureau’s other regional offices.

Had it not been for the Los Angeles region’s self-initiated and systematic review, certain data errors would have gone uncorrected because they were not identified by the affected jurisdiction. For example, regional staff found instances where college dormitories were counted in the wrong geographic location, which, in turn, affected the population counts of their surrounding locales. Such was the case with California State University Monterey Bay (CSUMB) and the University of California at Santa Barbara (UCSB). As a result, the Bureau transferred a population of more than 1,400 between the towns in which they were initially counted and in which CSUMB is located and shifted a population of more than 2,700 between the city and the unincorporated area of the county in which UCSB is located.

The Bureau’s Charlotte Region, while also more active than the Bureau’s other offices that generated internal CQR cases, seemed to be less methodical and comprehensive than Los Angeles in its approach. For example, although Charlotte geographers detected the duplicate count of almost 2,700 students at the University of North Carolina mentioned in the previous section, their research was not the result of any systematic review. Rather, it came about largely because of the curiosity of key employees in the Charlotte office, who were also alumni of the school. (See app. II for more details on the circumstances surrounding the duplicate count.)

Better Guidance and Training Could Improve Implementation

Vague guidance was one reason for the disparate handling of internal cases. For example, the CQR procedural manual indicates that the Bureau’s 12 regional offices were to research CQR cases “as appropriate.” However, the manual did not define whether this meant that the regional offices should initiate their own data reviews or merely verify CQR cases submitted by governmental units. More generally, numerous geographers we interviewed—the primary users of the manual—did not find it user- friendly, noting it was confusing, complex, or impractical. For example, a geographer pointed out that the manual did not have an index covering the eight chapters and 26 appendixes, which would have helped them more quickly find information and procedures. In addition, we found that the manual and other documents did not discuss how program staff were to address Full Count Review issues.

The Bureau’s training was also problematic and likely added to the implementation disparities. For example, geographers in five regions told us that during training they were instructed or given the impression they were not to generate additional internal cases beyond the small number of count errors that had already been identified at the beginning of the program. Also, geographers in two of these regions told us they were specifically told not to investigate any count errors they found that were outside the scope of the cases that governmental units submitted. Conversely, geographers in the other seven regions said they were not restricted in any way.

There were other training issues as well. According to the Bureau’s draft CQR program assessment, the final version of which is pending, some training materials were developed at the last minute and were never finalized, and training began before needed software was in place at all the research divisions. Proper training was particularly important because, as the draft evaluation notes, staff assigned to the CQR program had census experience but limited geographic and field operations knowledge. Others had limited or no Census 2000 software program experience.

Internal Control and Quality Assurance Problems Led the Bureau to Report Erroneous Data

Federal internal control standards call on agencies to employ edit checks and other procedures to ensure their information processing systems produce accurate and complete data. However, the Bureau’s internal controls in reporting CQR results were insufficient in that we found, after a partial review, a number of instances where the Bureau disseminated inaccurate data on its Web site where it maintains an errata report that lists the CQR revisions to the 2000 Census data.

Specifically, after comparing data from the errata report to the certified numbers in the CQR case files, we found errors with the reporting of CQR housing, group quarters, and population counts. Importantly, our review found that the revised, certified figures the Bureau provided to affected jurisdictions were correct. This is significant because the affected jurisdictions could use these updated numbers for revenue sharing and other programs that require census data. However, users who obtain information from the Bureau’s errata report—these can be people in academia, government, and the private sector—would not have the most up-to-date information. For example: The original state-level total housing unit count for Delaware mistakenly excluded 30,000 housing units.

The revised housing unit count for the Minnehaha County portion of Sioux Falls, South Dakota, was underreported by almost 47,000 housing units.

The Burlington County, New Jersey, revised total housing count mistakenly excluded about 145,000 units.

The errata report excluded 8 of the 12 American Indian and Alaska Native Areas that had revisions to their housing, group quarter, or total population counts.

The Bureau later corrected these errors after we brought them to its attention.

Although the Bureau had controls in place to ensure accurate research and reporting, the problems we found point to the need for the Bureau to tighten its procedures to prevent mistakes from slipping into its data products. For example, the CQR manual included some quality control steps, such as having headquarters divisions review field research and results. Further, field geographers told us they consulted one another about questions or procedures and checked each other’s work, and Bureau program managers had procedures in place to review final revisions and certify them as correct.

Documents in the CQR case files we reviewed substantiate these practices. Also, managers told us they randomly checked data entered into the files that are the basis for the revisions posted to the errata report. Still, the number of errors we found after only a partial review of the errata files raises questions about how effectively the Bureau implemented the quality assurance procedures, as well as the quality of the data we did not review. It also underscores the importance of adequate control activities to prevent these problems from recurring.

Web-based Errata Report Should Be More User- Friendly

Data users may have encountered problems trying to access certain information from the Web-based errata report. Additionally, because there was no link or cross-walk between some of the initial population data the Bureau released and the CQR revisions, users may have been unaware that some of the original numbers had been revised.

As shown in figure 6, the Bureau’s Web-based errata report presented revised data for states, American Indian and Alaska Native Areas, and other jurisdictions at the state or similar geographic level. Although the table had embedded links that were supposed to take users to revisions at lower levels of geography, these links did not always work and produced error messages instead. We found that unless the users’ software and Internet access paralleled the Bureau’s, users could not access the more detailed data using the embedded links. Bureau staff involved with posting the data to the Web site stated in the summer of 2004 that they were aware of the problem, but as late as March 2005, the problem had yet to be fixed.

At the same time, the CQR revisions may not be evident to users who access certain data from the 2000 Census data posted elsewhere on the Bureau’s Web site. This is because these sites lack notes or flags informing users that updated figures are available in the census errata report. For example, the Bureau’s American FactFinder Web site—the Bureau’s primary electronic source of census data—does not inform users that revised data on group quarter counts, including the number of correctional institutions, as well as data on their associated populations, exist as part of the Bureau’s notes and errata report.

American FactFinder presents data known as Summary File 1 (SF-1), which is the first data set the Bureau produces from the census, and is used for purposes of apportionment and redistricting. While the SF-1 data remain unchanged, other data users may find the revised numbers better suited to their needs. Figure 7 illustrates the existence of two sets of numbers without any explanation. Summary File data from American FactFinder show the population for Soledad, California, as 11,263. However, the Bureau’s errata report, which reflects the CQR revisions, shows the Soledad population at 23,015. Because American FactFinder lacks notes or links that tell users about the revised data, users might inadvertently obtain erroneous information.

According to Bureau officials, while they thought about adding notes directing users to the CQR revisions, they decided against it because they thought it would confuse more people than it would help. They reasoned that knowledgeable users, such as county planners and state data center staff, are likely aware of the CQR information and would therefore not need to be informed about the existence of the notes and errata Web site.

CQR Errors Highlight Problems with 2000 Census Address List Development Procedures

The errors uncovered by the CQR program highlight some of the limitations in the way in which the Bureau builds its address list for the decennial census, particularly in the procedures used to identify and locate group quarters. For the 2000 Census, the Bureau had three operations that were primarily designed to locate these types of dwellings. However, given the number of prisons and other group quarters geocoded to the wrong location, refinements are needed. Moreover, the Bureau’s draft CQR program assessment found that the Bureau’s Master Address File had numerous data entry errors including incorrect spellings, geocoding, and zip codes.

To its credit, the Bureau is planning several improvements for 2010, including integrating its housing unit and group quarter address lists. This could help prevent the type of duplicate counting that occurred at UNC when the same dormitories appeared on both lists. Likewise, the Bureau’s planned use of a satellite-based navigational system could help census workers more precisely locate street addresses.

Better Strategic Planning and Other Actions Could Improve Future Count Correction Efforts

The CQR program was preceded by a quality assurance program called Full Count Review, which ran from June 2000 through March 2001 and, like CQR, was designed to find problems with the census data. However, although the Bureau planned to fold unresolved full count issues into CQR, many full count issues were rejected from CQR because the latter program had more stringent documentation requirements. As a result, the Bureau was unable to resolve hundreds of additional data issues.

Numerous Unresolved Full Count Issues Could Not Be Folded into CQR as Planned

Under the Full Count Review program, analysts were to identify data discrepancies to clear census data files and products for subsequent processing or public release. Analysts did so by checking the data for their overall reasonableness, as well as for their consistency with historical and demographic census data and other census data products. The types of issues flagged during Full Count Review included potential discrepancies involving the counts and/or locations of group quarters, housing units, and individual households, among others.

As we noted in our July 2002 report, Full Count Review identified 4,809 potential data anomalies. However, of these, just five were corrected prior to the December 31, 2000, release of apportionment data and the April 1, 2001, release of redistricting data. The corrections included a military base, a federal medical center, and multiple facilities at two prisons and a college that were counted in the wrong locations. That the public law data were released with numerous data issues of unknown validity, magnitude, and impact, gave us cause for concern, and we noted that the Bureau missed an opportunity to verify and possibly improve the quality of the information.

When the Full Count Review program began, the Bureau planned to fold unresolved issues from that program into CQR. Indeed, according to a June 2000 memo on CQR policy agreements, “a by-product of [Full Count Review] is documentation of unresolved issues for potential use in CQR.” However, because the CQR program had more rigorous documentation requirements before it would accept a case compared to Full Count Review, a number of issues that were deemed suitable for Full Count Review but were unresolved, were rejected from CQR.

Of the 4,804 issues remaining after Full Count Review, 2,810 issues (58 percent), were not referred to CQR. Of the 1,994 issues (42 percent) that were referred to CQR, 537 were actually accepted by the program. The remaining 1,457 issues referred to CQR did not meet the Bureau’s CQR documentation requirements and, consequently, the Bureau took no further action on them.

The Full Count training materials we examined as part of our 2002 review did not provide any specific guidance on the type of evidence analysts needed to support data issues. Rather, the materials instructed analysts to supply as much supporting information as necessary. In contrast, the CQR program had more rigorous documentation requirements. Guidance available on the Bureau’s Web site required governmental units to supply maps and other evidence specific to the type of correction they were requesting, or the Bureau would not investigate their submissions.

Simply put, Full Count Review identified hundreds of data issues but lacked the time to investigate the vast majority of them. Then, when the remaining cases were referred to CQR, most were rejected because they could not meet CQR’s higher evidentiary bar.

A Mechanism for Correcting Public Law Data Will Be Critical for Future Enumerations

The Bureau lacked a program specifically designed to correct individual count errors contained in the apportionment and redistricting data. Because these numbers were later found to be flawed for some jurisdictions, as the Bureau proceeds with its plans for the 2010 Census, it will be important for it to explore options for reviewing and correcting this essential information before it is released.

Precision is critical because, in some cases, small differences in population totals could potentially impact apportionment and/or redistricting decisions. For example, according to an analysis by the Congressional Research Service, under the formula used to apportion seats in the U.S. House of Representatives, had Utah’s population count been 855 persons higher, it would have gained an additional congressional seat and North Carolina would have lost a seat. However, had the duplicate UNC count and other errors detected by the CQR program as of September 30, 2003, been uncovered prior to the release of the public law data, the already narrow margin determining whether Utah gained a House seat would have dropped to 87 persons. Although in this particular instance there would not have been a change in congressional apportionment, it illustrates how the allocation of House seats can be determined by small differences in population counts.

Other Aspects of CQR Could Have Been Better Planned

Better planning could have improved the CQR program in other ways. For example, the Bureau’s draft evaluation of CQR found, among other issues, that the three teams working on the planning and development phases of CQR should have tested implementation plans earlier in the process, and training materials were not based on the Bureau’s experience in conducting the 1990 CQR program. Also, there was no mechanism to prioritize cases based on the magnitude of the error. As a result, regional offices wound up expending considerable resources on CQR cases that only affected a handful of dwellings.

The draft evaluation also found that the two software applications the Bureau chose to administer and track CQR cases did not appear to be up to the task. Lost cases and documentation, poor integration with other applications, and the inability to produce reports were among the issues the evaluation cited.

More generally, the integration and coordination issues that affected CQR are not unique to that program; to the contrary, our past reports have found that other components of the 2000 Census were not well planned, which unnecessarily increased the cost and risk of the entire enumeration. The need for better strategic planning has been a consistent theme in many of our past recommendations to improve the Bureau’s approach to counting the nation’s population and represents a significant management challenge that the Bureau will need to address as it looks toward 2010.

The Bureau Is Making Count Correction Plans for 2010

The Bureau is beginning to develop plans for Full Count Review and CQR for the 2010 Census. As it does so, it will be important for it to develop an initiative or consolidated program that corrects both systemic and individual issues, and does so prior to the release of apportionment and redistricting data. Granted, this effort will be no simple task given the relatively short time between the closure of the local census offices and the need to release the public law data within the legally required time frames.

Still, there are steps the Bureau can explore to methodically check the data for nationwide systemic errors, obtain local input, and investigate any discrepancies, and do so in an expeditious manner. One approach might be to consolidate and leverage CQR, Full Count Review, and certain other Bureau programs.

Indeed, under the Full Count Review program, the Bureau obtained local input by contracting out some of the work to members of the Federal-State Cooperative Program for Population Estimates (FSCPE), an organization composed of state demographers that has worked with the Bureau since 1973 to ensure accurate population counts. The Bureau worked with FSCPE, in part, because it lacked sufficient staff to complete the review on its own, but also because the Bureau believed that the members’ knowledge of the demographic characteristics of their states could help the Bureau examine data files and products, including public law data. FSCPE members reviewed data for 39 states and Puerto Rico; Bureau employees reviewed data for the remaining states and the District of Columbia without FSCPE representation in Full Count Review. Both sets of analysts checked the data for their overall reasonableness, as well as for their consistency with historical and demographic data, and other census data products.

Bureau staff from its regional offices reviewed the data as well. They focused on identifying inconsistent demographic characteristics and did not necessarily concentrate on any one particular state or locality. Thus, the Bureau obtained local input that focused on individual states and smaller jurisdictions, and also performed its own, broader review.

Verifying any data discrepancies could be accomplished by beginning the count correction effort as local census offices complete nonresponse follow-up, when enumerators are still available to investigate issues. In fact, the Bureau is already planning to do this to some degree in 2010 under another operation called Coverage Improvement Follow-up (CIFU), where the Bureau is to call or visit housing units that have been designated as vacant or nonexistent but not confirmed as such by a second source. In the 2000 Census, CIFU began June 26, 2000, and ended on August 23, 2000. During that time, enumerators contacted 8.9 million housing units and counted 5.3 million people, according to the Bureau.

The Bureau could explore adding the count correction workload to enumerators’ CIFU assignments, which would enable the agency to reconcile possible data errors, as well as add any housing units and group quarters the Bureau missed during the initial enumeration (As noted in the background section, CQR could not add any residences that existed on Census Day but the Bureau had failed to count.).

Further, the Bureau could help automate the count correction process by using computers to flag any data that exceed any predetermined tolerances. The Bureau could also develop a system to prioritize count correction issues to help manage its verification workload.

Importantly, to the extent the Bureau reviews and corrects census counts prior to the release of the public law data, the Bureau might not need separate Full Count Review and CQR programs; a consolidated effort might be more cost effective. At a minimum, to the extent a separate CQR program is needed, it may not have to be as large or last as long because presumably the earlier program would have caught the bulk of the problems.

Regardless, given the possibility that similar data errors might again occur during the 2010 Census, exploring options for resolving them prior to the release of public law data would be a sound investment. Reapportionment and redistricting data would be more accurate; the Bureau’s credibility would be enhanced; and the need for a large-scale count correction program along the lines of CQR could be reduced or eliminated.

Conclusions

The CQR program played an important role in improving the quality of data from the 2000 Census. Although the net changes in housing and population counts from the program were small on a national scale, in a number of instances, they were substantial at the local level, and could affect various revenue sharing formulas and other programs that use decennial census data.

Because the program functioned as a safety net—a final opportunity to catch and correct mistakes that occurred along the chain of events that led to, and extended beyond Census Day 2000—the results shed light on the performance and limitations of certain upstream census operations, and areas where the Bureau should focus its efforts as its plans unfold for 2010. In this regard, the following is clear: although the Bureau puts forth tremendous effort to ensure a complete and accurate census, its numerous procedures and quality assurance operations will be challenged to stay ahead of the increasing difficulties associated with enumerating a population that is growing larger, more diverse, and increasingly hard to locate and count.

The timing of any count correction effort will also be critical. Indeed, we are concerned that key decisions using data from the 2000 Census employed figures that, for a number of jurisdictions, were later found to be flawed. As a result, it will be important for the Bureau to consider developing a count correction initiative that can complete its work in time to correct the public law data before that information is released.

Moreover, beyond the inherent demographic obstacles to a successful census, the results of our CQR review echo several of our past reports on other aspects of the census, which note that some of the Bureau’s difficulties stem from a lack of adequate strategic planning and other management challenges. Ultimately, the success of the 2010 Census will hinge on the extent to which senior Bureau leadership resolves these challenges.

With this in mind, resolute action is needed across three fronts. First, it will be important for the Bureau to ensure, via thorough field testing, that its planned changes to its address list development procedures help resolve the geocoding and other operational problems revealed by CQR. Second, it will be important for the Bureau to improve its count correction efforts by designing a program that can systematically and consistently review the public law data and make any corrections prior to the release of those figures. Third, it will be important for the Bureau to address persistent strategic planning challenges.

Recommendations for Executive Action

To help ensure the nation has the best possible data for purposes of apportionment, redistricting, and other uses of census data, we recommend that the Secretary of Commerce direct the Bureau to improve its count correction efforts for the 2010 Census by taking such actions as: 1. Thoroughly testing improvements to the Bureau’s group quarters and other address list development activities to help ensure the Bureau has resolved geocoding and other problems with its master address file. 2. Consolidating Full Count Review and CQR into a single program that systematically reviews and corrects any errors in the public law data prior to their release. 3. Expediting count correction efforts by initiating data reviews toward the end of nonresponse follow-up, when the Bureau starts getting complete data for geographic entities, and enumerators are available to help investigate any discrepancies. As part of this effort, the Bureau should consider using computers to systematically search for possible errors nationwide by checking data at the appropriate level of geography to ensure population, housing unit, and group quarter counts, as well as demographic characteristics, appear reasonable and are consistent with population estimates. Those areas that are outside of predetermined tolerances should be flagged for further review. The Bureau should also pay special attention to ensure group quarters are properly geocoded and counted. 4. Prioritizing the investigation of errors based on the magnitude of the suspected error or similar triaging formula. 5. Ensuring that instructions on the Bureau’s Web site make it clear that updated information exists and that users can readily access this information.

6.

Improving the Bureau’s quality assurance procedures to help ensure there are no mistakes in the data the Bureau posts on its Web site. 7. Enhancing the training and guidance provided to regional offices to help ensure they share the same understanding of their roles and responsibilities and will implement the program consistently. 8. Addressing persistent strategic management challenges, in part, through early testing to help ensure information systems, training, and other activities are fully integrated.

Agency Comments and Our Evaluation

The Acting Deputy Secretary of Commerce provided written comments on a draft of this report on May 20, 2005, which are reprinted in appendix III. Commerce stated that “the report provides a good overview of program results and makes several useful observations and recommendations,” and specifically agreed with our finding that the process for conducting internal reviews was not consistently implemented. More generally, however, Commerce believes the shortcomings we describe reflect “a fundamental misunderstanding of the goals of the CQR program,” and noted that our observations and recommendations indicate we believe that CQR should have been designed to correct the public law data before they were released during the 2000 Census.

Our concern over the CQR program centers on the way it was implemented in 2001, rather than the fact that the Bureau did not design the program to correct the apportionment and redistricting numbers. We agree with Commerce that this was not the intent of CQR and, as Commerce notes, we acknowledge this in our report. At the same time, based on the lessons learned from the 2000 Census, enumeration errors are almost inevitable. Thus, our recommendations focus on the future, and specifically, the importance of developing mechanisms for the 2010 Census to review and correct errors in the public law data to the greatest extent possible before they are released. We have clarified the report to better reflect GAO’s position.

Commerce specifically addressed two of our eight recommendations, disagreeing with both of them. With respect to our recommendation to consolidate Full Count Review and CQR into a single program for the 2010 Census, Commerce noted that preliminary counts at the census tract or block level are needed to conduct an effective CQR program, and that information is not available until close to the deadline for releasing the apportionment data. Commerce maintains there would be little opportunity for local entities to review the counts and document potential problems and even less time for the Bureau to conduct the necessary research and field work.

Our report recognizes that it would be a challenge for the Bureau to review and correct census figures and still release the public law data by the legally required time frames. Still, as we note in our report, we believe the Bureau could expedite the process by taking such steps as (1) using computers to check census data for their overall reasonableness and flagging areas that exceed predetermined tolerances; (2) focusing on known trouble spots such as group quarters; and (3) beginning the review process earlier, such as, when local census offices complete their nonresponse follow-up efforts.

Moreover, as we state in the report, during the 2000 Census, the Bureau already had programs in place that obtained local input on the census numbers before the release of the public law data (Full Count Review), and conducted extensive field operations to investigate certain discrepancies (Coverage Improvement Follow-up). We believe that it will be important for the Bureau to not simply replicate these programs for the 2010 Census or make incremental improvements, but to see whether these programs could be better leveraged and be more strategically employed to improve the accuracy of the apportionment and redistricting data.

The other recommendation that Commerce specifically addressed was our call for the Bureau in 2010 to prioritize the investigation of errors based on the magnitude of the suspected problem. Commerce maintains that the Bureau’s policy in 2000 was to handle cases in the order they were received from local jurisdictions, and asserts this was a fair and reasonable practice. While this practice is not unreasonable, we continue to believe that it would be more cost-effective for the Bureau to give priority to those cases where it could achieve a greater return on its investment in resources (especially given our findings involving group quarters such as prisons and college dormitories that affected relatively large population clusters). Our recommendation echoes the Bureau’s draft evaluation of the CQR program, which noted that regional offices expended considerable resources on CQR cases that affected only a handful of dwellings. Moreover, as we state in our report, prioritizing the Bureau’s workload could help expedite the count correction process.

Commerce’s comments also included some technical corrections and suggestions where greater clarity was needed. We revised the report as appropriate.

We will send copies of this report to the Chairman of the House Committee on Government Reform, the Secretary of Commerce, and the Director of the U.S. Census Bureau. Copies will be made available to others on request. This report will also be available at no charge on GAO’s home page at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-6806 or williamso@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Change in State Populations As a Result of Count Question Resolution Program

Human Error and Other Factors Contributed to University of North Carolina Counting Errors

The duplicate counting of nearly 2,700 students at the University of North Carolina (UNC) at Chapel Hill during the 2000 Census resulted from a combination of factors. The incident is interesting because it shows how the various safety nets the Bureau has built to ensure an accurate count can be undermined by human error, the limitations of census-taking operations, and other events that in some cases occur years before Census Day (April 1, 2000).

The duplicate count was discovered after CQR began when the director of the Charlotte regional office (a UNC graduate), asked one of her geographers (also a UNC graduate), to see whether the UNC dormitories were counted in their correct locations. According to the geographer, the director’s curiosity was aroused after the CQR program found problems with the geocoding of dormitories at other schools in the Charlotte region. The geographer told us he initiated an internal CQR case in the summer of 2001 after discovering that two UNC dormitories were geocoded to the wrong census block. Upon further research, where he reviewed information from the census address file and the UNC Web site, the geographer concluded that, in addition to the geocoding error, a large number of dormitories and their occupants were counted in error.

Ultimately, by matching census records, the Bureau determined that 1,583 dormitory rooms in 26 buildings—and the 2,696 students who had resided in them—were included twice in the 2000 Census. On the basis of our interviews with Bureau staff and review of pertinent documents, the following sequence of events led to these erroneous enumerations: The Bureau divides the places where people live into two broad categories: group quarters, which include prisons, dormitories, and group homes; and housing units, which consist of single family homes, apartments, and mobile homes. During the 2000 Census, the Bureau had distinct procedures for building its group quarters and housing unit address lists and enumerating their residents. For example, the Bureau typically enumerates college dormitories by working with schools to distribute census questionnaires to students. Conversely, the Bureau enumerates residents of housing units by delivering questionnaires directly to them through the mail. In the UNC situation, the 26 UNC dormitories were listed correctly in the Bureau’s group quarters database and incorrectly in the Bureau’s housing unit database.

Concerned there could be systemic issues with the Bureau’s address list, staff at the Bureau’s headquarters investigated the source of the problem following the initial discovery by Charlotte employees. The headquarters review found that the dormitories were improperly included in the U.S. Postal Service’s address file, which it initially shared with the Bureau in November 1997 and continued to update through early 2000. The Bureau uses this database to help build its housing unit address list. Specifically, the Bureau discovered that the data field that normally contains a street address erroneously contained a unit number and the name of a UNC dormitory. The Bureau had no explanation for how the dormitory names got into the U.S. Postal Service’s address file.

Other procedures designed to verify census addresses produced conflicting results, compounding the problem. One procedure in 1998 mistakenly confirmed the dormitories as housing units, while another procedure— called block canvassing—correctly flagged the addresses for deletion from the Bureau’s housing unit address list. However, under the Bureau’s protocols, to ensure an address was not improperly removed from the census, an address had to be flagged twice to be deleted. During nonresponse follow-up in 2000, where enumerators visited housing units that failed to send back the questionnaires that were mailed to them, the Bureau had a third opportunity to uncover the error. Because the enumerators involved in this operation provided inconsistent information, the Bureau ultimately did not delete any housing units included in the initial census.

As part of the CQR case analysis, staff in the Bureau’s Decennial Statistical Studies Division checked the Bureau’s address file for any records that contained the word “dorm” in the address field to determine whether a similar duplication occurred at other schools. This would have picked up the word “dormitory” and its variants. On the basis of this search, the Bureau concluded that a similar issue was not problematic elsewhere in the country.

Comments from the Department of Commerce

GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, Robert Goldenkoff, Keith Steck, Timothy Wexler, Robert Parker, Michael Volpe, Andrea Levine, and Elena Lipson made key contributions to this report.