Data Quality: Improvements to Count Correction Efforts Could Produce More Accurate Census Data
Published: 2005-06-20T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The U.S. Census Bureau (Bureau) conducted the Count Question Resolution (CQR) program to correct errors in the count of housing units as well as dormitories and other group living facilities known as group quarters. GAO was asked to assess whether CQR was consistently implemented across the country, paying particular attention to whether the Bureau identified census errors that could have been caused by more systemic problems. GAO also evaluated how well the Bureau transitioned to CQR from an earlier quality assurance program called Full Count Review.
section_title: What GAO Found
paragraphs: The CQR program, which ran from June 30, 2001, to September 30, 2003, played an important role in improving the quality of data from the 2000 Census in that it corrected numbers affecting 47 states and over 1,180 governmental units. Although this is a small percentage of the nation's more than 39,000 government entities, the count revisions impacted private homes, prisons, and other dwellings and, in some cases, were significant. For example, when the Bureau deleted duplicate data on students at the University of North Carolina at Chapel Hill and made other corrections, that state's head count dropped by 2,828 people. Similarly, CQR found that more than 1,600 people in Morehead, Kentucky, were counted in the wrong location. GAO identified several shortcomings with the CQR program, including inconsistent implementation by the Bureau's regional offices and the posting of inaccurate data to the Bureau's Web-based errata report. Moreover, while CQR found the counting of group quarters to be particularly problematic, the Bureau did not perform an active, nationwide review of these known trouble spots, and thus missed an opportunity to potentially improve the accuracy of the data for these dwellings. Further, because CQR had more stringent documentation requirements compared to a preceding program called Full Count Review, CQR rejected hundreds of unresolved full count issues, missing another opportunity to improve the data. As its plans proceed for the 2010 Census, it will be important for the Bureau to address the operational issues GAO identified. Moreover, because the data for apportionment and redistricting were later found to be flawed for some jurisdictions, it will be important for the Bureau to develop a count correction program that is designed to systematically review and correct these essential figures.

section_title: Letter
paragraphs: Complete and accurate data from the decennial census are central to our  democratic system of government. As required by the Constitution, census  results are used to apportion seats in the House of Representatives. Census  data are also used to redraw congressional districts, allocate billions of  dollars in federal assistance to state and local governments, and for many  other public and private sector purposes. Failure to deliver quality data  could skew the equitable distribution of political power in our society,  impair public and private decision making, and erode public confidence in  the U.S. Census Bureau (Bureau).
To ensure it delivers accurate data, the Bureau employs a number of quality  assurance programs throughout the course of the census. One such effort  during the 2000 Census was the Count Question Resolution (CQR)  program, which enabled state, local, and tribal governments to formally  challenge the counts of housing units and “group quarters” (dormitories,  prisons, and other group living facilities), and their associated populations.  Bureau personnel could initiate a review of the counts as well.
Although the Bureau did not design CQR with the intention of  incorporating any of the corrections that resulted from it into Census 2000  data products—including the numbers used for congressional  apportionment and redistricting (figures commonly referred to as “public  law data”)—governmental entities could use the updated information when  applying for federal aid that uses census data as part of an allocation  formula, as well as for other purposes. Because the count corrections could  have political and financial implications for states and localities, it was  important for the Bureau to carry out CQR consistent with its protocols.  CQR began on June 30, 2001, and no new submissions were accepted after  September 30, 2003.
This letter responds to your request to review the conduct of the CQR  program. As agreed with your offices, we reviewed the results of the CQR  program and assessed whether the program was consistently implemented  across the country. In doing this, we paid particular attention to the extent  to which the Bureau reviewed the census data for errors that could have  been caused by broader, more systemic problems. We also evaluated how  well the Bureau transitioned to CQR from an earlier quality assurance  program called Full Count Review.
To meet these objectives, we reviewed relevant program documents and  examined case files and conducted on-site inspections at four of the  Bureau’s regional offices where some of the largest CQR corrections took  place. We also interviewed officials and staff responsible for administering  the CQR program at the Bureau’s headquarters and 12 regional offices. We  did our audit work between February 2004 and March 2005 in accordance  with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: The Bureau launched the CQR program on June 30, 2001, as the last in a  series of quality assurance initiatives aimed at improving the accuracy of  2000 Census data (see fig. 1). Specifically, the CQR program provided a  mechanism for state, local, and tribal governments to have the Bureau  correct errors in certain types of census data. The Bureau referred to these  challenges as “external cases.” Bureau personnel could also initiate  reviews of suspected count errors, independent of these challenges, for  further review. These were known as “internal cases.” Many of the internal  cases were unresolved issues inherited from Full Count Review. Indeed,  when the Full Count Review program began, the Bureau planned to fold  unresolved issues from that program into CQR. The Bureau accepted no  new submissions after the program officially ended on September 30, 2003,  although it continued to review challenges submitted before the deadline  and completed the final revisions in the summer of 2004.
Three types of corrections were permissible under the CQR program:  (1) boundary corrections, where a jurisdictional boundary of a functioning  governmental unit was in the wrong location; (2) geocoding corrections,  where the Bureau placed the living quarters and their associated  population in the wrong location; and (3) coverage corrections, where the  Bureau properly enumerated specific living quarters and their  corresponding population during the census but incorrectly added or  deleted the information during data processing. Bureau officials were to  research cases using existing Bureau data gathered during the 2000 Census;  they could not conduct any new fieldwork to resolve count questions. The  Bureau required governmental entities to accompany their challenges with  specific documentation before it would investigate their claims.
Importantly, under the design of CQR, if a governmental unit had evidence  that the Bureau missed housing units or group quarters that existed on  Census Day 2000 (April 1), but the Bureau’s records indicated that all of the  Bureau’s boundary information, geocoding, and processing were properly  implemented, the Bureau would not change the data. Rather, the Bureau  was to address this as part of the planning process for the 2010 Census.
If the CQR program corrected the population or housing unit counts of a  particular entity, the Bureau was to issue revised, official figures for that  jurisdiction. The governmental unit could then use the updated numbers  for future programs requiring 2000 Census data. CQR corrections were also  used to modify annual post-censal estimates beginning December 2002 and  were publicized on the Bureau’s Census 2000 and American FactFinder  Web sites (www.census.gov and www.factfinder.census.gov, respectively),  as part of the 2000 Census notes and errata. However, CQR was not  designed or publicized as a mechanism to correct the census results for  purposes of apportionment and redistricting.
In compliance with legal requirements, the Bureau produced  apportionment data by December 31, 2000, and redistricting data by April 1,  2001 (this information is known collectively as public law data). Although  the law does not require that states use census data to redraw the  boundaries of congressional districts, most states have always done so.  Nothing would preclude the states from using the corrected data for  redistricting. The general perception of the impartiality of the Bureau and  the great cost and administrative effort required to take a census have been  strong arguments in favor of using the Bureau’s data.
section_title: Scope and Methodology
paragraphs: As agreed with your offices, we assessed whether the program was  consistently implemented across the country, paying particular attention to  the extent to which the Bureau reviewed the census data for errors that  could have been caused by broader, more systemic problems, such as  shortcomings with a particular census-taking procedure. We also evaluated  how well the Bureau transitioned from an earlier quality assurance  program used in the 2000 Census, Full Count Review.
To assess the implementation of the CQR program, we obtained a  headquarters perspective by reviewing program documents and case files  at the Bureau’s offices in Suitland, Maryland, as well as program results  reported on the Bureau’s Web site. As part of this assessment, we reviewed  the program’s internal controls, especially those controls related to  ensuring data quality. We also interviewed Bureau officials responsible for  administering the program.
To determine how CQR was implemented in the field, we visited 4 of the  Bureau’s 12 regional offices—Charlotte, North Carolina; Denver, Colorado;  Kansas City, Missouri; and Los Angeles, California (see fig. 2).
We selected these regions because they included the six states and  10 governmental units within those states where the largest CQR count  revisions occurred. We supplemented these cases by selecting an  additional seven states and 61 places within the four regions for further  examination. The 61 localities were selected because they represented the  full spectrum of CQR cases and were geographically diverse.
At each of the four regions, we reviewed regional case file information and  interviewed Bureau personnel responsible for implementing CQR, such as  program managers and geographers. We also made a site visit to at least  one type of facility—including prisons, apartment buildings, and  dormitories—in each region to understand firsthand the nature of the  errors and the corrections made. To augment these four regional visits and  obtain a more complete picture of how CQR was implemented, we used a  structured telephone interview to elicit information from program officials  at the Bureau’s eight remaining regional offices that we did not visit in  person.
To determine the extent to which the Bureau reviewed census data for  systemic errors, and its procedures for folding unresolved cases from the  Full Count Review program into CQR, we examined program manuals,  memoranda, and other documents, and interviewed officials in the  Bureau’s headquarters and all of its regional offices. As part of this effort,  we also analyzed the CQR case-tracking data in an attempt to determine the  number of unresolved Full Count Review cases that were rolled into the  CQR program. However, we were unable to do this because the tracking  system did not contain information on which CQR cases originated as full  count issues.
We requested comments on a draft of this report from the Secretary of  Commerce. On May 20, 2005, we received the Acting Deputy Secretary’s  written comments and have reprinted them in appendix III; we address  them in the Agency Comments and Our Evaluation section of this report.
section_title: CQR Program Corrected Numerous Data Errors, but More Consistent Implementation and Other Improvements Are Needed
paragraphs: Overall, the CQR program corrected data affecting over 1,180 of the  nation’s more than 39,000 governmental units. The revisions impacted a  range of housing types including private homes with only a handful of  residents, to college dormitories and prison cell blocks with populations in  the thousands. At the same time, however, we identified several  shortcomings with the CQR program, including inconsistent handling of  internal cases by the Bureau’s regional offices and inaccurate data being  posted to the Bureau’s public Web site. Moreover, while CQR found the  counting of group quarters in their correct location—a problem known as  geocoding error—to be particularly challenging, the Bureau did not  perform a nationwide review of these known trouble spots, and thus  missed an opportunity to improve the accuracy of the data for these  dwellings.
subsections: section_title: CQR Program Corrected Errors in Hundreds of Governmental Units
paragraphs: Nationwide, the CQR program corrected count errors involving  governmental units in 47 states, Puerto Rico, and the District of Columbia  (see fig. 3). Three states—Maine, New Hampshire, and Rhode Island—had  no CQR corrections.
The corrections affected over 1,180 governmental units in the United  States. Although this is a small percentage of the nation’s more than  39,000 governmental units, the impact of those changes on local  governments was, in some cases, substantial, and could have implications  for federal assistance and state funding programs that use census numbers  in their allocation formulas, as well as other applications of census data.
For example, officials in one Kentucky county challenged the geocoding of  a housing unit located near new precinct and congressional district  boundaries. They told the Bureau that the new boundaries split the county,  and they were concerned that the geocoding error would affect where the  housing unit’s few occupants registered to vote. Because the housing unit  was improperly geocoded, the Bureau corrected the data.
With respect to fiscal effects, the Controller of the State of California uses  population figures as the basis for refunding a portion of state taxpayer  fees—including automobile licensing fees—to cities and counties. Because  of an error in the 2000 Census, Soledad, California, officials estimated it  lost more than $140,000 in state refunds when over 11,000 residents were  incorrectly counted in two nearby cities’ populations, according to city and  state officials. Although CQR eventually corrected the error, Soledad did  not recover the funds that went to the other cities.
Other examples of large CQR corrections include the following (See app. I  for a complete list of state-level population changes): North Carolina’s population count was reduced by 2,828 people, largely  because the Bureau had to delete duplicate data on almost 2,700  students in 26 dormitories (see fig. 4) at the University of North Carolina  (UNC) at Chapel Hill. The erroneous enumerations occurred, in large  part, because of mistakes that occurred in various preparatory activities  leading up to the 2000 Census. (See app. II for a more detailed discussion  of this incident.)
The population count of Morehead, Kentucky, increased by more than  1,600 when CQR found that a large number of students from Morehead  State University’s dormitories were erroneously excluded from the city’s  population. During the 2000 Census, the Bureau had incorrectly  identified the dormitories as being outside city limits and in an  unincorporated area of Rowan County.
The population count of Cameron, Missouri, was off by nearly  1,500 people when the Bureau found that the prison population of the  state’s Crossroads Correctional Center was inadvertently omitted from  the town’s headcount (see fig. 5). The correction to the town’s  population accounted for the entire 1,472 person increase in Missouri’s  total population under the CQR program.
The population of the city of Waseca, Minnesota, increased by more  than 1,100. The 2000 Census had mistakenly included the prison  population for the Waseca Federal Correctional Institute in two  surrounding townships in Waseca County. The CQR program resulted in  the population being shifted to the city.
The population of Colorado increased by more than 750 in large part  because a processing error in counting housing units in Grand Junction  initially excluded almost 700 people from the city’s population total.
The population of Denver and Arapahoe Counties in Colorado shifted by  more than 900 because the Bureau had incorrectly assigned the location  of two apartment complexes. As a result of CQR, the apartment  complexes were incorrectly identified and counted as being in Denver  but under CQR were later found to be in adjoining Arapahoe County.
section_title: CQR Program Was Unevenly Executed
paragraphs: The Bureau’s 12 regional offices did not always adhere to the same set of  procedures when developing internal cases, and this, in turn, produced  uneven results. Importantly, the procedures used to execute public  programs need to be well documented, transparent, and consistently  applied in order to ensure fairness, accountability, defensible decisions,  and reliable outcomes. To do otherwise could raise equity questions.
One variation in the way internal cases were handled was evident at the  Bureau’s Los Angeles Regional Office, which appeared to be the only region  to do comprehensive methodical research to actively identify possible  count errors beyond those that were submitted by governmental entities.  According to the office’s senior geographer, the geography staff developed  a structured approach to systematically examine census data from all the  prisons and colleges within the office’s jurisdiction, because the data on  both types of group quarters were known to be problematic. He added that  the more problems they found, the more they were motivated to keep  digging. The geographer noted that the in-depth review was possible  because the Los Angeles region covers only the southern half of California  and the state of Hawaii, and thus has fewer governmental units compared  to the Bureau’s other regional offices.
Had it not been for the Los Angeles region’s self-initiated and systematic  review, certain data errors would have gone uncorrected because they  were not identified by the affected jurisdiction. For example, regional staff  found instances where college dormitories were counted in the wrong  geographic location, which, in turn, affected the population counts of their  surrounding locales. Such was the case with California State University  Monterey Bay (CSUMB) and the University of California at Santa Barbara  (UCSB). As a result, the Bureau transferred a population of more than 1,400  between the towns in which they were initially counted and in which  CSUMB is located and shifted a population of more than 2,700 between the  city and the unincorporated area of the county in which UCSB is located.
The Bureau’s Charlotte Region, while also more active than the Bureau’s  other offices that generated internal CQR cases, seemed to be less  methodical and comprehensive than Los Angeles in its approach. For  example, although Charlotte geographers detected the duplicate count of  almost 2,700 students at the University of North Carolina mentioned in the  previous section, their research was not the result of any systematic  review. Rather, it came about largely because of the curiosity of key  employees in the Charlotte office, who were also alumni of the school.  (See app. II for more details on the circumstances surrounding the  duplicate count.)
section_title: Better Guidance and Training Could Improve Implementation
paragraphs: Vague guidance was one reason for the disparate handling of internal cases.  For example, the CQR procedural manual indicates that the Bureau’s  12 regional offices were to research CQR cases “as appropriate.” However,  the manual did not define whether this meant that the regional offices  should initiate their own data reviews or merely verify CQR cases  submitted by governmental units. More generally, numerous geographers  we interviewed—the primary users of the manual—did not find it user- friendly, noting it was confusing, complex, or impractical. For example, a  geographer pointed out that the manual did not have an index covering the  eight chapters and 26 appendixes, which would have helped them more  quickly find information and procedures. In addition, we found that the  manual and other documents did not discuss how program staff were to  address Full Count Review issues.
The Bureau’s training was also problematic and likely added to the  implementation disparities. For example, geographers in five regions told  us that during training they were instructed or given the impression they  were not to generate additional internal cases beyond the small number of  count errors that had already been identified at the beginning of the  program. Also, geographers in two of these regions told us they were  specifically told not to investigate any count errors they found that were  outside the scope of the cases that governmental units submitted.  Conversely, geographers in the other seven regions said they were not  restricted in any way.
There were other training issues as well. According to the Bureau’s draft  CQR program assessment, the final version of which is pending, some  training materials were developed at the last minute and were never  finalized, and training began before needed software was in place at all the  research divisions. Proper training was particularly important because, as  the draft evaluation notes, staff assigned to the CQR program had census  experience but limited geographic and field operations knowledge. Others  had limited or no Census 2000 software program experience.
section_title: Internal Control and Quality Assurance Problems Led the Bureau to Report Erroneous Data
paragraphs: Federal internal control standards call on agencies to employ edit checks  and other procedures to ensure their information processing systems  produce accurate and complete data. However, the Bureau’s internal  controls in reporting CQR results were insufficient in that we found, after a  partial review, a number of instances where the Bureau disseminated  inaccurate data on its Web site where it maintains an errata report that lists  the CQR revisions to the 2000 Census data.
Specifically, after comparing data from the errata report to the certified  numbers in the CQR case files, we found errors with the reporting of CQR  housing, group quarters, and population counts. Importantly, our review  found that the revised, certified figures the Bureau provided to affected  jurisdictions were correct. This is significant because the affected  jurisdictions could use these updated numbers for revenue sharing and  other programs that require census data. However, users who obtain  information from the Bureau’s errata report—these can be people in  academia, government, and the private sector—would not have the most  up-to-date information. For example:  The original state-level total housing unit count for Delaware mistakenly  excluded 30,000 housing units.
The revised housing unit count for the Minnehaha County portion of  Sioux Falls, South Dakota, was underreported by almost 47,000 housing  units.
The Burlington County, New Jersey, revised total housing count  mistakenly excluded about 145,000 units.
The errata report excluded 8 of the 12 American Indian and Alaska  Native Areas that had revisions to their housing, group quarter, or total  population counts.
The Bureau later corrected these errors after we brought them to  its attention.
Although the Bureau had controls in place to ensure accurate research and  reporting, the problems we found point to the need for the Bureau to  tighten its procedures to prevent mistakes from slipping into its data  products. For example, the CQR manual included some quality control  steps, such as having headquarters divisions review field research and  results. Further, field geographers told us they consulted one another about  questions or procedures and checked each other’s work, and Bureau  program managers had procedures in place to review final revisions and  certify them as correct.
Documents in the CQR case files we reviewed substantiate these practices.  Also, managers told us they randomly checked data entered into the files  that are the basis for the revisions posted to the errata report. Still, the  number of errors we found after only a partial review of the errata files  raises questions about how effectively the Bureau implemented the quality  assurance procedures, as well as the quality of the data we did not review.  It also underscores the importance of adequate control activities to prevent  these problems from recurring.
section_title: Web-based Errata Report Should Be More User- Friendly
paragraphs: Data users may have encountered problems trying to access certain  information from the Web-based errata report. Additionally, because there  was no link or cross-walk between some of the initial population data the  Bureau released and the CQR revisions, users may have been unaware that  some of the original numbers had been revised.
As shown in figure 6, the Bureau’s Web-based errata report presented  revised data for states, American Indian and Alaska Native Areas, and other  jurisdictions at the state or similar geographic level. Although the table had  embedded links that were supposed to take users to revisions at lower  levels of geography, these links did not always work and produced error  messages instead. We found that unless the users’ software and Internet  access paralleled the Bureau’s, users could not access the more detailed  data using the embedded links. Bureau staff involved with posting the data  to the Web site stated in the summer of 2004 that they were aware of the  problem, but as late as March 2005, the problem had yet to be fixed.
At the same time, the CQR revisions may not be evident to users who  access certain data from the 2000 Census data posted elsewhere on the  Bureau’s Web site. This is because these sites lack notes or flags informing  users that updated figures are available in the census errata report. For  example, the Bureau’s American FactFinder Web site—the Bureau’s  primary electronic source of census data—does not inform users that  revised data on group quarter counts, including the number of correctional  institutions, as well as data on their associated populations, exist as part of  the Bureau’s notes and errata report.
American FactFinder presents data known as Summary File 1 (SF-1),  which is the first data set the Bureau produces from the census, and is used  for purposes of apportionment and redistricting. While the SF-1 data  remain unchanged, other data users may find the revised numbers better  suited to their needs. Figure 7 illustrates the existence of two sets of  numbers without any explanation. Summary File data from American  FactFinder show the population for Soledad, California, as 11,263.  However, the Bureau’s errata report, which reflects the CQR revisions,  shows the Soledad population at 23,015. Because American FactFinder  lacks notes or links that tell users about the revised data, users might  inadvertently obtain erroneous information.
According to Bureau officials, while they thought about adding notes  directing users to the CQR revisions, they decided against it because they  thought it would confuse more people than it would help. They reasoned  that knowledgeable users, such as county planners and state data center  staff, are likely aware of the CQR information and would therefore not  need to be informed about the existence of the notes and errata Web site.
section_title: CQR Errors Highlight Problems with 2000 Census Address List Development Procedures
paragraphs: The errors uncovered by the CQR program highlight some of the limitations  in the way in which the Bureau builds its address list for the decennial  census, particularly in the procedures used to identify and locate group  quarters. For the 2000 Census, the Bureau had three operations that were  primarily designed to locate these types of dwellings. However, given the  number of prisons and other group quarters geocoded to the wrong  location, refinements are needed. Moreover, the Bureau’s draft CQR  program assessment found that the Bureau’s Master Address File had  numerous data entry errors including incorrect spellings, geocoding, and  zip codes.
To its credit, the Bureau is planning several improvements for 2010,  including integrating its housing unit and group quarter address lists. This  could help prevent the type of duplicate counting that occurred at UNC  when the same dormitories appeared on both lists. Likewise, the Bureau’s  planned use of a satellite-based navigational system could help census  workers more precisely locate street addresses.
section_title: Better Strategic Planning and Other Actions Could Improve Future Count Correction Efforts
paragraphs: The CQR program was preceded by a quality assurance program called Full  Count Review, which ran from June 2000 through March 2001 and, like  CQR, was designed to find problems with the census data. However,  although the Bureau planned to fold unresolved full count issues into CQR,  many full count issues were rejected from CQR because the latter program  had more stringent documentation requirements. As a result, the Bureau  was unable to resolve hundreds of additional data issues.
subsections: section_title: Numerous Unresolved Full Count Issues Could Not Be Folded into CQR as Planned
paragraphs: Under the Full Count Review program, analysts were to identify data  discrepancies to clear census data files and products for subsequent  processing or public release. Analysts did so by checking the data for their  overall reasonableness, as well as for their consistency with historical and  demographic census data and other census data products. The types of  issues flagged during Full Count Review included potential discrepancies  involving the counts and/or locations of group quarters, housing units, and  individual households, among others.
As we noted in our July 2002 report, Full Count Review identified 4,809  potential data anomalies. However, of these, just five were corrected prior  to the December 31, 2000, release of apportionment data and the April 1,  2001, release of redistricting data. The corrections included a military base,  a federal medical center, and multiple facilities at two prisons and a college  that were counted in the wrong locations. That the public law data were  released with numerous data issues of unknown validity, magnitude, and  impact, gave us cause for concern, and we noted that the Bureau missed an  opportunity to verify and possibly improve the quality of the information.
When the Full Count Review program began, the Bureau planned to fold  unresolved issues from that program into CQR. Indeed, according to a June  2000 memo on CQR policy agreements, “a by-product of [Full Count  Review] is documentation of unresolved issues for potential use in CQR.”  However, because the CQR program had more rigorous documentation  requirements before it would accept a case compared to Full Count  Review, a number of issues that were deemed suitable for Full Count  Review but were unresolved, were rejected from CQR.
Of the 4,804 issues remaining after Full Count Review, 2,810 issues  (58 percent), were not referred to CQR. Of the 1,994 issues (42 percent)  that were referred to CQR, 537 were actually accepted by the program. The  remaining 1,457 issues referred to CQR did not meet the Bureau’s CQR  documentation requirements and, consequently, the Bureau took no further  action on them.
The Full Count training materials we examined as part of our 2002 review  did not provide any specific guidance on the type of evidence analysts  needed to support data issues. Rather, the materials instructed analysts to  supply as much supporting information as necessary. In contrast, the CQR  program had more rigorous documentation requirements. Guidance  available on the Bureau’s Web site required governmental units to supply  maps and other evidence specific to the type of correction they were  requesting, or the Bureau would not investigate their submissions.
Simply put, Full Count Review identified hundreds of data issues but  lacked the time to investigate the vast majority of them. Then, when the  remaining cases were referred to CQR, most were rejected because they  could not meet CQR’s higher evidentiary bar.
section_title: A Mechanism for Correcting Public Law Data Will Be Critical for Future Enumerations
paragraphs: The Bureau lacked a program specifically designed to correct individual  count errors contained in the apportionment and redistricting data.  Because these numbers were later found to be flawed for some  jurisdictions, as the Bureau proceeds with its plans for the 2010 Census, it  will be important for it to explore options for reviewing and correcting this  essential information before it is released.
Precision is critical because, in some cases, small differences in population  totals could potentially impact apportionment and/or redistricting  decisions. For example, according to an analysis by the Congressional  Research Service, under the formula used to apportion seats in the U.S.  House of Representatives, had Utah’s population count been 855 persons  higher, it would have gained an additional congressional seat and North  Carolina would have lost a seat. However, had the duplicate UNC count and  other errors detected by the CQR program as of September 30, 2003, been  uncovered prior to the release of the public law data, the already narrow  margin determining whether Utah gained a House seat would have dropped  to 87 persons. Although in this particular instance there would not have  been a change in congressional apportionment, it illustrates how the  allocation of House seats can be determined by small differences in  population counts.
section_title: Other Aspects of CQR Could Have Been Better Planned
paragraphs: Better planning could have improved the CQR program in other ways. For  example, the Bureau’s draft evaluation of CQR found, among other issues,  that the three teams working on the planning and development phases of  CQR should have tested implementation plans earlier in the process, and  training materials were not based on the Bureau’s experience in conducting  the 1990 CQR program. Also, there was no mechanism to prioritize cases  based on the magnitude of the error. As a result, regional offices wound up  expending considerable resources on CQR cases that only affected a  handful of dwellings.
The draft evaluation also found that the two software applications the  Bureau chose to administer and track CQR cases did not appear to be up to  the task. Lost cases and documentation, poor integration with other  applications, and the inability to produce reports were among the issues  the evaluation cited.
More generally, the integration and coordination issues that affected CQR  are not unique to that program; to the contrary, our past reports have found  that other components of the 2000 Census were not well planned, which  unnecessarily increased the cost and risk of the entire enumeration. The  need for better strategic planning has been a consistent theme in many of  our past recommendations to improve the Bureau’s approach to counting  the nation’s population and represents a significant management challenge  that the Bureau will need to address as it looks toward 2010.
section_title: The Bureau Is Making Count Correction Plans for 2010
paragraphs: The Bureau is beginning to develop plans for Full Count Review and CQR  for the 2010 Census. As it does so, it will be important for it to develop an  initiative or consolidated program that corrects both systemic and  individual issues, and does so prior to the release of apportionment and  redistricting data. Granted, this effort will be no simple task given the  relatively short time between the closure of the local census offices and the  need to release the public law data within the legally required time frames.
Still, there are steps the Bureau can explore to methodically check the data  for nationwide systemic errors, obtain local input, and investigate any  discrepancies, and do so in an expeditious manner. One approach might be  to consolidate and leverage CQR, Full Count Review, and certain other  Bureau programs.
Indeed, under the Full Count Review program, the Bureau obtained local  input by contracting out some of the work to members of the Federal-State  Cooperative Program for Population Estimates (FSCPE), an organization  composed of state demographers that has worked with the Bureau since  1973 to ensure accurate population counts. The Bureau worked with  FSCPE, in part, because it lacked sufficient staff to complete the review on  its own, but also because the Bureau believed that the members’  knowledge of the demographic characteristics of their states could help the  Bureau examine data files and products, including public law data. FSCPE  members reviewed data for 39 states and Puerto Rico; Bureau employees  reviewed data for the remaining states and the District of Columbia without  FSCPE representation in Full Count Review. Both sets of analysts checked  the data for their overall reasonableness, as well as for their consistency  with historical and demographic data, and other census data products.
Bureau staff from its regional offices reviewed the data as well. They  focused on identifying inconsistent demographic characteristics and did  not necessarily concentrate on any one particular state or locality. Thus,  the Bureau obtained local input that focused on individual states and  smaller jurisdictions, and also performed its own, broader review.
Verifying any data discrepancies could be accomplished by beginning the  count correction effort as local census offices complete nonresponse  follow-up, when enumerators are still available to investigate issues. In  fact, the Bureau is already planning to do this to some degree in 2010 under  another operation called Coverage Improvement Follow-up (CIFU), where  the Bureau is to call or visit housing units that have been designated as  vacant or nonexistent but not confirmed as such by a second source. In the  2000 Census, CIFU began June 26, 2000, and ended on August 23, 2000.  During that time, enumerators contacted 8.9 million housing units and  counted 5.3 million people, according to the Bureau.
The Bureau could explore adding the count correction workload to  enumerators’ CIFU assignments, which would enable the agency to  reconcile possible data errors, as well as add any housing units and group  quarters the Bureau missed during the initial enumeration (As noted in the  background section, CQR could not add any residences that existed on  Census Day but the Bureau had failed to count.).
Further, the Bureau could help automate the count correction process by  using computers to flag any data that exceed any predetermined  tolerances. The Bureau could also develop a system to prioritize count  correction issues to help manage its verification workload.
Importantly, to the extent the Bureau reviews and corrects census counts  prior to the release of the public law data, the Bureau might not need  separate Full Count Review and CQR programs; a consolidated effort might  be more cost effective. At a minimum, to the extent a separate CQR  program is needed, it may not have to be as large or last as long because  presumably the earlier program would have caught the bulk of the  problems.
Regardless, given the possibility that similar data errors might again occur  during the 2010 Census, exploring options for resolving them prior to the  release of public law data would be a sound investment. Reapportionment  and redistricting data would be more accurate; the Bureau’s credibility  would be enhanced; and the need for a large-scale count correction  program along the lines of CQR could be reduced or eliminated.
section_title: Conclusions
paragraphs: The CQR program played an important role in improving the quality of data  from the 2000 Census. Although the net changes in housing and population  counts from the program were small on a national scale, in a number of  instances, they were substantial at the local level, and could affect various  revenue sharing formulas and other programs that use decennial census  data.
Because the program functioned as a safety net—a final opportunity to  catch and correct mistakes that occurred along the chain of events that led  to, and extended beyond Census Day 2000—the results shed light on the  performance and limitations of certain upstream census operations, and  areas where the Bureau should focus its efforts as its plans unfold for 2010.  In this regard, the following is clear: although the Bureau puts forth  tremendous effort to ensure a complete and accurate census, its numerous  procedures and quality assurance operations will be challenged to stay  ahead of the increasing difficulties associated with enumerating a  population that is growing larger, more diverse, and increasingly hard to  locate and count.
The timing of any count correction effort will also be critical. Indeed, we  are concerned that key decisions using data from the 2000 Census  employed figures that, for a number of jurisdictions, were later found to be  flawed. As a result, it will be important for the Bureau to consider  developing a count correction initiative that can complete its work in time  to correct the public law data before that information is released.
Moreover, beyond the inherent demographic obstacles to a successful  census, the results of our CQR review echo several of our past reports on  other aspects of the census, which note that some of the Bureau’s  difficulties stem from a lack of adequate strategic planning and other  management challenges. Ultimately, the success of the 2010 Census will  hinge on the extent to which senior Bureau leadership resolves these  challenges.
With this in mind, resolute action is needed across three fronts. First, it will  be important for the Bureau to ensure, via thorough field testing, that its  planned changes to its address list development procedures help resolve  the geocoding and other operational problems revealed by CQR. Second, it  will be important for the Bureau to improve its count correction efforts by  designing a program that can systematically and consistently review the  public law data and make any corrections prior to the release of those  figures. Third, it will be important for the Bureau to address persistent  strategic planning challenges.
section_title: Recommendations for Executive Action
paragraphs: To help ensure the nation has the best possible data for purposes of  apportionment, redistricting, and other uses of census data, we  recommend that the Secretary of Commerce direct the Bureau to improve  its count correction efforts for the 2010 Census by taking such actions as:  1. Thoroughly testing improvements to the Bureau’s group quarters and  other address list development activities to help ensure the Bureau has  resolved geocoding and other problems with its master address file. 2. Consolidating Full Count Review and CQR into a single program that  systematically reviews and corrects any errors in the public law data  prior to their release.  3. Expediting count correction efforts by initiating data reviews toward  the end of nonresponse follow-up, when the Bureau starts getting  complete data for geographic entities, and enumerators are available to  help investigate any discrepancies. As part of this effort, the Bureau  should consider using computers to systematically search for possible  errors nationwide by checking data at the appropriate level of  geography to ensure population, housing unit, and group quarter  counts, as well as demographic characteristics, appear reasonable and  are consistent with population estimates. Those areas that are outside  of predetermined tolerances should be flagged for further review. The  Bureau should also pay special attention to ensure group quarters are  properly geocoded and counted.  4. Prioritizing the investigation of errors based on the magnitude of the  suspected error or similar triaging formula. 5. Ensuring that instructions on the Bureau’s Web site make it clear that  updated information exists and that users can readily access this  information.
6.
Improving the Bureau’s quality assurance procedures to help ensure  there are no mistakes in the data the Bureau posts on its Web site.  7. Enhancing the training and guidance provided to regional offices to  help ensure they share the same understanding of their roles and  responsibilities and will implement the program consistently. 8. Addressing persistent strategic management challenges, in part,  through early testing to help ensure information systems, training, and  other activities are fully integrated.
section_title: Agency Comments and Our Evaluation
paragraphs: The Acting Deputy Secretary of Commerce provided written comments on  a draft of this report on May 20, 2005, which are reprinted in appendix III.  Commerce stated that “the report provides a good overview of program  results and makes several useful observations and recommendations,” and  specifically agreed with our finding that the process for conducting internal  reviews was not consistently implemented. More generally, however,  Commerce believes the shortcomings we describe reflect “a fundamental  misunderstanding of the goals of the CQR program,” and noted that our  observations and recommendations indicate we believe that CQR should  have been designed to correct the public law data before they were  released during the 2000 Census.
Our concern over the CQR program centers on the way it was implemented  in 2001, rather than the fact that the Bureau did not design the program to  correct the apportionment and redistricting numbers. We agree with  Commerce that this was not the intent of CQR and, as Commerce notes, we  acknowledge this in our report. At the same time, based on the lessons  learned from the 2000 Census, enumeration errors are almost inevitable.  Thus, our recommendations focus on the future, and specifically, the  importance of developing mechanisms for the 2010 Census to review and  correct errors in the public law data to the greatest extent possible before  they are released. We have clarified the report to better reflect GAO’s  position.
Commerce specifically addressed two of our eight recommendations,  disagreeing with both of them. With respect to our recommendation to  consolidate Full Count Review and CQR into a single program for the 2010  Census, Commerce noted that preliminary counts at the census tract or  block level are needed to conduct an effective CQR program, and that  information is not available until close to the deadline for releasing the  apportionment data. Commerce maintains there would be little opportunity  for local entities to review the counts and document potential problems  and even less time for the Bureau to conduct the necessary research and  field work.
Our report recognizes that it would be a challenge for the Bureau to review  and correct census figures and still release the public law data by the  legally required time frames. Still, as we note in our report, we believe the  Bureau could expedite the process by taking such steps as (1) using  computers to check census data for their overall reasonableness and  flagging areas that exceed predetermined tolerances; (2) focusing on  known trouble spots such as group quarters; and (3) beginning the review  process earlier, such as, when local census offices complete their  nonresponse follow-up efforts.
Moreover, as we state in the report, during the 2000 Census, the Bureau  already had programs in place that obtained local input on the census  numbers before the release of the public law data (Full Count Review), and  conducted extensive field operations to investigate certain discrepancies  (Coverage Improvement Follow-up). We believe that it will be important  for the Bureau to not simply replicate these programs for the 2010 Census  or make incremental improvements, but to see whether these programs  could be better leveraged and be more strategically employed to improve  the accuracy of the apportionment and redistricting data.
The other recommendation that Commerce specifically addressed was our  call for the Bureau in 2010 to prioritize the investigation of errors based on  the magnitude of the suspected problem. Commerce maintains that the  Bureau’s policy in 2000 was to handle cases in the order they were received  from local jurisdictions, and asserts this was a fair and reasonable practice.  While this practice is not unreasonable, we continue to believe that it  would be more cost-effective for the Bureau to give priority to those cases  where it could achieve a greater return on its investment in resources  (especially given our findings involving group quarters such as prisons and  college dormitories that affected relatively large population clusters). Our  recommendation echoes the Bureau’s draft evaluation of the CQR program,  which noted that regional offices expended considerable resources on CQR  cases that affected only a handful of dwellings. Moreover, as we state in our  report, prioritizing the Bureau’s workload could help expedite the count  correction process.
Commerce’s comments also included some technical corrections and  suggestions where greater clarity was needed. We revised the report as  appropriate.
We will send copies of this report to the Chairman of the House Committee  on Government Reform, the Secretary of Commerce, and the Director of  the U.S. Census Bureau. Copies will be made available to others on request.  This report will also be available at no charge on GAO’s home page at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-6806 or williamso@gao.gov. Contact points for our Office of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix IV.
section_title: Change in State Populations As a Result of Count Question Resolution Program
section_title: Human Error and Other Factors Contributed to University of North Carolina Counting Errors
paragraphs: The duplicate counting of nearly 2,700 students at the University of North  Carolina (UNC) at Chapel Hill during the 2000 Census resulted from a  combination of factors. The incident is interesting because it shows how  the various safety nets the Bureau has built to ensure an accurate count can  be undermined by human error, the limitations of census-taking operations,  and other events that in some cases occur years before Census Day (April  1, 2000).
The duplicate count was discovered after CQR began when the director of  the Charlotte regional office (a UNC graduate), asked one of her  geographers (also a UNC graduate), to see whether the UNC dormitories  were counted in their correct locations. According to the geographer, the  director’s curiosity was aroused after the CQR program found problems  with the geocoding of dormitories at other schools in the Charlotte region.  The geographer told us he initiated an internal CQR case in the summer of  2001 after discovering that two UNC dormitories were geocoded to the  wrong census block. Upon further research, where he reviewed  information from the census address file and the UNC Web site, the  geographer concluded that, in addition to the geocoding error, a large  number of dormitories and their occupants were counted in error.
Ultimately, by matching census records, the Bureau determined that 1,583  dormitory rooms in 26 buildings—and the 2,696 students who had resided  in them—were included twice in the 2000 Census. On the basis of our  interviews with Bureau staff and review of pertinent documents, the  following sequence of events led to these erroneous enumerations:  The Bureau divides the places where people live into two broad categories:  group quarters, which include prisons, dormitories, and group homes; and  housing units, which consist of single family homes, apartments, and  mobile homes. During the 2000 Census, the Bureau had distinct procedures  for building its group quarters and housing unit address lists and  enumerating their residents. For example, the Bureau typically enumerates  college dormitories by working with schools to distribute census  questionnaires to students. Conversely, the Bureau enumerates residents of  housing units by delivering questionnaires directly to them through the  mail. In the UNC situation, the 26 UNC dormitories were listed correctly in  the Bureau’s group quarters database and incorrectly in the Bureau’s  housing unit database.
Concerned there could be systemic issues with the Bureau’s address list,  staff at the Bureau’s headquarters investigated the source of the problem  following the initial discovery by Charlotte employees. The headquarters  review found that the dormitories were improperly included in the U.S.  Postal Service’s address file, which it initially shared with the Bureau in  November 1997 and continued to update through early 2000. The Bureau  uses this database to help build its housing unit address list. Specifically,  the Bureau discovered that the data field that normally contains a street  address erroneously contained a unit number and the name of a UNC  dormitory. The Bureau had no explanation for how the dormitory names  got into the U.S. Postal Service’s address file.
Other procedures designed to verify census addresses produced conflicting  results, compounding the problem. One procedure in 1998 mistakenly  confirmed the dormitories as housing units, while another procedure— called block canvassing—correctly flagged the addresses for deletion from  the Bureau’s housing unit address list. However, under the Bureau’s  protocols, to ensure an address was not improperly removed from the  census, an address had to be flagged twice to be deleted. During  nonresponse follow-up in 2000, where enumerators visited housing units  that failed to send back the questionnaires that were mailed to them, the  Bureau had a third opportunity to uncover the error. Because the  enumerators involved in this operation provided inconsistent information,  the Bureau ultimately did not delete any housing units included in the  initial census.
As part of the CQR case analysis, staff in the Bureau’s Decennial Statistical  Studies Division checked the Bureau’s address file for any records that  contained the word “dorm” in the address field to determine whether a  similar duplication occurred at other schools. This would have picked up  the word “dormitory” and its variants. On the basis of this search, the  Bureau concluded that a similar issue was not problematic elsewhere in the  country.
section_title: Comments from the Department of Commerce
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, Robert Goldenkoff, Keith Steck,  Timothy Wexler, Robert Parker, Michael Volpe, Andrea Levine, and Elena  Lipson made key contributions to this report.