2000 Census: Refinements to Full Count Review Program Could Improve Future Data Quality
Released: 2002-07-26T00:00:00

Letter

Scope and Methodology

To obtain information on the number and kinds of issues identified by the FSCPE and bureau analysts and to determine how the bureau used the information developed during the Full Count Review program, we analyzed the work papers submitted by FSCPE members and other participants in the Full Count Review program. We also analyzed data from the bureau’s Count Review Information System, a database that the bureau used to track issues flagged during the review process. We did not independently verify the information it contained.

To identify lessons learned for future improvements, we examined bureau training manuals, statements of work, process models, and other documents that described the objectives, processes, and decision-making criteria. We also reviewed the results of a survey the bureau conducted of FSCPE members that asked them to rate their experience with Full Count Review processes and tools, bureau staff, and the overall effectiveness of the Full Count Review program. In addition, we interviewed managers in the bureau’s Population Division and other officials responsible for implementing the Full Count Review program, as well as three FSCPE members.

We performed our audit in Washington, D.C., and the bureau’s headquarters in Suitland, Maryland, between May 2001 and April 2002. Our work was done in accordance with generally accepted government auditing standards.

On April 26, 2002, we requested comments on a draft of this report from the Secretary of Commerce. The Secretary forwarded the bureau’s written comments on June 11, 2002 (see app. II). We address them in the “Agency Comments and Evaluation” section of this report.

Background

Accurate census results are critical because the data are used to reapportion seats in the House of Representatives and for congressional redistricting. Moreover, census data remain an important element in allocating federal aid to state and local governments. With billions of dollars at stake, the data are scrutinized intensely for accuracy.

To help ensure the accuracy of census data, the bureau conducted a number of quality assurance programs throughout the course of the census. One such program was the Full Count Review program, which was designed to rapidly examine, rectify if possible, and clear census data files and products for subsequent processing or public release. The bureau expected data analysts to identify data discrepancies, anomalies, and other data “issues” by checking the data for its overall reasonableness, as well as for its consistency with historical and demographic data, and other census data products. The Full Count Review program ran from June 2000 through March 2001.

According to bureau officials, because the bureau could not complete the Full Count Review workload without a costly staff increase, some of the analysts’ work was contracted to members of the FSCPE, an organization composed of state demographers that works with the bureau to ensure accurate state and local population estimates.

The bureau contracted with 53 FSCPE members who reviewed data for 39 states and Puerto Rico. Bureau employees reviewed data for the 11 remaining states and the District of Columbia without FSCPE representation in Full Count Review. Bureau and FSCPE analysts were to ensure that (1) group quarters were correctly placed or “geocoded” on census maps, and that their population counts and demographic characteristics appeared reasonable and (2) population counts of other areas were in line with population estimates. They were to describe each issue flagged and provide supporting documentation derived from bureau resources and/or resources of the respective state government. Additionally, bureau officials stated that staff from the regional offices reviewed demographic data from the 50 states, Puerto Rico, and the District of Columbia. They focused on identifying inconsistent demographic characteristics and did not necessarily concentrate on any one particular state or locality. The bureau reimbursed state governments for wages and expenses FSCPE members incurred.

A separate set of employees from the bureau’s Population Division assessed issues identified by Full Count Review analysts based on (1) the adequacy of the documentation supporting each issue, and (2) whether or not they believed the issue to be resolvable through follow-up research by the bureau. Those issues deemed to have adequate documentation were classified as a “group quarters,” “housing unit,” or “household” or “other” issue. Bureau officials told us that the remaining issues could not be categorized because the nature of the issue could not be determined from the documentation.

The Numbers and Kinds of Issues Identified During Full Count Review

Bureau data show that after reviewing census data for 39 states and Puerto Rico, FSCPE members identified a total of 1,402 issues, or about 29 percent of the 4,809 issues collectively flagged during Full Count Review (see table 1). Since the bureau has yet to resolve most of these issues, it is not known whether they are necessarily errors.

Table 1 also shows that group quarters issues were those most frequently identified by the bureau, accounting for 1,599 of the 4,809 issues identified (33 percent). Group quarters issues relate to suspected discrepancies in the population counts and locations of prisons, dormitories, nursing homes, and similar group living arrangements. Analysts also identified 479 housing unit issues (10 percent of the total), and 288 household issues (6 percent of the total). With housing unit issues, the count of occupied housing units differed from what analysts expected while household issues had population data for occupied residences that differed from what analysts expected. There were also 383 issues (8 percent) that the bureau classified as “other”. They contained questions concerning the demographic characteristics of the data such as age, race, and gender. The bureau was unable to classify 2,060 issues (43 percent). Bureau officials told us that in these cases, analysts did not provide sufficient documentation for the bureau to determine the nature of the issue.

According to bureau officials, bureau analysts identified a larger number of issues than FSCPE members—and a far larger number of issues for which the bureau could not assign a type—because bureau analysts used an automated process that compared data from the 2000 Census to independent benchmarks such as the 1990 Census, and flagged any anomalies. This process alerted bureau officials that there were data discrepancies, but did not indicate their nature. By comparison, FSCPE members compared census data to administrative records and other data, and were better able to document specific issues.

Examples of the three issue categories and how they were found include: Group quarters issues: Analysts noticed that the group quarters population count in a particular census tract of a large midwestern city appeared to be too high, while a neighboring tract had a correspondingly low group quarters population count. By comparing state administrative records to information obtained from bureau resources, analysts determined that bureau data had placed college dormitories in the wrong tract.

Housing unit issues: An urban area had a large amount of redevelopment since the 1990 census. As part of this, several condominiums and apartment complexes were built which substantially increased the number of housing units in a particular census tract. However, when the analyst compared population data from the 1990 Census and 2000 Census, the 2000 Census did not appear to reflect this increase, and it was flagged.

Household issues: Data from the 2000 Census appeared to accurately reflect the large amount of new house construction that had taken place within a specific census tract. However, because the population count differed from that indicated by other data sources, the analyst flagged it as an issue to avoid undercounting the population.

The Bureau Resolved Few Issues Prior to Releasing Public Law Data

Bureau officials told us that they used the Full Count Review program to identify systemic errors such as those that could be produced by software problems. None were found. The officials noted that the bureau generally did not use the Full Count Review program to resolve individual issues. According to bureau officials, the bureau corrected data for 5 of the 4,809 issues prior to the December 31, 2000, release of reapportionment data and the April 1, 2001, release of redistricting data.

According to bureau officials, FSCPE members identified the five issues, all of which involved group quarters that were placed in the wrong locations, but the population counts were correct. They included (1) a military base in Nevada, (2) 10 facilities at a college in Wisconsin, (3) 9 facilities at a prison in New York City, (4) 14 facilities at a Washington prison, and (5) a federal medical center in Massachusetts.

Bureau officials said that the bureau was able to correct these issues for two reasons. First, FSCPE analysts found them early in the Full Count Review program, while the bureau was processing a key geographic data file and was thus able to incorporate the corrections before the data were finalized. Second, the FSCPE analysts had thoroughly documented the issues and recommended how the bureau should correct the errors. The five errors did not require additional research or field verification.

Bureau officials told us that they lacked the time to research the remaining issues, as well as field staff to inspect purported discrepancies prior to the release of the public law data. As a result, the bureau missed an important opportunity to verify and possibly improve the quality of the data, and instead the apportionment and redistricting data were released with more than 4,800 unresolved issues. Until these issues are resolved, uncertainties will surround the accuracy of the census data for the affected localities.

Some of the issues might be resolved under the CQR program, which the bureau designed to respond to challenges to housing unit and group quarters population counts received from state, local, or tribal governments. However, as shown in table 2, of the 4,804 issues remaining after Full Count Review, 1,994 (42 percent) were referred to CQR, and of these, 537 (11 percent) were accepted for further investigation. The remaining 1,457 issues referred to CQR did not meet the bureau’s documentation requirements and consequently, the bureau took no further action on them (see app. 1 for the disposition of Full Count Review data issues by state).

Lessons Learned That Can Inform Future Data Clearance Reviews

The overall results of the Full Count Review program and FSCPE members’ participation appear to be mixed. On the one hand, the bureau reported that the Full Count Review program was successful in that it met a number of performance goals. For example, the bureau reported that the Full Count Review program was comprehensive in its review of geography and content, and was completed in time to release the public law data on schedule.

Moreover, between January and February 2001, the bureau surveyed the 40 entities that participated in Full Count Review and the results suggest that most FSCPE members were satisfied with their Full Count Review experience. For example, respondents indicated that they were generally satisfied with such aspects of the program as its processes and technical tools, bureau staff, and the overall effectiveness of the review in terms of positioning states to use and understand census data. In addition, bureau officials believe the Full Count Review program benefited from FSCPE members’ local demographic knowledge.

Nevertheless, our review of the Full Count Review program highlighted several areas where there is room for future improvement. It will be important for the bureau to address these shortcomings as its preliminary plans call for a similar operation as part of the 2010 Census. According to bureau officials, the bureau plans to include a Full Count Review program in census tests it expects to conduct later in the decade.

Foremost among the areas in need of improvement is resolving, to the extent practical, a larger number of data issues prior to the release of apportionment data by December 31 of the census year, and redistricting data by April 1 of the following year. We found three factors that limited the bureau’s ability to do so.

First, according to bureau officials, resolving individual issues was outside the scope of the Full Count Review program. They explained that the program was poorly integrated with other census operations and units that could have investigated the issues and corrected the data if warranted. This was because the Full Count Review program, with FSCPE participation, was not conceived until February 1999, which was extremely late in the census cycle, coming just 14 months before Census Day, April 1, 2000.

The timing of the decision stemmed from the Supreme Court’s January 1999 ruling that prohibited the bureau from using statistical sampling for purposes of congressional apportionment (the bureau originally planned a “one-number” census that would have integrated the results of a sample survey with the traditional census to provide one adjusted set of census numbers). Faced with the larger workload of reviewing two sets of data— adjusted and unadjusted—the bureau decided to enlist the help of FSCPE members in order to meet the deadlines for releasing the public law data. Additionally, the bureau’s decision came after the 1998 dress rehearsal for the 2000 Census, which meant that the bureau had no opportunity to test the Full Count Review program in an operational environment.

Bureau officials explained that if more time or staff were available in the future, it would be possible to correct a larger number of individual issues prior to the release of the public law data. They noted that field staff would be needed to help verify issues, and the effort would require close coordination with several bureau units.

A second factor that affected the bureau’s ability to correct a larger number of issues was that the bureau’s requirements for documenting data issues were not clearly defined. For example, the training materials we examined did not provide any specific guidance on the type of evidence analysts needed to support data issues. Instead, the training materials told analysts to supply as much supporting information as necessary. This could help explain the variation that we observed in the quality of the documentation analysts provided. Indeed, while some analysts provided only minimal data, others supported issues with state and local administrative records, historical data, photographs, and maps. In some cases, the bureau had difficulty determining the precise nature of an issue or if in fact an issue even existed.

In contrast, the CQR program provides comprehensive guidelines on the documentation required for making submissions. The guidance available on the bureau’s CQR web site notes that before the bureau will investigate concerns raised by government and tribal officials, such officials must first supply specific information. The guidance then details the information needed to support boundary corrections, geocoding and coverage corrections, and group quarters population corrections.

A third, and related factor that affected the bureau’s ability to resolve a larger number of issues stemmed from the fact that the bureau had no mechanism for managing the Full Count Review workload. Unlike the CQR program, where the bureau required local governments to provide specific documentation before it would commit resources to investigate local data issues, the Full Count Review program had no filter for screening submissions based on the quality of the documentation. Better guidance on documenting issues for the Full Count Review program could make the bureau’s follow-up investigations more efficient.

Another area where there is room for improvement concerns the consistency and clarity in which the bureau communicated the objectives of the Full Count Review program and how the bureau planned to use analysts’ input. For example, materials used to train FSCPE members noted that one purpose of Full Count Review was to document issues and “fix what can be fixed.” However, this appears to be inconsistent with statements made by bureau officials, who noted that resolving individual issues was beyond the scope of the Full Count Review program. Moreover, according to one bureau official, it was not clear internally what was meant by “fix what can be fixed.”

None of the bureau’s documentation or training manuals that we reviewed explicitly stated that the bureau would only check for systemic errors. Because of the inconsistent message on the purpose of the Full Count Review program, the bureau may have set up the expectation that a larger number of issues would be resolved during Full Count Review. For example, one FSCPE member told us that he expected FSCPE members would identify any geographic discrepancies that contrasted with preliminary census data, and the bureau would investigate and make the necessary changes. He noted that both he and his staff were very “dismayed” to find out that certain discrepancies involving group quarters were not resolved prior to the release of the public law data. Another FSCPE member told us that participants were strongly motivated by the expectation that everything would be done to correct the census data.

Conclusions

The Full Count Review program was one of a series of quality assurance efforts the bureau implemented throughout the census that helped ensure the bureau released accurate data. Moreover, FSCPE members’ participation, and specifically their expertise and knowledge of local geography, demographics, and housing arrangements, had the potential to identify data issues that the bureau might have otherwise missed.

However, the fact that the apportionment and redistricting data were released with around 4,800 unresolved data issues of unknown validity, magnitude, and impact, is cause for concern, and indicates that the bureau missed an opportunity to verify and possibly improve the quality of the public law data. Given the importance of accurate census data and the resources that bureau staff and FSCPE members invest in the Full Count Review program, it will be important for the bureau to explore how to make better use of the program for correcting potential errors in census data in the future.

It will also be important for the bureau to clarify the purpose of the Full Count Review program and convey that purpose clearly and consistently to FSCPE members. Doing so could help ensure that the bureau meets FSCPE members’ expectations.

Recommendations for Executive Action

To help ensure the accuracy and completeness of census data and take full advantage of the Full Count Review program and FSCPE members’ participation, we recommend that the Secretary of Commerce direct the bureau to develop ways to resolve a larger number of data issues prior to the release of the public law data. Specifically, consideration should be given to (1) planning the Full Count Review program early in the census cycle and testing procedures under conditions as close to the actual census as possible, (2) integrating the Full Count Review program with other census organizational units and operations to ensure the bureau has sufficient time and field support to investigate issues, (3) developing clear guidelines on the minimum documentation needed for the bureau to investigate individual data issues, (4) categorizing issues on the basis of the quality and precision of the documentation, and investigating first those issues that are best documented and thus more easily resolved, and (5) exploring the feasibility of using staff from the bureau’s regional offices to help investigate data issues in the field prior to the release of public law data.

Moreover, to ensure no expectation gaps develop between the bureau and FSCPE members, the Secretary of Commerce should also ensure that the bureau clarifies and consistently communicates to participants the objectives of the Full Count Review program and how the bureau plans to use the information derived from it.

Agency Comments and Our Evaluation

The Secretary of Commerce forwarded written comments from the Bureau of the Census on a draft of this report (see app. II). The bureau concurred with all of our recommendations and had no comments on them. The bureau also provided minor technical corrections that we incorporated in our report as appropriate.

As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time, we will send copies to other interested congressional committees, the Secretary of Commerce, and the Director of the Bureau of the Census. Copies will be made available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. Corinna Wengryn, Ty Mitchell, and Robert Goldenkoff made major contributions to this report. If you have any questions concerning this report, please contact me on (202) 512-6806.

Appendix I: Disposition of Data Issues Appendix I: Disposition of Data Issues Identified During Full Count Review by State Identified During Full Count Review by State

Appendix II: Comments From the Department of Commerce

Appendix III: GAO Products on the Results of the 2000 Census and Lessons Learned for a More Cost-Effective Census in 2010

U.S. General Accounting Office. 2000 Census: Coverage Evaluation Matching Implemented As Planned, but Census Bureau Should Evaluate Lessons Learned. GAO-02-297. Washington, D.C.: March 14, 2002.

U.S. General Accounting Office. 2000 Census: Best Practices and Lessons Learned for a More Cost-Effective Nonresponse Follow-Up. GAO- 02-196. Washington, D.C.: February 11, 2002.

U.S. General Accounting Office. 2000 Census: Coverage Evaluation Interviewing Overcame Challenges, but Further Research Needed. GAO- 02-26. Washington, D.C.: December 31, 2001.

U.S. General Accounting Office. 2000 Census: Analysis of Fiscal Year 2000 Budget and Internal Control Weaknesses at the U.S. Census Bureau. GAO-02-30. Washington, D.C.: December 28, 2001.

U.S. General Accounting Office. 2000 Census: Significant Increase in Cost Per Housing Unit Compared to 1990 Census. GAO-02-31. Washington, D.C.: December 11, 2001.

U.S. General Accounting Office. 2000 Census: Better Productivity Data Needed for Future Planning and Budgeting. GAO-02-4. Washington, D.C.: October 4, 2001.

U.S. General Accounting Office. 2000 Census: Review of Partnership Program Highlights Best Practices for Future Operations. GAO-01-579. Washington, D.C.: August 20, 2001.

U.S. General Accounting Office. Decennial Censuses: Historical Data on Enumerator Productivity Are Limited. GAO-01-208R. Washington, D.C.: January 5, 2001.

U.S. General Accounting Office. 2000 Census: Information on Short- and Long-Form Response Rates. GAO/GGD-00-127R. Washington, D.C.: June 7, 2000.