Census 2000: Design Choices Contributed to Inaccuracy of Coverage Evaluation Estimates
Released: 2004-12-13T00:00:00

Letter

Background

Since 1980, the Bureau has used statistical methods to generate detailed estimates of census undercounts and overcounts, including those of particular ethnic, racial, and other groups. To carry out the 2000 Census’s Accuracy and Coverage Evaluation program (A.C.E.), the Bureau conducted a separate and independent sample survey that, when matched to the census data, was to enable the Bureau to use statistical estimates of net coverage errors to adjust final census tabulations according to the measured undercounts, if necessary. The Bureau obligated about $207 million to its coverage evaluation program from fiscal years 1996 through 2001, which was about 3 percent of the $6.5 billion total estimated cost of the 2000 Census.

While the A.C.E. sample survey of people was conducted several weeks after Census Day, April 1, the “as of” date on which the total population is to be counted, many of the processes were the same as the 2000 Census. For the census, the Bureau tried to count everybody in the nation, regardless of their dwelling, and certain kinds of dwellings, including single-family homes, apartments, and mobile homes, along with demographic information on the inhabitants. For A.C.E., the Bureau surveyed about 314,000 housing units in a representative sample of “clusters”—geographic areas each with about 30 housing units. The sample comprised roughly 12,000 of the about 3 million “clusters” nationwide.

As illustrated in figure 1, the Bureau used a similar process to develop address lists, collect response data, and tabulate and disseminate data— one for the decennial census and one for A.C.E. sample areas. For the Census, the Bureau mailed out forms for mail-back to most of the housing units in the country; hand-delivered mail-back forms to most of the rest of the country; and then carried out a number of follow-up operations designed to count nonrespondents and improve data quality. A.C.E. collected response data through interviewing from April 24 through September 11, 2000.

After the census and A.C.E. data collection operations were completed, the Bureau attempted to match each person counted on the A.C.E. list to the list of persons counted by the 2000 Census in the A.C.E. sample areas to determine exactly which persons had been missed or counted more than once by either A.C.E. or the Census. The results of the matching process, along with data on the racial/ethnic and other characteristics of persons compared, were to provide the basis for A.C.E. to estimate the extent of coverage error in the census and population subgroups and enable the Bureau to adjust the final decennial census tabulations accordingly. The matching process needed to be as precise and complete as possible, since A.C.E. collected data on only a sample of the nation’s population, and small percentages of matching errors could significantly affect the estimates of under- and overcounts generalized to the entire nation.

Since the 2000 Census, we have issued four other reports on A.C.E., addressing its cost and implementation as part of our ongoing series on the results of the 2000 Census, as well as a report on the lessons learned for planning a more cost-effective census in 2010. (See the Related GAO Products section at the end of this report for the assessments issued to date.) These reports concluded, among other things, that while (1) the address list the Bureau used for the A.C.E. program appeared much more accurate than the preliminary lists developed for the 2000 Census and (2) quality assurance procedures were used in the matching process, certain implementation problems had the potential to affect subsequent matching results and thus estimates of total census coverage error.

In the end, the Bureau decided not to use A.C.E.’s matching process results to adjust the 2000 Census. In March 2001, a committee of senior career Bureau officials recommended against using A.C.E. estimates of census coverage error to adjust final census tabulations for purposes of redistricting Congress. In October 2001, the committee also recommended against adjusting census data used for allocating federal funds and other purposes, largely because Bureau research indicated that A.C.E. did not account for at least 3 million erroneously counted persons (mostly duplicates) in the census, raising questions about the reliability of coverage error estimates. In March 2003, after considerable additional research, the Bureau published revised coverage error estimates and again decided not to adjust official census data, this time for the purpose of estimating the population between the 2000 and 2010 censuses.

In light of its 2000 experience, the Bureau officially announced in January 2004 that while it plans to fully evaluate the accuracy of the 2010 census, it will not develop plans for using these coverage error estimates to adjust the 2010 Decennial Census. Bureau officials have told us that there is insufficient time to carry out the necessary evaluations of the coverage estimates between census field data collection and the Bureau’s legally mandated deadline (within 12 months of Census Day) for releasing redistricting data to the states. Furthermore, the Bureau does not believe adjustment is possible. In responding to an earlier GAO report recommending that the Bureau “determine the feasibility” of adjusting the 2010 Census, the Bureau wrote that the 2000 Census and A.C.E. was “a definitive test of this approach” which “provided more than ample evidence that this goal cannot be achieved.” However, in March, the National Academy of Sciences (NAS) published a report that recommended that the Bureau and the administration request and Congress provide funding for an improved coverage evaluation program that could be used as a basis for adjusting the census, if warranted. The Academy agrees with the Bureau that 12 months is insufficient time for evaluation and possible adjustment; in the same publication, NAS recommended Congress consider extending the statutory deadline of 12 months for providing data for redistricting purposes, a suggestion which, if appropriate, could make adjustment possible.

Scope and Methodology

To identify the factors that may have contributed to A.C.E. missing coverage errors in the census, we reviewed evaluations of A.C.E. and the Bureau’s subsequent revisions to its estimation methodology, as well as changes made to the design from its 1990 attempts to estimate coverage. We interviewed Bureau officials responsible for A.C.E. decision making to obtain further context and clarification. We did not attempt to identify all factors contributing to the success or failure of A.C.E in estimating coverage error. Since our focus was on the process and decisions that led to the results rather than on determining the underlying numbers themselves, we did not audit the Bureau’s research, the underlying data, or its conclusions. We relied on the Bureau’s own reporting quality assurance processes to assure the validity and accuracy of its technical reporting, and thus we did not independently test or verify individual Bureau evaluations of their methodologies.

To identify the extent of the census errors not accounted for by A.C.E., we reviewed the descriptive coverage error estimates and the limitations and context of these data as described in the Bureau reports published by the Bureau in March 2001, October 2001, and March 2003.

On, August 9, 2004, we requested comments on the draft of this report from the Secretary of Commerce. On September 10, 2004, the Under Secretary for Economic Affairs, Department of Commerce forwarded written comments from the department (see app. I), which we address in the “Agency Comments and Our Evaluation” section at the end of this report.

Design and Other Bureau Decisions Created Difficulties and Blind Spots in Census Coverage Evaluation

The following Bureau decisions concerning the design of the census and the A.C.E. program created difficulties and blind spots for the coverage evaluation, possibly preventing A.C.E. from reliably measuring coverage error: (1) using residence rules that were unable to capture the complexity of American society, (2) excluding the group quarters population from the A.C.E. sample survey, (3) making various decisions that led to an increase in the number of “imputed” records in the census, (4) removing 2.4 million suspected duplicate persons from the census but not the A.C.E. sample, and (5) reducing the sample area wherein A.C.E. searched for duplicates during matching. However, the Bureau has not accounted for how these design decisions have affected coverage error estimates, which has prevented it from pinpointing what went wrong with A.C.E., and this in turn could hamper its efforts to craft a more successful coverage measurement program for the next national head count.

Bureau Attributes Inaccuracy of Coverage Error Estimates to Residence Rules That Did Not Capture Complexity of U.S. Society

Bureau officials attribute A.C.E.’s inaccuracy primarily to the fact that it used residence rules that do not fully capture the complexity of American society. According to senior Bureau officials, increasingly complicated social factors, such as extended families and population mobility, presented challenges for A.C.E., making it difficult to determine exactly where certain individuals should have been counted. Specifically, in developing A.C.E. methodology, Bureau officials assumed that each person in its sample could be definitively recorded at one known residence that the Bureau could determine via a set of rules. However, individuals’ residency situations are often complicated: Bureau officials cite the example of children in custody disputes whose separated parents both may have strong incentives to claim the children as members of their household, despite census residence rules that attempt to resolve which parent should report the child(ren). In such situations, wherein the residence rules are either not understood, are not followed, or do not otherwise provide resolution, the Bureau has difficulty determining the correct location to count the children. Bureau officials cite similar difficulties counting college students living away from home, as well as people who live at multiple locations throughout the year, such as seasonal workers or retirees.

A.C.E. design also assumed that follow-up interviews would clarify and improve residence data for people for whom vague, incomplete, or ambiguous data were provided and whose cases remained unresolved. However, the Bureau found it could not always rely on individuals to provide more accurate or complete information. In fact, in our earlier reporting on A.C.E. matching, we described several situations wherein conflicting information had been provided to the Bureau during follow-up interviews with individuals, and Bureau staff had to decide which information to use. More recently, the Associate Director for Decennial Census told us that returning to an A.C.E. household to try and resolve conflicting data sometimes yielded new or more information but not necessarily better information or information that would resolve the conflict. The Bureau plans to review and revise its census residence rules for 2010, which may clarify some of this confusion.

Other Design Decisions Also Created Blind Spots in Census Coverage Evaluation

While the Bureau emphasizes residence rules as the primary cause of A.C.E. failure, our research indicates some of the Bureau’s other design decisions created blind spots that also undermined the program’s ability to accurately estimate census error. For example, the Bureau decided to leave people living in group quarters—such as dormitories and nursing homes— out of the A.C.E. sample survey, which effectively meant they were left out of the scope of A.C.E. coverage evaluation (see fig. 2).

As a result, the matching results could not provide coverage error information for the national group quarters population of 7.8 million. In addition, the Bureau did not design A.C.E. matching to search for duplicate records within the subset of this population counted by the census, though later Bureau research estimated that if it had, it would have measured over 600,000 additional duplicates there. In response to our draft report the Department wrote that coverage evaluation was designed to measure some of these duplicates since during its follow-up interviews at households during A.C.E. matching, the Bureau included questions intended to identify college students living away at college.

While coverage evaluation in 1990 included some group quarters, such as college dormitories and nursing homes, within its sample, the Bureau reported that the high mobility of these people made it more difficult to count them, thus the 1990 estimates of coverage for this population were weak. The Bureau decided not to gather these data during 2000 A.C.E. data collection based in part on the difficulty of collecting and matching this information in the past, and in part as a result of a separate design decision to change the way to treat information for people who moved during the time period between the census and the coverage evaluation interviews. By excluding group quarters from the coverage evaluation sample, the Bureau had less information collected on a sample of this population that included some duplication, and the missing information may have enabled it to better detect and account for such duplication. In addition, by developing coverage error estimates that were not applicable to the group quarters population, the Bureau made the task of assessing the quality of the census as a whole more difficult.

Figure 2 also shows that another blind spot emerged as the Bureau increased the number of “imputed” records in the final census, though they could not be included in the A.C.E. sample survey. The Bureau estimates a certain number of individuals—called imputations—that they have reason to believe exist, despite the fact that they have no personal information on them, and adds records to the census (along with certain characteristics such as age and race/ethnicity) to account for them. For example, when the Bureau believes a household is occupied but does not have any information on the number of people living there, it will impute the number of people as well as their personal characteristics. The Bureau increased imputations in 2000 from about 2 million in 1990 to about 5.8 million records. Changes in census and coverage evaluation design from 1990 likely contributed to this increase. Specifically, the Bureau reduced the number of persons who could have their personal information recorded on the standard census form in 2000. In addition, the Bureau changed the way coverage evaluation accounted for people who moved between Census Day and the day of the coverage evaluation interview. These design changes resulted in less complete information on people and likely contributed to an increase in imputations. (These and other changes are explained in more detail in app. II.)

Because imputed records are simply added to the census totals and do not have names attached to them, it was impossible for A.C.E. to either count imputed individuals using the A.C.E. sample survey or incorporate them into the matching process. At any rate, since the true number and characteristics of these persons are unknown, matching these records via A.C.E. would not have provided meaningful information on coverage error. A.C.E. was designed to measure the net census coverage error, in essence the net effect of people counted more than once minus people missed, and included measurement of the effects of imputation on coverage error. The Bureau generalizes its estimates of coverage error to cover imputations and also maintains that its imputation methods do not introduce any statistical bias in population counts. But the formulas used by the Bureau to calculate its estimates of coverage error account for imputations by subtracting them from the census count being evaluated, not by measuring the error in them. And the Bureau did not attempt to determine the accuracy of the individual imputations, that is although the Bureau imputed persons they have reason to believe existed, the Bureau does not know whether it over- or underestimated such persons. As more imputations are included in the census total, the generalization of coverage error estimates to that total population becomes less reliable.

Similarly, the Bureau created an additional coverage error blind spot by including in the census 2.4 million records that it previously suspected were duplicates and thus were not included in the coverage evaluation. Prior to A.C.E. matching, the Bureau removed about 6 million persons from the census population, identifying them as likely duplicates. Then, after completing additional research on these possible duplicates, the Bureau decided to reinstate the records for 2.4 million of these persons it no longer suspected were duplicates. However, it did so after A.C.E. had completed matching and evaluating the census records from which the 2.4 million persons had been removed and for which coverage error estimation had begun (see fig. 3).

The Bureau documented in a memorandum that the exclusion of the records from A.C.E. processing was not statistically expected to affect A.C.E. results. However, later Bureau research concluded that over 1 million of these 2.4 million records were likely duplicates, none of which could have been detected by A.C.E. While the Bureau maintains that the reinstatement of the over 1 million reinstated likely duplicates did not affect the A.C.E. estimate in a statistically significant way, this suggests that the resulting A.C.E.-based estimate of national population itself is blind to the presence in the census of the over 1 million duplicates the Bureau reintroduced. For 2010, Bureau officials have chartered a planning group responsible for, among other things, proposing improvements to reduce duplication in the census, which may address some of the problem.

In addition to excluding populations from the scope of evaluation, the Bureau further curtailed its ability to measure coverage error by reducing A.C.E.’s search area to only one geographic ring around selected A.C.E. sample areas during the matching process. For the 1990 Census, the Bureau’s coverage evaluation program always searched at least one surrounding ring and an even larger ring in rural areas. However, in 1999, before a panel of the National Academy of Sciences, Bureau officials announced and defended the decision to not expand the search area except in targeted instances, saying that Bureau research indicated that the additional matches found in 1990 from the expanded search areas did not justify the additional effort. In its comments on our draft report, the Department writes that more important than the size of the search area is maintaining “balance”—i.e., search areas must be used consistently both to identify people who have been missed and to identify people who have been counted in error (including duplicates). The Department also justified the decision to reduce the search area in 2000 from 1990 in part by stating, “in an expected value sense, the reduced search area would have affected” the extra missed people and the extra miscounted people equally, or been balanced. However, later research discovered large numbers of the missed duplicates in the actual census by matching A.C.E. persons to census persons nationwide—far beyond the areas searched during A.C.E. matching. A 2001 Bureau report presenting the results of computer rematching of the A.C.E. sample concluded, “Our analysis found an additional 1.2 million duplicate enumerations in units that were out-of- scope for the A.C.E. but would have been in-scope for [1990 coverage evaluation].” In other words, if the Bureau had continued its 1990 practice of searching in housing units in larger geographic areas in 2000, the A.C.E. process might have identified more duplicates and yielded better results. The Bureau research cited above appears to question the decision to reduce the search areas. In fact, after the 2000 Census was completed, again before the National Academy of Sciences, a Bureau official suggested that any coverage evaluation methods for 2010 should conduct a more thorough search, perhaps expanding the search area to two or more geographic rings everywhere.

Bureau Has Not Fully Accounted for How Design Decisions Affected Coverage Error Estimates

This review has identified only some of the decisions that could have created problems in A.C.E. estimates. Because the Bureau has not attempted to account for how all of its design decisions relating to A.C.E. and the census affected the outcome of the program, the full range of reasons that A.C.E. estimates were not reliable remains obscure.

Bureau research documented and this report describes the magnitude of the direct effects of most of these design decisions in terms of the size of the census population affected, and the Bureau’s final reporting on the revised A.C.E. estimates mentions many design changes, but not together or comprehensively, and they do not explain how the changes might have affected the estimates of coverage error. Without clear documentation of how significant changes in the design of the census and A.C.E. might have affected the measurements of census accuracy, it is not apparent how problems that have arisen as a result of the Bureau’s own decisions can be distinguished from problems that are less under the Bureau’s control, i.e., difficulties inherent to conducting coverage evaluation. Thus the Bureau’s plans to measure the coverage error for the 2010 Census are not based on a full understanding of the relationship between the separate decisions it makes about how to conduct A.C.E. and the census and the resulting performance of its coverage measurement program. This in turn could hamper the Bureau’s efforts to craft a more successful coverage measurement program for the next national head count.

The Bureau Has Not Produced Reliable Revised Estimates of Coverage Error for the 2000 Census

While the Bureau produced a vast body of research regarding the census and A.C.E., including multiple reassessments and revisions of earlier work, the revised estimates are not reliable. The initial A.C.E. estimates of coverage error suggested that while historical patterns of differences in undercounts between demographic groups persisted, the Bureau had succeeded in 2000 in reducing the population undercounts of most minorities, and the subsequent revised estimates showed even greater success in reducing population undercounts in the Census. However, the large number of limitations described in the Bureau’s documentation of the methodology used to generate the revised estimates of coverage error suggest that these estimates are less reliable than reported and may not describe the true rate of coverage error. The Bureau, however, has not made the full impact of these methodological limitations on the data clear. Moreover, the final revised estimates of coverage error for the count of housing units and the count of people, which the Bureau expected to be similar if the estimates were reliable, differed, further raising questions about the reliability of the revised estimates.

Bureau’s Conclusion That A.C.E. Estimates Are Unreliable Is Based on Results That Are Unreliable Themselves

The Bureau undertook an extensive review of A.C.E.’s results over a 3-year period. In doing so, the Bureau revised its estimates of A.C.E. coverage error twice—first in October 2001 and again in March 2003. These revisions suggest that the original A.C.E. estimates were unreliable. Figure 4 illustrates how each of the revised A.C.E. estimates of coverage error reduced the undercount for most of the three major race/origin groups from the initial A.C.E. estimates. Note that the final revised estimate indicates that the total population was actually overcounted by one-half of 1 percent.

The differences in the revised estimates presumably provide a measure of the error in the original A.C.E. estimates. (The estimated net population undercounts—and their standard errors—for these groups are provided in app. III.) However, the revised estimates of coverage error may not be reliable enough themselves to provide an adequate basis for such a comparison to measure error in the original A.C.E. estimates.

First, the number of Bureau-documented limitations with respect to the methodologies used in generating A.C.E.’s revised estimates raises questions about the accuracy of the revised estimates. Within voluminous technical documentation of its process, the Bureau identified several methodological decisions wherein if the decisions had been made differently, they may have led to appreciably different results. Thus the methods the Bureau chose may have affected the estimates of census coverage error themselves and/or the measures of uncertainty associated with the estimates, limiting the general reliability of the revisions. The limitations in the methodologies for the revised estimates included the following: Errors in the demographic data used to revise estimates may have contributed to additional error in the estimates.

Errors stemming from the Bureau’s choice of model to resolve uncertain match cases were accounted for in initial March 2001 A.C.E. estimates but were not accounted for in the revised estimates in March 2003.

Alternative possible adjustments for known inefficiencies in computer matching algorithms would directly affect revised estimates.

The Bureau’s evaluations of the quality of clerical matching were used to revise the initial A.C.E. coverage estimates, leaving the Bureau less reliable means to measure the uncertainty in the revised estimates.

The Bureau Has Not Clearly Quantified and Reported the Full Impact of the Methodological Limitations on the Revised Estimates

For the earlier revision of coverage error estimates, the Bureau provided the range of impacts that could result from some different methodological decisions, enabling more informed judgments regarding the reliability of the data. For example, in support of its October 2001 decision to not adjust census data, the Bureau showed that different assumptions about how to treat coverage evaluation cases that the Bureau could not resolve could result in variations in the census count of about 6 million people. The Bureau also had reported previously the range of impacts on the estimates resulting from different assumptions and associated calculations to account for the inefficiency of computer matching. They found that different assumptions could result in estimates of census error differing by about 3.5 million people.

However, with the final revision of the A.C.E. coverage error estimates, the Bureau did not clearly provide the ranges of impact resulting from different methodological decisions. While the Bureau did discuss major limitations and indicated their uncertain impact on the revised estimates of coverage error, the Bureau’s primary document for reporting the latest estimates of coverage error did not report the possible quantitative impacts of all these limitations—either separately or together—on the estimates. Thus readers of the reported estimates do not have the information needed to accurately judge the overall reliability of the estimates, namely, the extent of the possible ranges of the estimates had different methodological decisions been made.

Sampling errors were reported alongside estimates of census error, but these do not adequately convey the extent of uncertainty associated with either the reported quantitative estimates themselves or the conclusions to be drawn from them. For example, the Bureau decided to make no adjustment to account for the limitation of computer matching efficiency when calculating its latest revision of estimates of coverage error, unlike the adjustment it made when calculating its earlier revised estimates. When justifying its adjustment made in its earlier revised estimates, the Bureau demonstrated that the choice of adjustment mattered to the calculated results. But the potential significance to the calculated results of the Bureau’s having made a different assumption was not reflected in the Bureau’s primary presentation of its estimates and their errors. The absence of clear documentation on the possible significant impacts of such assumptions could lead readers of the Bureau’s reporting to believe erroneously that all assumptions have been accounted for in the published statistics, or that the estimates of coverage error are more reliable than they are.

Unexpected Differences in Patterns of Coverage Error between the Housing and the Population Count Further Call into Question Reliability of Revised Estimates

According to Bureau reporting, when it examined the validity of the revised coverage error estimates the Bureau expected to see across demographic groups similar patterns between the coverage error for the count of the population and the count of housing. That is, if a population was overcounted or undercounted, then the count of housing units for that population was expected to be overcounted or undercounted as well. The Bureau developed estimates of coverage error in the count of housing units from A.C.E. data. But the comparisons of non-hispanic blacks and hispanics to non-hispanic whites in figure 5 shows that the relative housing undercounts are opposite of what was expected by the Bureau.

For example, the estimated population undercount for non-hispanic blacks is almost 3 percent greater than that of the majority group—non-hispanic white or other—but the estimated housing unit undercount for non-hispanic blacks is about 0.8 percent less than that of the majority group. In addition, while the Bureau estimated that the non-hispanic white majority group had a net population overcount of over 1 percent, the Bureau estimated the majority group as having its housing units undercounted by about one-third of a percent. (The estimated net housing unit undercounts—and their standard errors—for these groups are provided in app. III.)

Bureau officials told us that the problems A.C.E. and the census experienced with identifying coverage error in the population do not seem likely to have affected housing counts. However, when estimating coverage error for housing units for specific populations (e.g., by gender or race/ethnicity) errors in the population count can affect the reliability of housing tabulations. This is because when the Bureau tabulates housing data by characteristics like gender or race, it uses the personal characteristics of the person recorded as the head of the households living in each housing unit. So if there are problems with the Bureau’s count of population for demographic groups, for example by gender or sex, they will affect the count of housing units for demographic groups. While the unexpected patterns in population and housing unit coverage error may be reconcilable, Bureau officials do admit that problems with the estimations of population coverage error may also adversely affect the reliability of other measures of housing count accuracy they rely upon, such as vacancy rates. Bureau officials have indicated the need to review this carefully for 2010.

Bureau Reports Revised Estimates Resulted in Lessons Learned

While the multiple reassessments and revisions of earlier work did not result in reliable estimates, these efforts were not without value, according to the Bureau. Bureau officials stated that the revision process and results helped the Bureau focus for 2010 on detecting duplicates, revising residence rules, and improving the quality of enumeration data collected from sources outside the household, such as neighbors, as well as providing invaluable insights for its program of updating census population estimates throughout the decade.

Conclusions

The volume and accessibility over the Internet of the Bureau’s research may have made this the most transparent coverage evaluation exercise of a Decennial Census. However, as the Bureau has closed the book on Census 2000 and turned toward 2010, the reliability of the Bureau’s coverage estimates remains unknown. The Bureau made extensive efforts to evaluate the census and its coverage error estimates resulting from A.C.E., but these efforts have not been sufficient to provide reliable revised estimates of coverage error. So while much is known about operational performance of the 2000 Census, one of the key performance measures for the 2000 census remains unknown.

Moreover, neither Congress nor the public know why the coverage evaluation program did not work as intended, because the Bureau has not provided a clear accounting of how census and A.C.E. design decisions and/or limitations in the A.C.E. revision methodology discussed in this report accounted for the apparent weakness—or strengths—of A.C.E. Without such an accounting, the causes of problems and whether they can be addressed will remain obscure. And as the Bureau makes plans for coverage evaluation for the 2010 Census, whether that program approximates A.C.E.’s design or not, the Bureau will be missing valuable data that could help officials make better decisions about how to improve coverage evaluation.

Finally, this lack of information calls into question the Bureau’s claim (made in response to a prior GAO recommendation that the Bureau determine the feasibility of adjustment) that it has already established that using coverage evaluation for adjustment purposes is not feasible. Without clearly demonstrating what went wrong with its most recent coverage evaluation and why, the Bureau has not shown that coverage evaluation for the purpose of adjustment is not feasible. In fact, this report mentions two census improvements—to residence rules and to efforts to identify and reduce duplicates—that the Bureau is already considering that could make A.C.E. estimates more reliable, and perhaps even feasible. Furthermore, although the Bureau reports that its experience with revising A.C.E. estimates has provided lessons, it remains unclear how the Bureau will use its published coverage error estimates to make decisions leading to a more reliable measure of coverage error in 2010, or how the unreliable estimates can be of value to policymakers or the public.

Recommendations for Executive Action

As the Bureau plans for its coverage evaluation of the next national head count in 2010, we recommend that the Secretary of Commerce direct that the Bureau take the following three actions to ensure that coverage evaluation results the Bureau disseminates are as useful as possible to Congress and other census stakeholders: To avoid creating any unnecessary blind spots in the 2010 coverage evaluation, as the Bureau plans for its coverage evaluation in 2010, it should take into account how any significant future design decisions relating to census (for example, residence rules, efforts to detect and reduce duplicates, or other procedures) or A.C.E. (for example, scope of coverage, and changes in search areas, if applicable), or their interactions, could affect the accuracy of the program.

Furthermore, in the future, the Bureau should clearly report in its evaluation of A.C.E. how any significant changes in the design of census and/or A.C.E. might have affected the accuracy of the coverage error estimates.

In addition GAO recommends that in the future the Bureau plan to not only identify but report, where feasible, the potential range of impact of any significant methodological limitation on published census coverage error estimates. When the impact on accuracy is not readily quantifiable, the Bureau should include clear statements disclosing how it could potentially affect how people interpret the accuracy of the census or A.C.E.

Agency Comments and Our Evaluation

The Under Secretary for Economic Affairs at the Department of Commerce provided us written comments from the Department on a draft of this report on September 10, 2004 (see appendix I). The Department concurred with our recommendations, but took exception to some of our analyses and conclusions and provided additional related context and technical information. In several places, we have revised the final report to reflect the additional information and provided further clarification on our analyses.

The Department was concerned that our draft report implied that A.C.E. was inaccurate because it should have measured gross coverage error components, and that this was misleading because the Bureau designed A.C.E. to measure net coverage errors. While we have previously testified that the Bureau should measure gross error components, and the Department in its response states that this is now a Bureau goal for 2010, we clarified our report to reflect the fact that the Bureau designed A.C.E. to measure net coverage error.

Further, although the Department agreed with our finding that the Bureau used residence rules that were unable to capture the complexity of American society thus creating difficulty for coverage evaluation, the Department disagreed with our characterization of the role four other census and A.C.E. design decisions played in affecting coverage evaluation. Specifically, the Bureau does not believe that any of the following four design decisions contributed significantly to the inaccuracy of the A.C.E. results: 1. The Treatment of the Group Quarters Population—The Department commented that we correctly noted that group quarter residents were excluded from the A.C.E. universe who would have been within the scope of A.C.E. under the 1990 coverage evaluation design, and that a large number of these people were counted more than once in 2000. The Department maintains that the Bureau designed A.C.E. to measure such duplicates. We believe this is misleading. As the Department noted, during its follow-up at housing units the Bureau included questions intended to identify the possible duplication of college students living away at college, and we have now included this in our final report. But as we stated in our draft report, A.C.E. did not provide coverage error information for the national group quarters population. Moreover, during A.C.E. the Bureau did not search for duplicate people within the group quarters population counted by the census, as it did within housing units counted by the census. In fact, later Bureau research estimated that if it had done so, the Bureau would have identified over 600,000 additional duplicates there. As such, our finding that this may have contributed to the unreliability of coverage error estimates still stands. 2. The Treatment of Census Imputations—The Department stated that A.C.E. was designed to include the effects of imputations on its measurement of coverage error and that there was no basis for our draft report stating that as more imputations were included in the census then coverage error estimates became less reliable. While we agree that the Bureau’s estimates of coverage error accounted for the number of imputations, as we report, and as the Department’s response reiterated, no attempt was made to determine the accuracy of the imputations included in the census. Thus any errors in either the number or demographic characteristics of the population imputed by the Bureau were not known within the coverage error processes or estimation. As a result, in generalizing the coverage error estimates to the imputed segment of the population, the Bureau assumed that the imputed population had coverage error identical to the population for which coverage error was actually measured. Furthermore, the larger the imputed segment of the population became the more this assumption had to be relied upon. Since the real people underlying any imputations are not observed by the census, the assumption is, in its strictest sense, untestable, thus we maintain that increasing the number of imputations included in the census may have made generalizing the coverage error estimates to the total census population less reliable. 3. The Treatment of Duplicate Enumerations in the Reinstated Housing Units—The Department wrote that our draft report incorrectly characterized the effects of reinstating duplicates into the census. The Department indicated that A.C.E., having been designed to measure net coverage error, treated the over 1 million likely duplicates “exactly correctly” and that including them in the census had no effect on the mathematical estimates of coverage error produced by A.C.E. We reported that, according to Bureau research, introducing the additional duplicates into the census appeared to have no impact on the A.C.E. estimates. But we view this fact as evidence of a limitation, or blind spot, in the Bureau’s coverage evaluation. The fact that 2.4 million records, containing disproportionately over 1 million duplicate people could be added to the census without affecting the A.C.E. estimates demonstrates a practical limitation of those coverage error estimates. We maintain that the resultant measure of coverage error cannot be reliably generalized to the entire population count of which those 1 million duplicates are a part. 4. Size of the Search Area—The Department wrote that a search area like that used in 1990 would have done little to better measure the number of students and people with vacation homes who may have been duplicated in 2000. It described our conclusion regarding the reduction in search area from 1990 as not supported by the relative magnitudes of these situations. And finally, the Department offered additional support for the decision to reduce the search area by describing the reduced search area as balanced, or “in an expected value sense” [emphasis added] affecting the number of extra missed people and the extra miscounted people equally.

In our final report we added a statement about the Department’s concern over the importance of balance in its use of search areas. But we disagree that our conclusion is unsupported, since in our draft report we explicitly cited Bureau research that found an additional 1.2 million duplicate enumerations in units that were out-of-scope for 2000 A.C.E. but that would have been in-scope for 1990’s coverage evaluation.

In addition, the Department offered several other comments.

Regarding our finding that the Bureau has not produced reliable revised estimates of coverage error for the 2000 Census, and, specifically, that the full impact of the Bureau’s methodological limitations on the revised estimates has not been made clear, the Department wrote that the Census Bureau feels that further evaluations would not be a wise use of resources. We concur, which is why our recommendations look forward to the Bureau’s preparation for 2010.

The Department commented that it did not see how we could draw conclusions about the reliability of the Bureau’s coverage evaluation estimates if we did not audit the underlying research, data, or conclusions. We maintain that the objectives and scope of our review did not require such an audit. As we described, and at times cited, throughout our draft report, we used the results of the Bureau’s own assessment of the 2000 Census and its coverage evaluation. That information was sufficient to draw conclusions about the reliability of the A.C.E. estimates. As a result, there was no need to verify individual Bureau evaluations and methodologies.

The Department expressed concern that our draft report implied that the unexpected differences in patterns of coverage error between the housing and the population count were irreconcilable. That was not our intent, and we have clarified that in the report.

The Department expressed concern over the report’s characterization of the 1990 coverage error estimates for group quarters as weak in part due to the high mobility of this population. However, the 1990 group quarters estimates are described as “weak” in a Bureau memorandum proposing that group quarters be excluded from the 2000 coverage evaluation. The memorandum also explains how the mobility within the group quarters population contributes to the resulting estimates. We have not revised the characterization of the group quarters coverage error estimates or the causal link due to the mobility of that population, but we have revised our text to state more clearly that the 1990 estimates being discussed are those for group quarters.

As agreed with your offices, unless you release its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time we will send copies to other interested congressional committees, the Secretary of Commerce, and the Director of the U.S. Census Bureau. Copies will be made available to others upon request. This report will also be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me on (202) 512-6806 or by e-mail at daltonp@gao.gov or Robert Goldenkoff, Assistant Director, at (202) 512-2757 or goldenkoffr@gao.gov. Key contributors to this report were Ty Mitchell, Amy Rosewarne, and Elan Rohde.

Comments from the Department of Commerce

Bureau Design Decisions Increased Imputations

The Bureau made various design decisions that resulted in an increase in the number of “imputations”—or people guessed to exist—included in the census population that could not be included within the A.C.E. sample survey. The Bureau believes certain numbers of people exist despite the fact that the census records no personal information on them; thus it projects, via computer-executed algorithms, numbers and characteristics of people and includes them in the census. Such records are simply added to the census totals, and do not have names attached to them. Thus it was impossible for A.C.E. to either count imputed individuals using the A.C.E. sample survey or incorporate them into the matching process. Since the true number and the characteristics of these persons are unknown, matching nameless records via A.C.E. would not have provided any meaningful information on coverage evaluation. The number of people the Bureau imputed grew rapidly in 2000, from about 2 million in 1990 to about 5.

One of the reasons for the large increase in imputations may be the decision by the Bureau to eliminate field edits—the last-minute follow-up operation to collect additional information from mail-back forms that had too little information on them to continue processing—from field follow-up in 2000. While acknowledging that this decision may have increased imputations for 2000, a senior Bureau official justified the decision by describing the field edits in 1990 as providing at times a “clerical imputation” that introduced a subjective source of error, which computer- based imputation in 2000 lacked.

The Bureau also reduced the number of household members for whom personal information could be provided on standard census forms, and this also contributed to the increase in imputations. Households reporting a household size greater than 6 in 2000—the number for whom personal characteristics could be provided—were to be automatically contacted by the Bureau to collect the additional information. Yet not all large households could be reached for the additional information, and the personal characteristics of the remaining household members needed to be imputed. Again, A.C.E. would have been unable to match people counted by its sample survey to imputations, so imputed people were excluded from A.C.E. calculations of coverage errors.

An A.C.E. design choice by the Bureau that likely increased the amount of data imputed within the A.C.E. sample survey was how the Bureau decided to account for people who moved between Census Day and the day of the A.C.E. interview. Departing from how movers were dealt with in 1990, and partly to accommodate the initial design for the 2000 Census, which relied on sampling nonrespondents to the census, for 2000 the Bureau relied on the counts of the people moving into A.C.E. sample areas to estimate the number of matched people who had actually lived in the A.C.E. areas on Census Day but moved out. This decision resulted in the Bureau having less complete information about the Census Day residents in A.C.E. sample areas who had moved out, and likely increased the number of imputations that were later required, making it more difficult to match these moving persons to the census. A Bureau official also cited this decision as partial justification for not including group quarters in A.C.E. search areas.

The extent that imputation affected the accuracy of the census is unknown. The National Academy of Sciences discussed in an interim report on the 2000 Census the possibility of a subset of about 1.2 million of these imputations being duplicates. That report stated that, for example, “it is possible that some of these cases—perhaps a large proportion—were erroneous or duplicates,” and described another subset of about 2.3 million that could include duplicates. However, this Academy report did not include any data to suggest the extent of duplicates within these groups, and it may similarly have been possible for the number of persons in this group to have been underestimated. The Bureau maintains that the imputations were necessary to account for the people its field operations led it to believe had been missed, and that its imputation methods do not introduce statistical bias.

Bureau Estimates of Population and Housing Unit Net Undercounts

As shown in Table 1, the initial A.C.E. results suggested that the differential population undercounts of non-Hispanic blacks and Hispanics—the difference between their undercount estimate and that of the majority groups—persisted from Bureau estimates from its coverage evaluation in 1990. Yet they also demonstrated that the Bureau had apparently succeeded in reducing the magnitude of those differences since its evaluation of the 1990 Census.

Subsequent revised results published in October 2001 for three race/origin groups indicated that differential undercounts were generally lower than the initial A.C.E. estimates, but that only the undercount estimate for Hispanics was still statistically different from zero. Finally, the latest revised estimates of undercount reported in March 2003 that of these three major race/origin groups, only the non-Hispanic black and non-Hispanic white percentage undercounts were significantly different from zero, in addition to the national net overcount.

Unlike the estimates of census population accuracy, which were revised twice since initial estimates, the census housing count accuracy estimates have not been revised and are based on the initial A.C.E. data. A subset of those results, including those provided here, were also published in October 2001.

Glossary

This glossary is provided for reader convenience, not to provide authoritative or complete definitions.

Accuracy and Coverage Evaluation

The Bureau’s Accuracy and Coverage Evaluation (A.C.E.) program was intended to measure coverage error (see below) for the 2000 Decennial Census. The program was to enable the Bureau to more accurately estimate the rate of coverage error via a sample survey of select areas nationwide, and if warranted, to use the results of this survey to adjust census estimates of the population for nonapportionment purposes.

Census Adjustment

The use of statistical information to adjust official census data.

Count of Housing

A tally of certain kinds of dwellings, including single-family homes, apartments, and mobile homes, along with demographic information on the inhabitants.

Count of Population

The headcount of everybody in the nation, regardless of their dwelling.

Coverage Error

The extent that minority groups are over- or undercounted in comparison to other groups in the census.

Coverage Evaluation

Statistical studies to evaluate the level and sources of coverage error in censuses and surveys.

Duplicates

When the census erroneously counts a person more than once.

Residence Rules

The rules the Bureau uses to determine where people should be counted.

Related GAO Products

2010 Census: Cost and Design Issues Need to Be Addressed Soon. GAO-04-37. (Washington, D.C.: January 15, 2004). 2000 Census: Coverage Measurement Programs’ Results, Costs, and Lessons Learned. GAO-03-287. (Washington, D.C.: January 29, 2003). 2000 Census: Complete Costs of Coverage Evaluation Programs Are Not Available. GAO-03-41. (Washington, D.C.: October 31, 2002). 2000 Census: Coverage Evaluation Matching Implemented as Planned, but Census Bureau Should Evaluate Lessons Learned. GAO-02-297. (Washington, D.C.: March 14, 2002). 2000 Census: Coverage Evaluation Interviewing Overcame Challenges, but Further Research Needed. GAO-02-26. (Washington, D.C.: December 31, 2001).

GAO’s Mission

The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to www.gao.gov and select “Subscribe to Updates.”

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Congressional Relations

Public Affairs