2020 Census: Local Administrative Records and Their Use in the Challenge Program and Decennial
Released: 2013-03-25T00:00:00

Letter

Background

Administrative records are a growing source of information about individuals and households. Administrative records include records from government agencies, such as tax data and Medicare records, as well as commercial sources from major national data vendors. National administrative records refer to data compiled and maintained nationwide, including files compiled for the purpose of administering federal programs. In comparison, data compiled and maintained by municipalities are referred to as local administrative records, such as building permits and local tax records.

According to Bureau officials, for the 2020 Census, the Bureau is researching how to determine the quality and usefulness of administrative records for obtaining addresses or information about individuals. Administrative records could reduce the cost of the census if they can help the Bureau reduce the workload for several operations, including address list building; quality assurance; and nonresponse follow-up, which, at $1.6 billion in 2010 and lasting several weeks, was the largest and most costly census field operation.testing program has nine research and testing projects that are exploring the expanded use of administrative records for these purposes.

The Bureau’s 2020 research and The Bureau already uses administrative records to produce annual population estimates for the nation, states, counties, cities, towns, and townships as part of its program to estimate changes in population size and distribution since the previous census. The Bureau produces estimates at the state and county-level based on births, deaths, migration, and changes in the number of people who live in group quarters, such as college dormitories and nursing homes. Estimates of the population of subcounty communities—which consist of both incorporated places, such as cities, boroughs, and villages, and minor civil divisions, such as towns and townships—are primarily based on data on housing units, occupancy rates, and persons per household plus an estimate of the population in group quarters.

Local governments may challenge these population estimates through a process established by the Bureau’s Population Estimates Challenge Program. Local data sources for challenges have included building permits, non-permitted construction, demolition permits, non-permitted demolitions, certificates of occupancy, utility connection data, and real and personal property tax information on residential units.

The Bureau permitted localities to submit challenges for population estimates from 2001 to 2008. Counties and subcounties could challenge the Bureau’s estimate based on evidence that the number of housing units in their locality differed from the Bureau’s estimate of housing units. From 2001 to 2008, the Bureau reports revising population estimates from 287 challenges. These challenges were from 211 governments in 36 states and the District of Columbia. There were as few as 3 challenges in 2001 and as many as 61 each in 2006 and 2007. In 2010, the Bureau temporarily halted the challenge program beginning for estimates from 2009 to accommodate the 2010 decennial census. The Bureau is resuming the challenge program in 2013.

According to the Bureau, communities that challenged their population estimate saw their estimate revised upward by an average of about 9.4 percent over the 8-year period. These challenges ranged from an estimated increase of over 186 percent for Bluffton, South Carolina, in 2008, to a decrease of almost 18 percent for Winthrop, Massachusetts, during the same year. In total, four communities submitted challenges containing evidence that resulted in a decrease in population.

The effect of the program on smaller communities that participated was much larger, on average in percentage terms, than it was for larger communities that participated. According to Bureau data summarizing the program, communities with populations of less than 100,000 averaged an almost 13 percent upward revision in their population estimate—including the results of the challenge—compared to an average revision of less than 2 percent for communities with a population of more than 1 million (see table 1).

The Bureau Revised Rules Governing Data Sources and Procedures for the Challenge Program to Improve Accuracy of Population Estimates

The Bureau Changed the Challenge Program to Standardize Evidentiary Requirements and Limit the Scope

Previously, the Bureau provided communities that chose to challenge their population estimate with examples of the types of records they could use to support their calculations, including permits for new residential construction, public utility connection data, real and personal property tax information on residential units, and records of annexations and other types of legal boundary changes. In practice, according to agency officials, the Bureau generally accepted all challenges, largely without regard to the data sources provided so long as they supported calculations of population change and covered the reporting periods required by the challenge program. The rule changes are intended to improve the quality of data that communities use to challenge the Bureau’s population estimates and will affect the scope of what county and state governments can challenge. According to Bureau officials, these changes are based on research that shows that estimates based on some methods and records (e.g., births, deaths, and migration) are substantially more accurate than estimates based on others. The rules vary for different levels of government—subcounty, county or equivalent, and state.

Subcounty. Under the new rules, subcounty governments should use building, demolition, and mobile home permits, and group quarter counts to challenge population estimates. The Bureau plans to reject challenges that rely solely on other types of data, except in cases where the data provide overwhelming evidence that the Bureau’s estimate of population growth is in error.research shows that public utility records can vary widely in their reliability to indicate population growth, as well as in their availability to the public. Therefore, utility records will be treated as corroborating, or secondary, evidence to support other preferred data expected to be provided in support of challenges. Utility records were ordinarily accepted as a basis for successful challenges in the past.

For example, according to a senior Bureau official, County and equivalent. Under the changes, county governments and their equivalents should use birth and death records, immigration data, and group quarters counts to challenge population estimates—a significant reduction in the types of records from what the Bureau has historically accepted. According to Bureau officials, research conducted over the past several years has demonstrated that county-level estimation methods based on other data sources do not produce, on average, as accurate estimates of population. Consequently, the Bureau plans to reject county-level challenges relying on other data sources. Under the changes, the Bureau will only accept county-level challenges to either the accuracy of the data the Bureau itself used when producing estimates or to whether the Bureau carried out its estimation procedures properly, such as in handling data files properly or carrying out calculations. Bureau officials acknowledged that local governments might see this reduction in the scope of permissible county-level challenges as being too restrictive, as evidenced by some public comments submitted in response to the proposed changes. Bureau officials stated that they will continue their dialogue with representatives of the Federal-State Cooperative for Population Estimates—a partnership between the Bureau and state governments—and other researchers about improving methods of estimating population, and may revisit the structure of the challenge program if future research demonstrates that an alternative method of estimating county population consistently outperforms that used by the Bureau. Figure 1 shows changes in the types of administrative records accepted by the challenge program.

State. The Bureau will generally no longer permit state governments to submit their own challenges. Bureau officials stated that counties and cities are the most appropriate entities to submit a population challenge for their community because they have greater knowledge of their population. Bureau officials also said that they want to avoid situations where a state may challenge estimates for communities where respective local governments disagree with such challenges. The Bureau will allow states to submit challenges for counties or equivalents where there is no seat of government, such as in certain New England states and in parts of Alaska, but will otherwise require all communities affected by a challenge to have their government communicate directly with the Bureau.

Other Changes to Procedures are Intended to Improve Accuracy of Population Estimates

The changes to procedures are intended to improve the accuracy of revisions to population estimates stemming from challenges. The Bureau modified procedures so that challenges by subcounty governments to the Bureau’s estimates of people living in traditional housing—not living in group quarters arrangements—will no longer affect county-level population estimates. Previously, successful subcounty challenges were added to respective county-level populations. The Bureau justifies the change with its research demonstrating that its method for estimating the county population that lives in housing is generally better than the method typically used to challenge subcounty estimates. According to Bureau officials, the method used to challenge subcounty estimates introduced an upward bias for communities that were experiencing population decline or a slowing of population growth. According to the change, any challenge that results in an increase to the estimate of a subcounty community’s population living in traditional housing will be offset by a downward revision to the estimate for all other communities in the same county, in effect reallocating the estimated population within the county. Bureau officials emphasized that under the new procedures, as before, if a government successfully challenges the Bureau’s estimate of the population living in group quarters arrangements within its community, the revision will be added to the county population as well, and will not be offset by changes to populations in other communities in the county.

The Bureau also changed procedures so that it routinely reviews subcounty population challenges in light of each community’s population growth trend, requiring corroborating information when a government claims population growth that is inconsistent with the trend. For example, one local government in the Midwest successfully challenged the Bureau’s 2006 population estimate and received an increased population estimate of about 5.5 percent, even though evidence it submitted showed the number of housing units in the community was about 3.5 percent lower than the Bureau’s original calculation for that year. Another local government successfully challenged the Bureau’s 2007 population estimate and received an increased population estimate of about 7.4 percent, even though its evidence showed an increase in housing units of only about 0.8 percent. According to 2010 Census counts in both of these cases, the average actual annual population growth over the decade was far below the challenge result, and the trend in population was declining. A senior Bureau official with whom we spoke said that, in addition, this and other changes in procedures will reduce the incentive that some communities have had to file a challenge in order to provide a community only a temporary reprieve from an otherwise declining trend in their population.

The Bureau Reviews Quality of Calculations and Requires Local Governments to Certify that Data Are Accurate

Challenge program officials told us that the program focuses its quality assurance on (1) reviewing the calculations presented in the documentation submitted by local governments as part of challenge submissions and (2) checking documents and calculations for internal consistency. Specifically, different worksheets local governments could choose to complete under the program have different data and calculations required to support a challenge, which Bureau staff review for accuracy and consistency. For example, Bureau staff would identify when local governments provided data on local building permits with a significant lag time, and then revise the submitted calculations accordingly since some of the buildings would thus have been constructed outside the time frame for which the population change was being estimated.

Bureau officials told us that the Bureau assures quality of locally- submitted records by requiring a community’s highest elected official— such as a mayor or county commission chair—to certify the validity of data used in any challenge to a population estimate that its government submits. The Bureau takes such certification at “face value” and does not examine the quality of these records, which Bureau officials said would be prohibitively time consuming to investigate or verify. Bureau officials said that no change to that approach is planned for the future of the program. However, the Bureau is considering how to describe a quality threshold that local records should satisfy, or steps that local governments can take to check the quality of their records.

Additionally, the Bureau distributes a review guide containing standardized procedures to local governments that are interested in participating in the challenge program. The guide includes instructions on filling out standardized worksheets and descriptions of the types of administrative records that can be relied on as sources of data for the challenge. According to Bureau officials, this helps to ensure consistency across challenges and potential revisions to population estimates.

Moving forward, according to officials, the Bureau is preparing a quality assurance plan for Bureau staff who review challenges to better ensure proper handling and processing of challenges, as well as the review of calculations. The Bureau intends to develop the plan over the coming months, after the program has resumed. The Bureau is also undertaking an agency-wide effort to update its record keeping policies, which would include revising rules for retaining documentation submitted as part of the challenge program. Changes to record keeping policies would help the Bureau maintain challenge program documentation—including calculations of revised estimates—in the event the Bureau needs to perform any follow-up reviews.

In our review of the 287 challenges submitted to the Bureau resulting in revised population estimates from 2001 to 2008, we identified a number of cases where documentation of challenges appeared either incomplete or inconsistent. For example, in some cases, documentation of corrections to local government’s calculations was missing, as was certification from a community’s highest elected official that data submitted with a challenge were valid. Also missing were notices to local governments from the Bureau on whether challenges were accepted. A lack of documentation could make it difficult for an independent verification of the integrity of the program. We discussed these documentation issues with Bureau officials. In response, they explained the steps they were taking to address them, such as documenting a record keeping policy for the program that includes descriptions of what documentation to create for each challenge and checklists of specific items to be retained in files. Bureau officials agreed to provide us with copies of the revised record keeping policy when the challenge program is resumed later in 2013. Because of these planned actions, we are not making a recommendation to the Bureau at this time.

The Bureau has Limited Plans for Using Local Records to Supplement National Records for 2020

The Bureau is exploring the use of local records for the 2020 Census, but this effort will be of a lower priority than research on the use of national administrative records, in part because national records show greater promise than local records for controlling costs. The Bureau’s research on the use of national administrative records for the 2020 Census is focused on their possible use for such purposes as (1) building the address list, (2) counting people, and (3) quality assurance and evaluation processes. In contrast, Bureau officials believe that the use of local administrative records is most likely to support the development of the 2020 address list. The Bureau has used local census records in this regard in prior decennials.

In previous enumerations, the Bureau developed its address list in part by going door-to-door and canvassing every block in the country to verify street addresses. However, the process of going door-to-door is labor intensive. As a result, for 2020, the Bureau is exploring how to reduce much of the costly, national canvassing done in the past by combining both local and national administrative data with the United States Postal Service data it already uses, allowing the Bureau to continuously update its master address file throughout the decade. In particular, the Bureau is considering how it can more seamlessly integrate regular input from local government address lists and related geographic information systems into its existing address list and map development processes. The Bureau would like to update this information continuously rather than wait to receive the input as part of a one-time decennial update. In addition, the Bureau will research the quality of these updates to determine the extent to which it can rely on them without necessarily having to verify them door-to-door.

Bureau officials stated that if time and resources permit, the Bureau will consider other ways that local records can be used to supplement the use of national records. The officials stated that while it is important to continue research on local records, because they may be helpful in targeting decennial operations to hard-to-count groups or those in certain geographic areas, the results of 2010 Census research and testing on national records has led Bureau officials to conclude that continuing research on national records, such as those listed in figure 2, should be a higher priority. Bureau officials believe that research on the more broadly available national records could yield cost savings more quickly than research on locally available records, and that given resource constraints, they should attempt to “lock in” at least some of the more likely cost savings before pursuing more uncertain ones. Moreover, legislation may be needed to allow the Bureau to expand its use of national records for decennial purposes. Figure 2 shows the national and local records the Bureau is considering and the operations these records could support.

Beyond helping to develop the address list, local administrative records that have been used in the challenge program, such as school enrollment data, could supplement national records to either improve or evaluate the counting of targeted populations. Some Bureau stakeholders have suggested to the Bureau that these records, in some locations, may be more comprehensive and accurate than those records on which the Bureau already relies. For example, the Bureau is researching how statewide data, such as Supplemental Nutrition Assistance Program data from Illinois, Maryland, New York, and Texas, could enhance person and housing unit coverage—access to similar records is under negotiation with several other states. By matching records in these files with the 2010 Census and other administrative records, the Bureau is determining their suitability for use to identify addresses or people and their characteristics, as well as their accuracy. Additionally, the Bureau is considering research on other state-level files, such as those maintained by Bureau partners in Florida and Montana, which maintain records on utility use and residential construction, respectively.

However, the Bureau would face some challenges using these records for decennial purposes. Bureau officials said local records are generally more difficult to systematically access or to apply broadly to census operations. In some cases, laws restrict the use of these records to certain purposes, so the Bureau may need to negotiate their use with various parties or work towards legislative change. For example, going beyond school system enrollment data and obtaining access to student and school lunch data that might help target hard-to-reach populations is restricted to use for educational purposes by the Family Educational Additionally, some of these records are not Rights and Privacy Act.aggregated nationally, which could make it difficult for the Bureau to obtain them. According to the Bureau, in other cases such as the Supplemental Nutrition Assistance Program, not all states maintain records equally well. If the Bureau ultimately has the time and resources to supplement national data with local data, officials stated that they will work to address these issues.

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Commerce. In response, we received written comments from the department, which are reprinted in appendix II. In its comments, the department stated that it appreciates the time and effort that we put into the draft report and thanked us for responding to technical comments provided earlier by Bureau staff.

As arranged with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days after the date of this report. At that time, we will send copies of this report to the Acting Secretary of Commerce, the Under Secretary of Economic Affairs, the Acting Director of the U.S. Census Bureau, and interested congressional committees. The report also is available at no charge on GAO’s website at http://www.gao.gov.

If you have any questions about this report please contact me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. The GAO staff that made major contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

To describe the U.S. Census Bureau’s (Bureau) changes to how local administrative records will be used in the challenge program, we reviewed the August 10, 2012, Federal Register outlining proposed changes; the final rule, which was issued in the January 3, 2013, Federal Register; Bureau reports and presentations, which served as the basis for the changes; and comments received from state and local data experts solicited by the Federal Register on the proposed changes prior to the Bureau’s issuance of the final rule. We also interviewed current and retired officials from the Bureau’s Population Division responsible for implementing the challenge program.

To describe the changes to how the Bureau will assure the quality of population estimates updated by the challenge program, we reviewed the Federal Register notice outlined earlier, and we interviewed Bureau officials to identify additional quality assurance steps the Bureau intends to implement. To better understand the current procedures and implications of these changes, we reviewed documentation the Bureau provided to local governments on submitting accurate documentation to challenge population estimates, including worksheets used to calculate revised population estimates. We conducted a case file review to identify specific quality assurance steps the Bureau has previously taken to review the quality of submissions from local governments. During our case file review, we observed a number of instances where case files and other documentation of challenges appeared to be incomplete or inconsistent. We reviewed each of the documentation weaknesses, and we shared our observations with ranking Bureau executive and program managers.

To examine the Bureau’s plans to use the types of local administrative records currently used by the challenge program to improve the cost or quality of the 2020 Decennial Census, we reviewed Bureau documentation on research and testing of administrative records for the 2020 Census, as well as our prior reports on the Bureau’s research and testing efforts. Additionally, we interviewed the Bureau officials responsible for the research and testing efforts related to administrative records to understand which local records and processes from the challenge program the Bureau is considering for the 2020 Census.

Appendix II: Comments from the Department of Commerce

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

Other key contributors to this report include Ty Mitchell, Assistant Director; David Bobruff; Benjamin Crawford; Sara Daleski; Robert Gebhart; Will Holloway; Andrea Levine; Mark Ryan; and Timothy Wexler.