American Community Survey: Key Unresolved Issues
Released: 2004-11-08T00:00:00

Letter

Background

Now that the Census Bureau has congressional approval to begin the full ACS, data collection will begin in November 2004. The ACS test survey of a sample of 800,000 housing units, which has been conducted since 2000, will end in December 2004. The Bureau has been using this survey, known as the ACS Supplementary Survey, to test procedures and to produce annual data for geographic areas with populations of 250,000 or more. As one part of the test program, the supplementary survey data for 2000 have been compared with corresponding data from the 2000 Census long form to evaluate the quality of the ACS data and to provide users with information to make the transition from the long-form data to the full ACS data. According to the plan the Congress approved, the first annual ACS data for geographic areas with populations larger than 65,000 will be published beginning in 2006 with data for 2005; 3-year averages for geographic areas with populations between 20,000 and 65,000 will begin in 2008; and 5-year averages for geographic areas with populations smaller than 20,000, including Census tracts and block groups, will begin in 2010. The 5-year averages for 2008–12 will replace the 2010 Decennial Census long form for small geographic areas; they will be published in 2013 and will incorporate population and housing characteristics data from the 2010 Decennial Census short form.

In replacing the long form, the ACS will provide the same long-form data items at the same level of geographic area detail but in a more timely way. Whereas the long form provided small geographic detail once a decade, the ACS will provide annual estimates for large geographic areas and estimates for smaller areas in terms of 3-year or 5-year averages; the 5-year averages will provide data at the same geographic area level as the long form. According to the Census Bureau, these 5-year averages will be about as accurate as the long-form data; the annual and 3-year averages will be significantly less reliable than the long-form data but more reliable than existing annual household surveys the Census Bureau conducts.

In the remainder of the Background section of this report, we briefly describe the major differences between the ACS and the Decennial Census long form. We also discuss the Census Bureau’s outreach program, designed to involve stakeholders and users in shaping the ACS. Appendix III provides additional background information on the evolution of the ACS plan, appendix IV on the ACS testing and measurement program. Appendix II describes recent NAS findings on Continuous Measurement (CM) and the ACS.

Multiyear Averages

The 2000 Census long form used a decennial sample of about 19 million housing units; the full ACS will use an annual sample of 3 million housing units. In order to provide reliable estimates for geographic areas with populations of 65,000 or less, monthly ACS responses will be cumulated over several years—3 years for places with populations of 20,000 to 65,000 and 5 years for places with populations smaller than 20,000. Because of the statistical properties of these averages and users’ unfamiliarity with them, the Census Bureau has long recognized the need to provide guidance on such topics as the reliability of the averages for areas with rapidly changing population and the use of multiple estimates for states and other, larger geographic areas.

The Concept of Residence

For the 2000 Decennial Census, the ACS test programs, and federal household surveys, including the Current Population Survey (CPS), seasonal residents are recorded in a geographic area according to a concept of usual residence. As we noted above, under this concept, people who spend their winter in Florida and the rest of the year in New Hampshire, for example, are recorded as residents of New Hampshire; college students living away from home in dormitories are recorded as residents of the college.

For the full ACS, the Census Bureau has announced its decision to change the concept to current residence. According to the Census Bureau, although each concept requires that a person have only one residence at any point in time, current residence recognizes that the place of residence does not have to be the same throughout a year, allowing ACS data to more closely reflect the actual characteristics of each area. The Census Bureau plans to use current residence because the ACS is conducted every month and produces annual averages rather than point-in-time estimates, unlike the Decennial Census. Current residence is uniquely suited to the ACS, because it continuously collects information from independent monthly samples throughout all months of all years. Because the ACS is designed to produce a continuous measure of the characteristics of states, counties, and other places every year, the new residence rules were needed for seasonal and migratory individuals.

Reference Period

The underlying population and housing characteristics data for the 2000 Census long form were for April 1, 2000. For the ACS test program, the underlying population and housing characteristics varied. For all years except 2000, they were for July 1; for 2000, they were for April 1. For the full ACS, because the data are collected monthly, the reference period will be the average for the year, and the Census Bureau will assume this average is equivalent to data for July 1.

Independent Controls for Population and Housing Characteristics

The ACS will use population characteristics (age, sex, race, and ethnicity) and housing characteristics (occupied and vacant units) derived from an independent source and not from the results collected in the survey. Using independent controls for these characteristics is standard practice to correct sample survey results for the effects of nonresponse and undercoverage. Population and housing characteristics from the 2000 Census short form were used as independent controls for the 2000 Census long form, down to the tract level. For the ACS supplementary surveys, independent controls were from ICPE, which uses Decennial Census short-form data as benchmarks and administrative record data to interpolate between and extrapolate from the census benchmarks. ICPE develops and disseminates annual estimates of the total population and the distribution by age, sex, race, and Hispanic origin for the nation, state, counties, and functioning government units. ICPE provides annual estimates of population and housing characteristics at the county level, and for some subcounty levels, as of July 1, using the usual residence concept for seasonal residents.

Dollar-Denominated Data Items

According to current Census Bureau plans, annual estimates of dollar- denominated data items, such as income, rent, and housing-related expenses, will be presented after adjustment for inflation in order to facilitate comparisons over time. As in the ACS test programs, only annual estimates with this adjustment will be presented. The Census Bureau also has decided to continue to adjust annual data collected each month in the ACS to a calendar year basis. It will be using the Consumer Price Index (CPI) for the annual and monthly adjustments for all geographic areas.

Operational Differences

The long form and ACS will also differ in how operations are conducted, such as nonresponse follow-up and data capture. For the 2000 Census long form, nonresponse follow-up was conducted for all nonrespondents. For the ACS supplementary surveys and for the full ACS, nonresponse follow- up will be conducted for a sample of one-third of all nonrespondents. For the 2000 Census long form, all data items were entered using automated optical character recognition procedure; data from the ACS will be manually keyed.

Group Quarters

The ACS supplementary surveys excluded persons living in group quarters. Group quarters—which include nursing homes, prisons, college dormitories, military barracks, institutions for juveniles, and emergency and transitional shelters for the homeless—accounted for roughly 2.8 percent of the population in 2000. The Census Bureau decided not to cover these persons in the supplementary surveys, to avoid duplication with the 2000 Census, and because it lacked funding to cover them in subsequent years. Procedures for including in the ACS persons living in group quarters beginning with 2005 are discussed in the Census Bureau’s ACS Operations Plan, issued in March 2003. In addition, it has announced that it intends to continue testing procedures to improve the mailing list for group quarters to be used for the 2010 Decennial Census.

Outreach

The Census Bureau has long recognized the need to seek input from stakeholders and users in making decisions for all its programs. The Census Bureau sponsors technical reports that NAS prepares. (In appendix II, we summarize recent NAS reports on the ACS and related decennial censuses.) The Census Bureau has also held conferences on the ACS and has contracted with Westat Inc. to organize two conferences of experts on specific aspects of the ACS. Additionally, the Census Advisory Committees, which are Census Bureau–appointed advisory committees whose members represent professional associations such as the American Statistical Association (ASA) and the American Marketing Association, meet twice a year. The Census Bureau and other federal statistical agencies also participate in the quarterly meetings of the Council of Professional Associations on Federal Statistics, whose members include professional associations, businesses, research institutes, and others interested in federal statistics.

To obtain input from other federal agencies, the Office of Management and Budget (OMB) established an interagency advisory committee for the ACS in 2000. The committee’s major purpose was to coordinate the review of questions to be included in the ACS. Because of the committee’s limited focus, the Census Bureau established the ACS Federal Agency Information Program in 2003, responding to a recommendation we made. This program is designed to assist each federal agency that has a current or potential use for ACS data to achieve a smooth transition to using the ACS.

Outstanding Issues Jeopardize ACS’s Replacement of the Long Form

From its beginnings in the mid-1990s, the Census Bureau’s development plan for the ACS was designed to ensure that the ACS would satisfactorily replace the Decennial Census long form as the major source of small geographic area data. In our review of the plan, we found that the Census Bureau, as well as key ACS stakeholders, had for many years identified the key issues that needed to be resolved if the ACS were to reach this goal.

We have identified the following unresolved issues from our research (described in appendix I): the methodology to be used for deriving independent controls for population and housing characteristics with ACS definitions of place of residence and reference date, improvements needed to operational procedures, methods for valuation and presentation of dollar-denominated data items, comprehensive analysis of the comparability between new ACS data and corresponding data from the 2000 Census long-form and 2004 supplementary survey, and the provision of user guidance on multiyear averages.

Despite the Census Bureau’s early identification of issues critical to the successful replacement of the 2010 Decennial Census long form as the new source of small geographic area data, we found that its plans to resolve these issues have been only partially completed. Furthermore, we found that despite recent changes to the ACS implementation schedule, it is not fully synchronized with the Census Bureau’s time schedule for implementing the testing program for the 2010 Decennial Census. Consequently, if these issues are not resolved in a timely manner, the Census Bureau’s plan to replace the 2010 Decennial Census long form with the 2008–12 ACS averages for detailed geographic areas will be jeopardized.

A Methodology for Independent Controls for Population and Housing Characteristics Is Lacking

It is standard practice to use independent controls for population and housing characteristics to correct the results of sample surveys for the effects of nonresponse and undercoverage. For the 2000 Census long form, characteristics from the 2000 Census short form were used as independent controls down to the tract level. For the annual ACS supplementary surveys, characteristics from ICPE were used as the independent controls.

Independent controls for the full ACS will require a new methodology. Short-form data are available only once every 10 years, and the annual ICPE estimates do not provide data for the detailed geographic areas needed to prepare long-form detail and do not use the ACS residence concept or reference period. The new methodology is critical to the reliability of the ACS estimates of small geographic areas that ICPE does not provide and of areas that have large numbers of seasonal residents.

Census Bureau staff have long recognized the need for the new methodology. For example, a 1995 paper by Love, Dalzell, and Alexander expressed concern about population controls and residence rules as well as the need for consultation with users on these topics. They reported that the Census Bureau was planning to conduct research using data from the 1996 test sites to produce controls at the census tract and block group levels. They also noted that the Census Bureau would need to conduct research on the residence rule. A 2000 paper by Alexander and Wetrogan also discussed the issue of population controls. They reviewed possible methods for using ICPE to develop controls for the ACS and noted the need to consult with users on how to present information on the differences in ACS controls and ICPE in ACS publications.

Key stakeholders, including experts on the ACS we interviewed in August 2003 (listed in appendix I), expressed similar concerns about the methodology.

It appears that no progress had been made on a new methodology until the Census Bureau reported in October 2003 to its advisory committees on the status of a new methodology to derive controls. It announced that when full ACS collection starts in November 2004, (1) interim procedures would be used and (2) a final methodology would not be determined until after the necessary research was completed. The Census Bureau did not provide a date when the methodology would be incorporated.

In our review of Census Bureau presentations about the new methodology (described in detail in appendix V), we found that it had no plans to maintain time-series consistency of the population and housing controls by routinely incorporating the regular revisions to ICPE estimates into the ACS. Without such revisions, there could be a significant lack of comparability in the ACS data being averaged, and the reliability of multiyear estimates would be reduced. For example, without such revisions, the 2008–12 averages that are to replace the 2010 Decennial Census long form would be based on controls extrapolated from the 2000 Census for 2008–09 and controls from the 2010 Census for 2010–12. In addition, time-series consistency in the annual ACS data would be reduced, especially in the data for 2010 and previous years. Census Bureau officials told us that they were not planning any such revisions, unless the inconsistencies between 2010 ICPE and 2010 Census characteristics were significant, even though there were significant inconsistencies between the 2000 ICPE estimates and the 2000 Census data, especially for small geographic areas.

We found that regularly incorporating all revisions to ICPE into the ACS would improve ACS reliability and that planning would give users advance notice on the Census Bureau’s revision practice. The need for such planning is critical, as evidenced by the failure that occurred in January 2004, when a revised set of ICPE data was incorporated into the calculation of monthly CPS data on employment. Initially, the revised employment estimates were released without a revision of the pre-2004 data, resulting in a significant discontinuity between December 2003 and January 2004. As a result of users’ dissatisfaction about the discontinuity, a consistent set of employment estimates was released.

Finally, failure to adequately involve stakeholders in the decision process may contribute to significant misunderstanding about the use of the ACS estimates and corresponding estimates from the Decennial Census. In past decennial censuses, except for the very smallest geographic areas, the population and housing characteristics data published as part of the long- form detail were the same as the official data based on data collected on the short form. Because of differences in the residence and reference period concepts and the use of multiyear averages for small geographic areas, there will be less consistency between the ACS averages for 2008-12 and the 2010 Census data.

Operational Issues Have Not Been Addressed

The Census Bureau has identified operational issues with the ACS test programs, primarily from its evaluation studies on the 2000 Decennial Census and Census Bureau staff research papers on comparisons between data collected in the ACS 2000 Supplementary Survey and the 2000 Decennial Census long form. These issues (described in detail in appendix V) include problems with questionnaire design, nonresponse followup, and data capture, as well as coverage of persons living in group quarters.

For example, the Census Bureau conducted a study to evaluate the design of the ACS questions that are needed to implement the residence concept and reference period for the ACS. The study suggested that additional testing was needed for the questions about multiple residences and noted “that asking these questions on a person basis may produce different and probably better data than asking them on a household basis.” Similarly, the authors found potential problems with the identification of seasonal residents. We were not able to identify in the Census Bureau’s plans whether these issues would be addressed before implementation of the full ACS.

We also found, for the implementation of the full ACS for 2005, that the Census Bureau had addressed only the inclusion of group quarters and that it may not resolve the issue of questionnaire design until 2010. In addition, even for group quarters, it is planning for improvements that may not be included until 2010. Furthermore, not all problems have been identified because of the delays in the Census Bureau’s completing the evaluation studies of comparisons of long-form and ACS data items. Moreover, the Census Bureau’s plans do not provide for external consultations on key decisions about resolving issues.

Although the Census Bureau has acknowledged the importance of the timing of incorporating changes to resolve the various issues, any delay in implementing solutions to 2010 would not meet the needs of the ACS collection and production schedule. For example, in its March 2003 ACS operations plan, the Census Bureau recognized the need for maintaining questionnaire continuity to calculate consistent multiyear averages. It also has reported that it needs to incorporate changes in the ACS questionnaire no later than 2008 because changes introduced after 2008 and before 2013 would create inconsistencies in calculating the 5-year averages that are to replace the 2010 Decennial Census long form. Nevertheless, we found that the Census Bureau’s current time schedule does not call for resolving issues such as questionnaire design before 2008.

Incorporating changes into the ACS beginning with 2008 will help maintain the reliability of the 5-year averages for small geographic areas; failing to incorporate them beginning with 2005 will reduce the reliability of the annual changes in the ACS data.

With regard to external consultation, we found that the Census Bureau’s plans do not include time for consulting with stakeholders and users, despite NAS, BLS, and Census Advisory Committee suggestions and recommendations. For example, in a February 15, 2001, report to the Census Bureau, the NAS Panel on Research on Future Census Methods recommended that it conduct evaluation studies on “the effectiveness of operations used to designate special places and enumerate the group quarters and homeless populations.” Members of the Census Advisory Committee had raised similar concerns. In a 2003 report prepared for BLS, their consultant had made a number of recommendations about the questions on employment.

We found that the Census Bureau needs to develop a time schedule so that changes can be introduced to minimize inconsistencies between the 2005 and subsequent ACS data and to ensure that all necessary changes are made so that the ACS data for 2008–12 that will replace the 2010 Decennial Census long form will be collected consistently. In addition, the prompt completion of the ACS—long-form comparison studies and related evaluations will provide sufficient time for the Census Bureau to consult with stakeholders and to provide users with the information they need to understand the effect of making changes to the ACS questionnaires or procedures between 2005 and 2008.

Plans for Valuation and Presentation of Dollar- Denominated Data Items Are Questionable

When the Census Bureau began releasing data from the ACS test programs, all dollar-denominated items such as incomes, housing values, rents, and housing-related expenditures were adjusted for inflation. As in the ACS test programs, only annual estimates with this adjustment will be presented, and when the Census Bureau releases ACS data for each new year, it revises all dollar-denominated data for prior years. It makes a similar inflation adjustment for the annual income data collected in the CPS, but it releases the unadjusted estimates. The Census Bureau also has decided to continue to adjust annual data collected each month in the ACS to a calendar year basis. It will be using the CPI for the annual and monthly adjustments for all geographic areas.

The treatment of dollar-denominated data items is critical to all users of these data. It is particularly critical for federal agencies that will be using the ACS instead of the long form for many government programs to determine the allocation of funds or program eligibility. It is also critical to users of dollar-denominated items for small geographic areas because the inflation adjustments under the current procedure are based on a national average index.

In our review of the development and implementation of the ACS, we identified questions on the appropriateness of the methodology for the adjustment and the suppression of the unadjusted annual values. A report prepared for HUD found problems with the calculation of the adjustment and the use of the adjustment for income measures used for HUD programs. The report also noted that the lack of the unadjusted annual data would severely limit HUD’s use of calculations appropriate to its program needs. Research by Census Bureau staff questioned the adjustment for incomes when they found that it was a probable source of difference between income data from the supplementary survey and corresponding data from the CPS and the 2000 Census long form. (We discuss these findings in detail in appendix V.)

Our statisticians reviewed these findings and found a similar problem with the calculation of the adjustment because of the lack of trending adjustment. We found that the Census Bureau could estimate calendar year values using a combination of past trends in related series, information from other ACS respondents, or known information such as changes in cost-of-living adjustments for various transfer payment programs and changes in wage rates. We also found that converting ACS data from monthly to calendar year data is similar to conversion issues faced by other agencies that collect annual statistics compiled on a fiscal- year basis and that the procedures these agencies use could be adapted for the ACS. With regard to the use of a national cost-of-living adjustment, we have previously reported that for purposes such as allocating federal funds to states using income and poverty data, the CPI, a national measure of inflation, does not reflect variations in geographic areas. Census Bureau staff have reported similar findings.

The HUD and Census Bureau findings and our review raise serious questions about the inflation adjustments. We found no documentation explaining the rationale for the adjustment for either the ACS or the CPS, where its use is limited to income data. Bureau officials informed us that alternative procedures had not been examined and that stakeholders or users had not been consulted on the adjustment.

Evaluations of Comparisons Are Incomplete and Users Lack Information on ACS Time- Series Consistency

We noted above that one of the Census Bureau’s major justifications for the ACS test programs has been its comparing data collected in these programs, and corresponding data from the 2000 Decennial Census short and long forms, to identify operational problems. Another major justification for the ACS test programs has been the use of these comparisons, and comparisons with corresponding data from the CPS, to inform users in making the transition from the 2000 long form to the ACS. “These data will also contribute to a comparison with data from Census 2000 that is necessary because there are differences in methods and definitions between the census and the ACS. Moreover, decision makers will want to compare an area’s data to those from Census 2000. Comparisons using data from the operational test and from the 31 sites are essential to determine how much measured change between Census 2000 and future years of the ACS is real and how much is due to operational differences between the ACS and the census.”

Despite acknowledging the importance of these comparisons, the Census Bureau’s publication of evaluations of the comparisons has been delayed, and their scope has been reduced in terms of levels, data items, and time period. The lack of information will create problems for ACS users who will be comparing the annual ACS data for 2005 (to be released in mid- 2006) with 2000 Decennial Census data or comparing annual ACS supplementary survey data beginning with 2000.

In addition to delaying the release of the evaluation studies, the Census Bureau has reduced their scope. For the evaluations of ACS test site data, local experts did not participate in the evaluation of the comparisons for 27 of the 31 test sites. For the 4 test sites that were studied by local experts, they did not cover subcounty local government units. For evaluations of ACS supplementary survey data, the Census Bureau has eliminated the analyses of comparisons between (1) the 2000 supplementary survey and the 2000 long form for geographic areas with populations of 250,000 or more and (2) the supplementary surveys for 2000–02 to corresponding data from the CPS. It has reduced the scope of its evaluation studies by also eliminating comparisons of single-year estimates for most subnational areas and comparisons of data items such as financial characteristics of housing.

NAS found that the Census Bureau has not placed sufficient priority on completing the necessary evaluation studies. Furthermore, we found that the Census Bureau does not have a plan that includes the timely completion of all the studies. Once the studies are complete, it will need to incorporate the findings into ACS operations, consult with stakeholders, and provide users with the information they need to make the transition from the long form to the ACS. The plan will be needed to ensure that as many changes as possible can be introduced before the first annual ACS estimates are published in 2006 and that all necessary changes are implemented before 2008.

We found that the delays in completing the evaluations and their reduction in scope are likely to affect the use of the ACS in improving the small geographic area estimates of unemployment and poverty. For example, Labor uses the unemployment data extensively to administer a variety of federal programs. Several other departments use the poverty rates for similar purposes.

Users Are Not Informed on Key Properties of Multiyear Averages

One of the major differences between the ACS and the long form is that the ACS will provide data for geographic areas with populations smaller than 65,000 in terms of multiyear averages. Experts outside and inside the Census Bureau have identified serious issues regarding the statistical properties of multiyear averages and have recommended that the Census Bureau provide guidance to federal agencies and others on their use. We found that stakeholders have urged the Census Bureau for many years to provide guidance on the strengths and weaknesses of these averages. The most recent request for guidance on using multiyear averages came in the July 2003 report by the NAS Panel on Research on Future Census Methods: “The Census Bureau should issue a user’s guide that details the statistical implications of the difference between point-in-time and moving average estimates for various uses.” In the report’s executive summary, the panel also stated that “The Census Bureau must do significant work in informing data users and stakeholders of the features and the problems of working with moving average-based estimates.” It also expressed particular concern about the use of the multiyear (or moving) averages in fund allocation formulas.

Stakeholders have requested guidance on topics such as (1) the reliability of multiyear averages for areas with rapidly changing populations, (2) the reliability of trends calculated from annual changes in multiyear averages, and (3) the selection of ACS data for geographic areas with populations larger than 20,000 for which there will be multiple estimates. The Census Bureau has recognized the need for such guidance but has not announced any information about its contents or when it might be available, even though the guidance is needed well in advance of the release of the first multiyear averages in 2008.

We also found that plans for research to evaluate the statistical properties of multiyear averages are limited. The contracts to evaluate 3-year averages for the ACS test sites cover only averages for 1999–2001, with no comparisons with averages for 2000–02, 2001–03, or 1999–2003. In addition, the evaluation studies discussed earlier lack any time-series dimension, such as comparisons of the supplementary surveys with annual data from the CPS. Thus, it appears that the Census Bureau has missed the opportunity to test (1) distortion and stability in multiyear averages, (2) differences between multiple estimates for the same geographic areas, and (3) the use of annual ACS data for small geographic areas.

Meaningful External Consultation on Key Issues Is Needed

We found that in recent years, the Census Bureau has used its outreach efforts with stakeholders and users primarily to gain support for the ACS. Although it also has solicited advice from NAS panels, advisory committee members, and experts at workshops and conferences on some of the issues we have identified in this report, there is no indication that the Census Bureau will be following this advice. (For additional information, see appendix V.) Likewise, it has not yet followed similar advice from us, other government agencies, or even its own staff.

It has been more than a year since the Census Bureau announced, in March 2003, that it was looking into establishing an ACS partnership program that would involve advisory groups and expert panels to help it improve the program. We found that no such program has been established yet. Given that many key issues remain unresolved and that the Census Bureau has no plans to seek advice on resolving them, key aspects of the ACS will receive little or no input unless the Census Bureau revises its plans.

Improving Timeliness and Quality of Small Geographic Area Data Would Increase Costs

In 1994, the Congress began to fund testing of the survey to replace the 2000 Decennial Census long form, beginning with the 2000 Census. In reviewing the development of the ACS, we found that the Census Bureau was planning to replace the 2000 long form by starting the ACS program with an annual sample of 4.8 million housing units for 1999, 2000, and 2001 and reducing the sample for subsequent years to 3 million. The larger sample would have provided 3-year averages for all small geographic areas for 2000 and would have provided data for the smallest geographic areas of the same quality as the traditional long form. In fiscal year 1998, plans to introduce the ACS to replace the census long form were delayed until after the 2000 Census was completed.

When the Census Bureau submitted its plans in 1998 to replace the long form for the 2010 Decennial Census, a similar increase in sample size for 2009–11 was not proposed. Thus, compared with the plans for 2000, data for small geographic areas for 2010 would be delayed by a year and would be based on 5-year averages. When we reviewed the previous plan and other alternatives to the proposed ACS that would provide more timely and reliable data for small geographic areas, we determined that the only viable alternative to the current plans would be to expand the sample size for 2009–11, as proposed earlier.

This expansion would allow the Census Bureau to publish data for geographic areas with populations smaller than 20,000 a year earlier, and it would provide more reliable small- area data than under the currently planned 5-year averages. In addition, if the Congress were to provide the additional funds for this alternative, an additional year would be available for the Census Bureau to resolve issues we have identified in this report by giving it until the collection of data for 2009 rather than for 2008. According to Census Bureau estimates, increasing the sample size for the 3 years would add about $250 million to the estimated $500 million cost for the 3 years, using the smaller sample.

Resolving Outstanding Issues Needs a Time Schedule

The most recent Census Bureau schedule for implementing the ACS over the complete cycle of the 2010 Decennial Census was prepared in December 2003. Except for the completion of the questionnaire for the 2008 ACS, the milestones do not cover the resolution of issues that it has already identified and issues we identify in this report. (See table 1.) Ideally, all these issues should be resolved before the first annual results of the full ACS sample are released. However, the Census Bureau has already announced that final plans for calculating independent population and housing controls with ACS residence and reference concepts will not be available for several years, the 2004 test plans for the 2010 Decennial Census will cover group quarters and residence rules, reports from the 2004 tests will not be completed until 2005, and the 2006 test plans for 2010 also cover group quarters.

In addition, the Census Bureau has announced that comparisons of 2000 ACS and 2000 Census long-form data critical to the transition to the full ACS will be limited. Nevertheless, users who need the evaluation of these comparisons to compare data from the 2000 Decennial Census long form with data from the new ACS data or from the ACS supplementary surveys would benefit from the early resolution of other issues. For example, resolving issues before the release of the first 3-year averages (2005–07) would improve the consistency between these averages and the subsequent ACS data. Resolving all issues for the 2008 ACS is critical if these data are to be fully consistent with the ACS data for 2009–12 and the 2008–12 averages are to be fully consistent with the 2010 Decennial Census short-form data. As we noted above, the Census Bureau’s schedule does call for timely completion of the 2008 questionnaire. However, if questions to be included in the 2010 Census short form are changed during the congressional and OMB approval processes, currently scheduled for 2008 and 2009, data collected on the 2010 Census short form will be inconsistent with the ACS data.

Conclusions

The Census Bureau’s development of the ACS goes back several decades and has included intensive research and field testing programs, as well as substantial outreach efforts, in particular through the reports and workshops at NAS. However, its current plan to begin full implementation of the ACS for 2005 has several critical deficiencies. The Census Bureau has not completed its testing program, and it has not acted to resolve key issues already identified by the ACS test program, by evaluation studies of the 2000 Decennial Census, by Census Bureau research studies, and by stakeholders and users, including us, NAS, and other federal agencies. Furthermore, the ACS implementation plan and the 2010 Decennial Census test programs are not synchronized, and there is no comprehensive program for external consultation on the resolution of these issues. Without prompt resolution of issues such as those relating to the calculation of independent controls for small geographic areas and the consistency of data used to calculate multiyear averages, the ACS will not be an adequate replacement for the long form in the 2010 Decennial Census. If the Census Bureau is not able to use the ACS to replace the long form, the Congress and other stakeholders need to be advised in 2005 in order to allow for the Census Bureau time to reinstate the long form for the 2010 Census.

Recommendations for Executive Action

To ensure that the ACS is an adequate replacement for the Decennial Census long form, we recommend that the Secretary of Commerce direct the Census Bureau to (1) revise the ACS evaluation and testing plan and focus on the issues we have identified in this report; (2) provide key stakeholders, such as the National Academy of Sciences, with meaningful and timely input on decisions relating to these issues; and (3) make public the information underlying the Census Bureau’s decisions on these issues when it makes the decisions. We also recommend that the Secretary direct the Census Bureau to prepare a time schedule for the 2010 Decennial Census that provides for resolving these issues by incorporating all operational and programmatic changes into the 2008 ACS so that the 5-year averages for 2008–12 will adequately replace the 2010 Decennial Census long-form data for small geographic areas. These revisions should be reflected in the single, comprehensive project plan for the 2010 Census, as we have previously recommended.

Agency Comments and Our Evaluation

In written comments on a draft of this report, the Secretary of Commerce provided comments on our recommendations. (The Secretary’s comments are reprinted in appendix VI.) He disagreed with our recommendation that the ACS evaluation and testing plan needed to be revised to focus on issues we have identified in this report, stating that the current ACS testing and evaluation plan already included these issues. In following up on the Secretary's response, we learned that there is not yet a written plan, but only a rough outline of the types of work planned. Therefore, we believe our recommendation remains valid. The Secretary did not accept our recommendation to provide key stakeholders more direct and timely input into decisions on these issues because he believes that the present consultation process is adequate. We disagree, because as noted in Appendix II of our report, the Census Bureau has not been responsive to recommendations from several National Academy of Sciences reports relating to the ACS. The Secretary agreed with the recommendation that the Census Bureau provide public documentation for key decisions on issues we have identified in this report. The Secretary did not respond directly to our recommendation that he direct the Census Bureau to prepare a schedule for the 2010 Census that ensures that all necessary changes are made in time for the 2008 ACS so the 5-year ACS averages for 2008-2012 will be an adequate replacement for the 2010 long form for small geographic areas.

The Secretary provided comments on the five major outstanding issues that, in our view, jeopardize the ACS as a replacement of the long form: lack of methodology for independent controls, operational issues not addressed, questionable plans for dollar-denominated items, incomplete evaluations and lack of information on ACS time-series consistency, and lack of information about multiyear averages.

The Secretary disagreed with our findings about the lack of a methodology for independent population and housing controls. He stated that a methodology for the ACS was already in place. On the issue that changes to that methodology are needed to account for the difference in the ACS residence concept, the Secretary agreed that a change was needed but stated that it could be delayed for several more years. On the issue of independent controls for subcounty areas, he stated that the Census Bureau had no plans to develop such controls, which we found were used for the 2000 Census long form, but that it might develop such controls using data from the ACS or administrative records. However, he did not respond to our findings about the use of existing subcounty area data from the ICPE or from the 2010 Census short form. The Secretary stated that the Census Bureau also had no plans to revise the ICPE. On the issue of the ACS reference period, the Secretary reported that the Census Bureau had recently decided to assume that July 1 would be used as the reference period. The Secretary did not comment on GAO’s findings about the lack of plans to incorporate into the ACS 2010 Census data and related revisions to the ICPE estimates for previous years.

We disagree with the Secretary’s comments about the independent subcounty population and housing controls and believe that their use in the ACS is needed for the ACS to be an adequate replacement for the 2010 Census long form for small geographical areas. We found that independent controls from the 2000 Census short form were used for detailed geographic areas for the 2000 Census long form and that differences in counts of population and housing (occupied and vacant) between the long form and the short form were limited to the smallest geographic areas. The similar use of 2010 Census short-form counts in the ACS also would minimize differences in these counts from the ACS and the 2010 Census. Consequently, we disagree with the Bureau’s plan not to commit to the development of subcounty controls and its plans not to base these controls on ICPE total population and housing estimates, which are prepared annually for all general government units, and the more detailed and reliable data from the 2010 Census short form.

We also disagree with the Secretary that the implementation of a new methodology for independent controls with subcounty controls and the new residence concept can wait until 2008. As we noted in our report, we found that controls for subcounty areas with population of more than 65,000 will be needed before the 2005 ACS estimates are released for these areas in 2006 and that controls for subcounty areas with populations of more than 20,000 will be needed before the first multiyear averages are released in 2008. (For the 2000 Census long form, controls for most areas of this size were from the 2000 Census short form.) With regard to the new residence concept, a decision to delay introducing a new methodology until 2008 would create time-series inconsistencies between 2000–2007 and 2008 and subsequent years. These inconsistencies could be very significant for geographic areas with a large population of seasonal residents.

The Secretary did not comment on our findings about the need for a methodology to revisions relating to the ICPE into the ACS. We found that this methodology, which is important to both the time-series consistency of the annual ACS estimates and to the multiyear averages, is not covered by the current ACS methodology, but that it will be needed when the 2010 Decennial Census short form data become available. We found that it has been the Census Bureau’s practice for the ICPE, whose estimates are used as the independent controls for the ACS, to be benchmarked to the decennial census short-form data and that it uses similar practices for many other Census Bureau programs. For the ICPE, the Bureau will replace the 2010 ICPE estimates with the 2010 Census data, and use the differences in these estimates to revise the ICPE estimates back to the previous benchmark year, which for 2010 will be 2001. (Table 4 of our report shows the impact of benchmarking on county population data for 2000.) It should be noted that we found that this practice is not followed in all Census Bureau programs. For example, for the Annual Economic and Social Supplement to the CPS, the Census Bureau introduced the benchmark information from the 2000 Decennial Census into the 2001 estimates and presented the data on both the old and the revised basis. This approach, to present estimates on an old and new basis for a single year, may be appropriate for an annual survey. However, GAO found that because of the use of multiyear averages in the ACS, it is imperative that the ACS estimates for all years beginning with 2001 be revised. Without such a revision program, ACS estimates for 2010, which we assume will not be released until the 2010 Census short-form data have been incorporated, will be inconsistent with the 2009 estimates. In addition, the ACS estimates for 2008 and 2009 used to calculate the 5-year averages that will replace the 2010 Census long form will be based on controls that are inconsistent with those for 2010–12. Based on the revisions for 2000 shown in our report, there could be many significant inconsistencies, especially for small geographic areas.

Although the Secretary did not comment on the issue of revision, in its technical comments on our draft report, the Census Bureau reported (comment 22) that with regard to incorporating 2010 Census data, it has decided “to make appropriate changes to the population controls when necessary, including the possibility of reweighting the data around the 2010 time period and for all multiyear estimates.” We disagree with the Census Bureau’s approach primarily because it is not consistent with the practices used by the Census Bureau to incorporate census data into surveys and programs such as the ICPE and monthly retail sales that are controlled or benchmarked to a census or similar data set. For these surveys, it revises all previously published data on a predetermined schedule using a transparent statistical procedure. Most important, these procedures do not depend on the size of revisions, which can only be determined after a benchmark is completed. Regardless of the benchmarking procedures adopted for the ACS, we believe that the Census Bureau needs to have extensive consultation with external stakeholders to make its decision. In addition, because of the complexity of most benchmarking procedures, the Census Bureau needs to begin this consultation as soon as possible.

With regard to the recent Census Bureau decision about the reference period for the ACS, we are pleased that a decision has been made because any delay in this decision would have resulted in additional time-series inconsistencies in the ACS. We have changed our report to reflect this decision. Unfortunately, we have no documentation on the research underlying the decision and, as has been the case in other key decisions, we do not believe that there was any public discussion of this decision.

The second issue identified in our report related to the operational aspects of the ACS, including questionnaire design and the collection of data for persons living in group quarters. On these issues, the Secretary limited his comments to the questionnaires and addressed our findings that improvements identified as part of the 2000 Census cognitive testing research and research based on comparisons of ACS and 2000 Census long-form data would not be completed until 2008. The Secretary noted that the Census Bureau has resolved the issue of finalizing the ACS questions, including the questions to be asked on the 2010 Census short form before 2008. Although this recent decision appears to have resolved the scheduling issue, we believe that uncertainties remain as to whether this schedule can be met. For example, the ACS milestones in the latest available schedule call for final approval of the questions by the Congress and by OMB in 2008 and 2009, respectively, so that any changes made as a result of these steps would not be incorporated into the 2008 questions.

As the Census Bureau has recognized, failure to maintain consistency in the questions for the 2008-2012 ACS will result in inconsistencies in the 5- year averages centered on 2010, which are the averages designed to provide the small geographic area data that would have been collected on the 2010 Census long form. In addition, the recently released ACS evaluation reports identify issues on which new research is necessary, including the issues with the questions on disability identified in our report, but the Census Bureau has not indicated its plan to complete this additional research or to consult with stakeholders about decisions related to the research. Although the Secretary did not comment on our findings with regard to group quarters, we remain concerned that the work on group quarters being conducted as part of the 2004, 2006, and 2008 tests for the 2010 Census will not be reflected in the ACS beginning with 2008.

Our report also identified as unresolved issues the two inflation adjustments that the Census Bureau is applying on all dollar-denominated ACS items. The first adjustment is used to convert annual data collected each month in the ACS to a calendar year basis. This adjustment recognizes that the annual data collected in the ACS are for different periods because the data are collected monthly and cover the previous 12 months. The second adjustment is used to present dollar-denominated items in dollars of the most recent calendar year. This adjustment eliminates the impact of inflation when comparing data across years. The index used for both adjustments is the national-level CPI. The Secretary correctly observed that the CPI is a generally accepted measure of inflation and that most federal programs that allocate funds do not use regional measures of inflation. However, these observations did not directly address GAO’s findings about the adjustments or the concerns raised by HUD in its report on future use of the ACS, which are discussed in appendix V of our report. For example, the Secretary did not address our finding about a lack of a rationale for adjusting items other than incomes for changes in overall inflation rather than adjusting with indexes, such as wage rates or rent, that are directly related to the item being adjusted. He did indicate that the Census Bureau would reconsider its present policy of showing only the inflation-adjusted annual estimates and multiyear averages. We believe our findings about the need for the Census Bureau to provide a comprehensive rationale for the two adjustments still apply.

The Secretary disagreed with the issue we identified on completeness of the Census Bureau’s comparison and evaluation reports. He noted that after our draft report was completed, the Census Bureau released seven additional comparison reports and that it planned to prepare additional reports to evaluate issues we identified on the time-series consistency of the annual ACS estimates. However, despite earlier statements by the Census Bureau to compare and evaluate differences between state-level estimates from the Census 2000 Supplementary Survey (C2SS) and the 2000 Census long form, these reports did not include any reference to the preparation of such comparisons, and the Secretary did not indicate they would be prepared. Because the focus of the long form and the ACS is on data from small geographic areas, we believe that reports on states and on other areas with population of 250,000 or more should be prepared.

The last issue we identified was the need to provide users with guidance on the interpretation of key properties of multiyear averages. The Secretary agreed about the need but noted that guidance is not needed in 2005. He reported on a newly created NAS panel that will be studying many of the key issues identified in our report. However, we believe that the Census Bureau should begin to release guidance on the averages before the first multiyear averages are released in 2008. One area in which such guidance will be needed is the interpretation and use of the multiple ACS estimates. When the 2005–07 averages are released in 2008, users will have annual estimates for some of these areas for 2006 as well as the 3- year averages, which will be centered on 2006. In 2010, when the first 5- year averages are released (2005–09), users will have three sets of ACS estimates for places with populations larger than 20,000. For example, for each state, there will be an annual estimate for 2007 as well as 3-year and 5-year averages centered on 2007.

The comments from the Secretary also include a list of detailed technical comments from the Census Bureau. We reviewed each of these comments and revised the report where appropriate.

As agreed with your offices, unless you release the report’s contents earlier, we plan no further distribution of it until 30 days from its issue date. We will then send copies to the Secretary of Commerce, the Director of the U.S. Census Bureau, and others who are interested. Copies will be made available to others on request. This report will also be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-9750. Other staff who made major contributions to this report are listed in appendix VII.

Appendix I: Scope and Methodology

We used a combination of approaches and methods to examine the Census Bureau’s plans to develop, test, and implement the American Community Survey (ACS). We reviewed published and unpublished ACS-related Census Bureau reports, papers, presentations, budget documents, and congressional testimony; National Academy of Sciences (NAS) reports; congressional testimony delivered by outside experts; and consultants’ reports prepared for the Census Bureau, the Bureau of Labor Statistics (BLS), and the Department of Housing and Urban Development (HUD).

We reviewed an extensive set of internal planning documents prepared between 1992 and 1995 that the Census Bureau provided, relevant papers Census Bureau staff presented at professional association meetings and similar symposiums from 1995 on, and evaluation reports based on the 2000 Census. We also reviewed official Census Bureau presentations in special reports, congressional testimony, and recent advisory committee meetings. We reviewed similar materials NAS and consultants prepared for the Census Bureau and other federal agencies, as well as materials we prepared. The most important documents we reviewed are listed in the bibliography, organized by document type, at the end of this report.

In addition, we conducted independent research and analysis. To assess the evaluations the Census Bureau conducted to assist users in making the transition from the 2000 Census long form to the ACS, we obtained data from the 2000 Census and 2000 ACS (the Census 2000 Supplementary Survey) and prepared comparisons of key detailed data items at the state level. To determine the potential effect of replacing independent population and housing characteristics controls from the 2000 Census with corresponding data from the 2010 Census, we compared county-level intercensal estimates for April 1, 2000, based on the 1990 Census, with 2000 Census counts. We also analyzed the Census Bureau’s use of independent controls for estimates of population and housing characteristics for previous decennial censuses and its plans for the ACS.

Appendix II: Recent NAS Findings on Continuous Measurement and the ACS

“Although we believe that the proposed continuous measurement system deserves serious evaluation, we conclude that much work remains to develop credible estimates of its net costs and to answer many other fundamental questions about data quality, the use of small- area estimates based on cumulated data, how continuous measurement could be integrated with existing household surveys, and its advantages compared with other means of providing more frequent small-area estimates. In our judgment, it will not be possible to complete this work in time to consider the use of continuous measurement in place of the long form for the 2000 census.” “With regard to proposals to drop the long form in the next decennial census and substitute a continuous monthly survey to obtain relevant data, substantial further research and preparatory work are required to thoroughly evaluate the likely effect and costs of these proposals. Continuous measurement deserves serious consideration as a means of providing more frequent small-area data; however, the necessary research and evaluation cannot be completed in time for the 2000 census.”

Although 1994 saw the first proposals to implement the continuous measurement methodology as a replacement for the 2000 Census long form, the Census Bureau changed its plans in 1998, shifting to implementation to replace the long form in 2010.

Since 1995, NAS has produced several reports that relate totally or in part to the ACS, including a summary of a September 13, 1998, Committee on National Statistics workshop at NAS, two interim reports, a letter report, and a final report, by the Panel on Research on Future Census Methods, and a report released in early 2004 by the Panel to Review the 2000 Census. (In this appendix, we do not discuss NAS reports after 1995 in which the ACS was discussed as a potential data source for federal programs.)

With few exceptions, the members of these two NAS panels and the workshop participants reported findings that cover most of what we have identified as unresolved issues and summarize in this appendix. The NAS reports and ours differ somewhat in emphasis. We have focused on the production and use of ACS data, whereas NAS focused more on data collection and processing methodologies. These differences may reflect the fact that NAS panel members are very sophisticated users who are more likely to use ACS microdata files and make their own adjustments for methodological issues; they make little use of the regular ACS publications.

1998 NAS Workshop on the ACS

NAS sponsored a 1-day workshop in September 1998 to discuss methodological issues related to the ACS. Experts prepared “thought pieces” on issues NAS staff selected, with input from Census Bureau staff. The workshop’s specific discussion topics were combinations of information across areas and across time, funding formula, weighting and imputation, sample and questionnaire design, and calibration of the output from this survey with that from the long form. The thought pieces and comments on them prepared Census Bureau staff for the discussions at the workshop. “has been on refining data collection, leaving the final answers to the difficult analysis questions for later. Thus, procedures for nonresponse and undercoverage adjustment were modeled, to the extent possible, after current procedures used for the census long form.

Now that data collection has matured as the ACS demonstration phase is well under way, the Census Bureau is developing a research plan and initiating research to address all issues relating to ACS methodology. Fall 1998 therefore seemed an opportune moment for a workshop to assist the Census Bureau in developing a research agenda to deal with many of these challenging issues.”

The report contained no specific recommendations but identified areas where additional research was needed, including issues we have expressed concern about, such as the availability of multiple ACS estimates for geographic areas with populations larger than 20,000 and the likelihood of differences between ACS estimates and estimates from a Decennial Census short form. From our perspective, the most relevant of the workshop’s specific issues were (1) combining information across time, (2) weighting and imputation, and (3) calibrating the output from this survey with that from the long form.

Technical papers in the workshop’s agenda book contained considerable discussion of time-series issues. The discussion in this section of the workshop focused on replacing moving averages with time-series modeling and using current household survey data to develop models. Speaking for the Census Bureau, Alexander stated that “Our current plan is to release annual data for even very small areas and let users perform their own time series analyses. We welcome ideas about what the Bureau’s role should be . . .” “to make comparisons between the long form and ACS for all states, large metropolitan areas, large substate areas, and population groups. “The objective of the 1999–2001 comparison is to understand the factors associated with the differences between the 1999–2001 ACS and the 2000 long form in the 31 areas, using the second comparison study to develop a calibration model to adjust the 2000 long-form estimates to roughly represent what the full ACS would have yielded in 2000.”

Chapter 7 of the report was devoted to a discussion of calibration. The report stated that the model would “determine the effects that would be expected when switching from the long-form estimates to those from the ACS on various applications of long-form data.” Once adjusted, the calibrated long-form data for 2000 can be compared with ACS data that are collected following full field implementation in 2003, “in order to understand the dynamics over time of such characteristics as poverty and employment.” “We very much like the idea of viewing information from an ongoing comparison of ACS to CPS and other surveys as a way to help understand how the ACS ‘error profile’ might be changing over time and using this to help interpret ACS data in the context of the long-term time series of census estimates.”

The use of independent controls for population and housing characteristics was also discussed at the workshop, but very generally, because the Census Bureau had not yet developed proposals for the controls. For example, the report’s chapter 5 discussed improving the existing population controls. The Census Bureau reported discomfort with the quality of the existing county-level controls (from ICPE) and agreed that the ACS could be used to improve these estimates. The Census Bureau also acknowledged that differences in residence rules and reference period would complicate the calculation of population weights. However, no discussion was reported of how the population counts from the 2010 Census would be used. “the development of estimates that (a) sum to estimates at higher levels of geographic aggregation and (b) more closely approximate direct estimates at higher levels of aggregation . . . in the event that aggregate estimates are not constrained to (approximately) equal direct estimates (and also the release of direct estimates at lower levels of aggregation for analysis purposes) . . . .”

2000 Interim Report

The Panel on Research on Future Census Methods, sponsored by the Census Bureau, was formed to examine alternative designs for the 2010 Census and to assist the Census Bureau in planning tests and analyses to help assess and compare their advantages and disadvantages. In addition to the first interim report, Designing the 2010 Census, released in 2000, a letter report was issued in 2001, and a second interim report was issued in 2003 (both discussed below). The panel issued a final report in 2004. “The Census Bureau should develop a detailed plan for each evaluation study on how to analyze the data collected and how to use the results in decision making concerning 2010 census design. The Census Bureau should then use these plans to identify the benefits and resources required for each evaluation study, set priorities among them, and allocate sufficient resources for the careful completion of all or, at least, the highest priority evaluations.” “The American Community Survey is a proposed national, continuous, mailout-mailback survey of 250,000 households per month, with field follow-up that makes use of techniques closely related to those used in the census. Therefore, rather than rely exclusively on the two or three large-scale census tests, which are always at least slightly limited in their generalizability by the specific locations selected, the Census Bureau could use the ACS as a platform for testing possible changes in the census. This work could serve as preliminary testing to the larger mid-decade tests for the census design.” “The decennial census makes use of one residence rule definition, the ACS uses a second, and a third approach is being tested in the alternative questionnaire study. As the Census Bureau is well aware (based on the allocation of an experiment to this issue), confusion over residence rules is a source of possibly substantial error in the census. . . . The Census Bureau needs to find the residence rule (within the set of rules satisfying legal and other restrictions) that results in the most accurate estimates. To learn more about this issue, the panel proposes an ACS-short-form match study in 2000 to examine this and other short- form measurement error issues.” “The development of the ACS raises a number of issues related to the quality of and planning for the 2010 census. There are also many other important technical issues raised by the introduction of the ACS into the federal statistical system. Formation of a technical working group could help to address many of these issues.”

2001 Letter Report

The 2001 letter report—addressed from Benjamin King, Chair of the Panel on Research on Future Census Methods, to William Barron Jr., Acting Director of the Census Bureau—was prepared in response to a December 7, 2000, presentation by Census Bureau staff on the major elements of the Census Bureau’s strategy for the 2010 Census. The panel recommended that the Census Bureau produce a “business plan” for the 2010 Census that would provide an overall framework for development. It recommended that this plan include (1) a statement of objectives, (2) a timeline for completing tasks, (3) a cost-benefit analysis, and (4) more complete information on coordinating tasks within the Census Bureau. The panel also recommended the preparation of specific types of evaluation studies. “The Bureau is currently conducting a wide array of evaluation studies and experiments designed to assess the quality of the 2000 census and inform approaches to the 2010 census. As noted above, the panel applauds the scope of these evaluation studies. However, the panel is concerned that the Bureau has not sufficiently focused its evaluation program and has instead labeled most of its evaluation categories as high priority.” “comparison of estimates from the ACS and 2000 census long-form data, in sites where both are available; coverage of the population, disaggregated by demographic and geographic subgroups; the effectiveness of major automated systems for data collection, capture, and processing; the quality and completeness of long-form data collection; and the effectiveness of operations used to designate special places and enumerate the group quarters and homeless populations.” “to broaden its justification for the ACS, detailing the need for and use of long-form data and how those data needs will be addressed through the ACS, perhaps in conjunction with the CPS and other demographic surveys. Accordingly, the Bureau should expedite ongoing evaluations that assess the quality of ACS data relative to the quality associated with the traditional census long form.”

2003 Interim Report

“The most basic question the panel faces regarding the ACS is whether it is a satisfactory replacement for the census long form. We recognize that significant estimation and weighting challenges must be addressed and that more research is needed on the relative quality of ACS and long-form estimates.” “The Census Bureau should carry out more research to understand the differences between and relative quality of ACS estimates and long-form estimates, with particular attention to measurement error and error from nonresponse and imputation. The Census Bureau must work on ways to effectively communicate and articulate those findings to interested stakeholders, particularly potential end users of the data.

The Census Bureau should make ACS data available (protecting confidentiality) to analysts in the 31 ACS test sites to facilitate the comparison of ACS and census long-form estimates as a means of assessing the quality of ACS data as a replacement for census long-form data.

Again, with appropriate safeguards, the Census Bureau should release ACS data to the broader research community for evaluation purposes.

The Census Bureau should issue a user’s guide that details the statistical implications of the difference between point-in-time and moving average estimates for various uses.

The Census Bureau should identify the costs and benefits of various approaches to collecting characteristics information should support for the full ACS not be forthcoming.

These costs and benefits should be presented for review so that decisions on the ACS and its alternatives can be fully informed.” “The fact that the Census Bureau has not done more in comparing the data collected from the 31 test sites, the C2SS, and the 2001 and 2002 Supplementary Surveys with the data collected by the 2000 census long form is disappointing. Such analyses would help assess the quality of ACS data and would be helpful in making the argument for transition from the long form to the ACS. This deficiency is probably due to limited analytic resources at the Census Bureau and creates an argument for ‘farming out’ this analysis to outside researchers.” “The ramifications of this basic concept emerge when moving average estimates are entered into sensitive allocation formulas or compared against strict eligibility cutoffs. A smoothed estimate may mask or smooth over an individual year drop in level of need, thus keeping the locality eligible for benefits; conversely, it may also mask individual-year spikes in activity and thus disqualify an area from benefits. It is clear that the use of smoothed estimates is neither uniformly advantageous nor disadvantageous to a locality; what is not clear is how often major discrepancies may occur in practice.” “It is incorrect to use annual estimates based on moving averages over several years when assessing change since some of the data are from overlapping time periods and hence identical. At the least, the results will yield incorrect estimates of the variance of the estimates of change. Therefore, users should be cautioned about this aspect of the use of moving averages.”

In both recommendations on evaluations and moving averages, the panel called for the Census Bureau to engage in a greatly expanded effort to inform users and stakeholders. It also suggested that the Census Bureau farm out some of the research efforts. “Eight years later, faced with the task of offering advice on making the vision of continuous measurement a reality in the 2010 census, the similarity between the arguments then and now is uncanny. Similar, too, are the points of concern; the current panel is hard-pressed to improve upon the basic summary of concerns outlined by our predecessors. We are, however, much more sanguine that a compelling case can be made for the ACS and that it is a viable long-form replacement in the 2010 census.”

However, while the panel was identifying its concerns, it also supported full funding of the ACS, believing that existing “flaws” in the plan could be resolved.

The 2000 Census: Counting under Adversity

“the Census Bureau, the administration, and Congress agree on the basic design for the 2010 census no later than 2006 in order to permit an appropriate, well-planned dress rehearsal in 2008. In particular, this agreement should specify the role of the new American Community Survey (ACS). Further delay will undercut the ability of the ACS to provide, by 2010, small-area data of the type traditionally collected on the census long-form sample and will jeopardize 2010 planning, which currently assumes a short-form-only census.” “the Bureau should also study the effects of imputation on the distributions of characteristics and the relationships among them and conduct research on improved imputation methods for use in the American Community Survey (or the 2010 census if it includes a long-form sample).” “publish distributions of characteristics and item imputation rates, for the 2010 census and the American Community Survey (when it includes group quarters residents), that distinguish household residents from the group quarters population (at least the institutionalized component). Such separation would make it easier for data users to compare census and ACS estimates with household surveys and would facilitate comparative assessments of data quality for these two populations by the Census Bureau and others.”

The panel’s findings were similar to our findings, with one major difference. The panel’s findings imply that some research on the ACS can be conducted after the results of the 2010 Census short form become available. In contrast, we see that such research is needed in order to improve the ACS by 2008, the first year in which ACS data will enter into the calculation of the 5-year average estimates (2008–12) that will replace the long form.

Appendix III: The Decennial Census Long Form and the Evolution of the ACS Plan

Decennial Census Long Form

In the decennial census for 1940 and for 1950, the Census Bureau used a single form to collect, from all households, population and key characteristics such as age and gender and, from a sample of households, detailed demographic, economic, and housing items. In the 1940 Census, the Census Bureau used a sample of 5 percent of the population to collect data on questions on income, internal migration, and Social Security status, as well as on more refined questions on unemployment. In addition, the Congress authorized a new set of questions about the types of plumbing, heating, and appliances in dwellings.

Beginning with the 1960 Census, the first conducted by mail, it became necessary to use separate forms—a short form to collect population data from all households and a long form to collect the detailed items from a sample of households. In the 2000 Census, for example, the Census Bureau conducted a sample of 17 percent of the population and asked 45 questions on the long form. Since 1960, the long form has evolved into a cost-efficient way to collect data federal agencies need that minimizes respondent burden. For 2000, for example, the long form consisted of 45 questions that the Census Bureau developed working through OMB and with the consent of the Congress. Each question provided information required by statute. Thus, the 2000 long form provided all federal departments and agencies with critical data, and it was estimated that these data were used to allocate more than $200 billion in federal funds.

Evolution of ACS Plan

In the 1950s, Census Bureau officials and users of Decennial Census data had begun to develop a program to provide intercensal data on population characteristics. The first major proposal to provide intercensal data called for a mid-decade census that would provide information every 5 years. In 1976, the Congress enacted legislation to require a mid-decade census beginning with 1985, but did not fully fund the program.

In the late 1980s, the Census Bureau shifted efforts to provide intercensal estimates to a program based on CM methodology, or Continuous Measurement. This approach would provide for more timely population data as well as the detailed demographic, economic, and housing data collected every 10 years by the Decennial Census long form. The program would integrate a new sample survey, existing surveys, administrative records, and statistical modeling. After a thorough analysis of alternatives based on this methodology, the Census Bureau developed a plan similar to the current ACS to replace the 2000 Census long form.

Initial $2.6 million funding for the CM program was included in the 2000 Decennial Census budget for fiscal year 1995. These funds were to develop, test, and evaluate a CM program to replace the Decennial Census long form and to provide more timely long-form type data. In the program description in the budget documents, the Census Bureau reported that it planned to develop the new program that would integrate a new sample survey, existing surveys, administrative records, and statistical modeling. Table 2 shows that about $330 million has been provided to fund the CM program since 1995, with funding provided separately until 2003 and additional funding from both the 2000 and 2010 Decennial Census programs. Beginning with 2003, all funding has been provided as part of the 2010 Census program. The Census Bureau requested $165 million for fiscal year 2005.

In 1996 and 1997, funding was provided to field-test what became the ACS, to replace the 2000 Census long form. The ACS was to begin in 1999 with an annual sample of 4.8 million housing units for 1999, 2000, and 2001 and 3 million housing units for subsequent years. Under this plan, a 3-year average of ACS data for 1999–2001 was to replace the 2000 Census long form. It would provide the same detailed items and same level of geographic detail as the traditional long form with about the same quality. Annual ACS data would subsequently be provided for geographic areas with populations of 65,000 or more, 3-year averages would provide ACS data for geographic areas with populations larger than 20,000, and 5-year averages would provide ACS data for small geographic areas, such as census tracts, small towns, and rural areas. The 5-year average for 2010, 2020, and beyond would replace future Decennial Census long forms.

In the 1998 budget request, the Census Bureau shifted the timing for replacing the long form from the 2000 Census to the 2010 Census. As a result, it was funded to conduct annual supplementary surveys of 750,000 households beginning with 2000, in addition to the ACS testing at four test sites (or counties). The Census 2000 Supplementary Survey, known as C2SS, and the surveys for subsequent years were to be used to test the feasibility of collecting long-form data at the same time as, but in a separate process from, the Decennial Census. Data from C2SS and the supplementary surveys were also to be used to test ACS data usability and reliability and to evaluate operational and programmatic issues associated with implementing the ACS. Also, the number of test sites was increased to 31 by 1999. Funding to compare and evaluate differences between data collected from the 2000 Census long form and the ACS testing programs began in 1999, to develop data to expand coverage to group quarters and Puerto Rico in 2001. Plans to integrate existing surveys, administrative records, and statistical modeling into the new program were dropped in 2001.

The 1998 budget request also reported that the Census Bureau would proceed with plans to replace the 2010 Census long form with an ACS based on an annual sample of 3 million housing units, as with the previous plan. Unlike that plan, the sample size for 2009–11 would not increase to provide 3-year averages for 2010. This revised plan called for full implementation of the ACS in 2003. Full ACS data for 2003 to 2007 would have made 5-year averages available in 2008, 4 years before the long-form sample statistics from the 2010 Census would become available. However, budget decisions by the Congress delayed full implementation until the fourth quarter of fiscal year 2004.

Appendix IV: Continuous Measurement ACS Testing and Development Program

The Congress initially provided funds for testing the CM methodology in 1994. As we have noted, the Census Bureau had begun formal testing of the CM program in 1996 with an operational test of the ACS in four counties; this test was expanded to 31 test sites by 1999. A second testing program, the Supplementary Survey program, began in 2000 as a part of the 2000 Decennial Census. The Census Bureau designed C2SS to test the feasibility of collecting long-form data at the same time as, but in a separate process from, the 2000 Decennial Census. Data from C2SS and the same supplementary surveys, beginning with 2001, were also to be used to test ACS data usability and reliability. According to the Census Bureau, these surveys were to be used to examine technical, statistical, and operational issues associated with implementing the ACS and to document the key results in a series of reports.

Before field testing began, the Census Bureau had conducted an extensive research program to identify the issues related to using the CM methodology and to replacing the long form. The research program resulted in a series of 20 reports, known as the Continuous Measurement Series, between 1992 and 1995. These reports, most of which were prepared by Charles Alexander, addressed a wide range of topics such as replacing the 2000 Census long form, collecting intercensal population data, and integrating the ACS with existing household surveys. The reports on replacing the long form identified the key issues that needed testing, and they served as the primary input to the Census Bureau’s ACS test program. These issues included those subsequently tested by the Census Bureau as well as the unresolved issues we identify in this report.

Following the CM reports, Census Bureau staff presented papers from 1995 through 2001 on ACS testing at various professional association and similar meetings, as well as at a 1998 symposium on the ACS sponsored by the Census Bureau. For example, the 1995 paper by Love, Dalzell, and Alexander discussed issues related to the evaluation of the 1996 test site results, expressing concern about population controls and residence rules as well as the need for consultation with users. They reported that the Census Bureau was planning to conduct research using data from the 1996 test sites to produce controls at the census tract and block group level. They also noted that the Census Bureau would need to conduct research on the residence rule.

Alexander and Wetrogan also discussed the issue of population controls in their 2000 paper. They reviewed possible methods for using ICPE to develop controls for the ACS and discussed using ACS estimates on the foreign-born U.S. population to improve the Census Bureau’s foreign- migration component of the intercensal estimates. (They reported that this effort would be part of what the Census Bureau had previously referred to as the Program of Integrated Estimates.) They also noted the need to consult with users on how to present information on the differences in ACS controls and ICPE in ACS publications.

Several papers have focused on the key role of evaluating differences among the ACS test data, census long-form data, and CPS data. Alexander, Dahl, and Weidman reported in 1997 that during the demonstration period, they would be working closely with experts familiar with specific test sites to learn about the quality of the ACS estimates. For example, they reported that the Census Bureau would be looking into sources of differences between the 1999–2001 ACS test-site average estimates and the 2000 Census long-form results and using the results of differences between the 2000–02 national sample and the 2000 long form to generate model- based estimates for small geographic areas. The authors noted that these model-based estimates, based largely on information from test sites, would be used to interpret changes between 2000 and future ACS estimates.

In another 1997 paper, Davis and Alexander reported the Census Bureau’s action plan for evaluation studies. They called for evaluating the results of all test sites and releasing the expert review of the analyses of the differences between the 1999–2001 ACS and the 2000 Census long form. The schedule called for releasing this information before beginning the implementation of the full ACS. Alexander’s 1998 paper on completed research, research in progress, and planned research included among the four items for planned research a “close study of differences between 1999–2001 ACS and 2000 long form in comparison areas.”

The quality of the ACS measures of income was the subject of the paper Posey and Welniak presented at the Census Bureau’s 1998 symposium on the ACS. They compared income reported in the 1996 ACS and 1990 Decennial Census in an effort to evaluate the quality of the ACS income data. One of the adjustments they made to compare the two series was for the effect of inflation between 1990 and 1996. They noted that the results of the comparisons indicated a potential problem that may relate to the ACS inflation adjustment. (They described the calculation of the adjustment, which is based on the CPI, but did not provide a rationale for using the adjustment in the ongoing ACS data.)

Alexander and two BLS staff reported in 1999 on the potential for using the ACS to improve labor force data from the CPS for state and smaller geographic levels. They stressed that to develop procedures for making these improvements, much research would be needed to evaluate differences between the ACS and CPS.

The last research paper in this period was Alexander’s 2001 paper focusing on the origins of the CM methodology and its developers. He discussed the ACS in the context of the methodology, noting several important differences related to the nature of the ACS. He included a review of the Census Bureau’s testing and evaluation program, noting that the ACS test- site program had been expanded and that national sample supplementary surveys had been added. He said that these test data would be compared with the 2000 Census long-form data and that in 2001 and 2002, the Supplementary Survey would be used as part of the transition to the ACS. He also pointed to unresolved issues relating to the residence rule and the multiyear averages, because they would provide users with multiple estimates for geographic areas with populations larger than 20,000.

Between 2001 and 2003, the Census Bureau has issued three official reports and one internal report on the status of the ACS testing and development program. In Demonstrating Operational Feasibility, published in July 2001, the Census Bureau gave a brief history of the ACS development program, which by 2001 was focused on preparing for full implementation in 2003 (although the Census Bureau later revised this to 2004) but on its operational feasibility, using data from C2SS. On the basis of the Census Bureau’s analysis of the results of its tests of operation feasibility, it reported the tests a success. However, it recognized that more evaluation on measures of data quality was necessary, as well as on differences between ACS and 2000 Census long- form data. The Census Bureau announced that over the next 2 years it would issue reports comparing data from the 2000 Census long form at the national, state, and smaller geographic areas with data from the C2SS and the ACS development program.

Demonstrating Survey Quality, published in May 2002, focused on measures of C2SS survey quality, summarizing sampling and nonsampling error levels in both C2SS and the 31 ACS test sites. The Census Bureau used available, generally accepted measures of quality. On the basis of its analysis of the results of these quality tests, the Census Bureau reported the tests a success. This conclusion rested on test results that showed the C2SS program capable of providing reliable long-form data.

As in the July 2001 report, the Census Bureau recognized that more evaluation was necessary on measures of data quality as well as on differences between ACS and 2000 Census long-form data and the detailed estimates produced from C2SS. The Census Bureau repeated its commitment that over the next year and a half, it would release other reports to (1) analyze in detail basic demographic characteristics (relationship, race, tenure) produced from the C2SS at the national and state levels, including comparisons between C2SS and Census 2000; (2) describe the data release plan and products for the ACS and the usability and accessibility of estimates resulting from ACS methods; and (3) give several detailed analyses of selected social, economic, and housing characteristics (education, income, commuting patterns), including comparisons between C2SS and Census 2000 at the national and some subnational levels.

In June 2002, shortly after Demonstrating Survey Quality was released, a team of Census Bureau specialists who had been working on the ACS for several years prepared an internal report on testing. They presented a revised program development plan and identified key questions to be answered in testing the adequacy of the ACS in replacing the Decennial Census long form. Their plan included the preparation of a series of nine evaluation reports over 2 years. The reports that evaluated differences between the 2000 Census short-form data (100 percent reported) and corresponding C2SS items were included in Demonstrating Survey Quality. Three reports to be completed between October 2002 and January 2003 would evaluate differences between the detailed housing, social, and economic characteristics between C2SS and the 2000 Census long form, as described in Demonstrating Survey Quality. (Although this schedule was later extended to the end of 2003, these three reports still had not been released when we prepared our final draft of this report.)

Finally, the team’s plan included a report that would focus on the comparisons of 3-year averages for the basic demographic, housing, social, and economic characteristics from the C2SS and ACS test sites and comparable estimates in the 2000 Census long form. The last report in the plan would compare data for 2001 and 2002 with measures shown in Demonstrating Operational Feasibility. The plan did not provide completion dates for these reports.

American Community Survey Operations Plan, Release 1, published in March 2003, identified research projects to be completed in preparation for full implementation of the ACS. Two projects were on “weighting and estimation,” which covered the methodology for using independent population and housing controls, and on “program of integrated estimates,” which covered the calculation of these controls from the Census Bureau’s intercensal population estimates program. The operations plan also reported on the schedule for completing several comparison and evaluation projects with ACS and 2000 Census long-form data discussed in Demonstrating Survey Quality. It discussed the need to evaluate multiyear estimates from the supplementary surveys to demonstrate the usability, reliability, and stability of ACS estimates over time, and it stated that a report comparing 3-year ACS data with data from the 2000 Census long form would be released in mid-2003.

The Census Bureau reported that the results of these research projects would not be available in 2004. Instead, it said, it would use interim procedures, taking “extensive long-term investigation and experimentation” to develop final procedures. For the ACS weighting and estimation project, the Census Bureau reported that it would be using an interim adjustment to adjust the intercensal population and housing characteristics estimates to the ACS residence concept. The Census Bureau reported that ACS estimates of occupied housing units, households, and householders should agree at all geographic levels.

For the program of integrated estimates project, the operations plan discussed the need for more research to introduce improvements to the estimates from ICPE. (The ACS estimates are weighted to a population benchmark, either the most recent Decennial Census results or the most recent ICPE estimates.) The Census Bureau reported that because the accuracy of the intercensal estimates is important to overall ACS accuracy, it is important to use ACS data wherever appropriate to improve the intercensal estimates. The plan for the program on integrated estimates will use information from the 2000 Census, more current ACS distributions of population characteristics, and administrative records to produce improved population and housing unit estimates for all areas, including small areas. The plan also discussed improving housing characteristics by incorporating ACS distributions of local area vacancy rates and household characteristics into statistical models to better estimate subcounty populations. No time schedule for completing the research was provided.

Finally, the March 2003 American Community Survey Operations Plan, Release 1 discussed a plan in the ACS to cover group quarters. Persons living in group quarters live in places that the Census Bureau does not classify as housing units—for example, nursing homes, prisons, college dormitories, military barracks, institutions for juveniles, and emergency and transitional shelters for the homeless. Such residences accounted for roughly 2.8 percent of the population in 2000. Although data on group quarters were collected at the test sites beginning with 1999, data were not collected in C2SS or subsequent supplementary surveys. The operations plan discussed the use of an updated Census 2000 Special Places file for the sampling frame for the full ACS. In this case, the plan noted, training field representatives on collecting data from this population is to begin in October 2004, so that full data collection production can begin in January 2005.

Census Bureau staff made a presentation on comparison and evaluation reports at the April 2003 meetings of the Census Advisory Committee. The paper’s author reported that work was under way on the comparison reports noted in the March 2003 operations plan, and she described the methodology to be used to evaluate differences between the 2000 long form and C2SS. She also reported that the results of the comparisons would be used to identify how the ACS should be improved but that additional research would be needed to address consistency over time between the 2000 Census and the full ACS. She stressed the importance of evaluating consistency “in educating users on the transition from the decennial census sample estimates to the ACS estimates.”

With regard to the comparison report of selected demographic, housing, social, and economic characteristics of 3-year estimates from the ACS test sites to the 2000 Census, the Census Bureau let four contracts with local experts to conduct comparisons of 3-year averages of ACS data for 1999–2001 for selected test sites with selected 2000 Census long-form data as well as 2000 Census population and housing unit characteristics. The comparisons, prepared at the county and census tract levels, would be made for measures of data quality (self-response rates, sample unit nonresponse rates, item nonresponse rates, and sample completeness ratios), as well as for data levels (counts, percentages, means, and medians) for demographic, social, economic, and housing characteristics.

In summer 2003, Census Bureau staff presented a number of research papers on the ACS at the annual Joint Statistical Meetings. Papers evaluated differences between long-form and C2SS data items, such as persons with disabilities, educational attainments, and income. Most of the papers that provided comparisons with long-form data indicated whether differences were statistically significant for every comparison. Comparisons were presented at a variety of geographic levels (national, state, and test site levels). Some papers cited operational differences as possible explanatory factors, but information was not presented using a standard set of factors.

The Census Bureau published ACS-2010 Census Consistency Review Plan, an internal document, at the beginning of October 2003. Its purpose was to identify methods for major operations used in the ACS and for the 2010 Census that were likely to lead to inconsistent results and to recommend ways to address these inconsistencies. Papers prepared on these operations were to discuss how an issue might result in inconsistencies between the ACS and 2010 Census results and to set forth options for dealing with consistency issues, including a research process. The plan identified residence rules and group quarters as two topics. It did not discuss completing the work in time to incorporate it into the full ACS in the next several years.

Also in October 2003, the Census Bureau made two public announcements related to the ACS development plan at the Census Advisory Committee meetings. Two papers related directly to projects described in American Community Survey Operations Plan, Release 1. In “Enhancing the Intercensal Population Estimates Program with ACS Data: Summary of Research Projects,” Weidman and Wetrogan reported on research to improve the intercensal estimates by using ACS data for two “high priority” areas—international migration and internal migration. This work was being conducted within the Program of Integrated Estimates. The second paper described options for determining population control weights for ACS implementation in fall 2004 but did not indicate that research was under way to determine the effect of the options.

Another source of information related to ACS development was the various reports prepared as part of the Census 2000 Testing, Experimentation, and Evaluation Program. Schneider’s January 2004 report compared employment, income, and poverty estimates from the 2000 Census long form and the CPS. From this comparison, the author concluded that this work should be continued in an effort to use the results of the comparisons to improve consistency between data collected in the CPS and data in the ACS; the ACS uses the same questions as the 2000 long form. The author also identified for additional research long-form questions that performed badly, based on a reinterview survey.

From May to July 2004, the Census Bureau released seven ACS evaluation reports. Four reports compared data from the 2000 Census long form and the C2SS at the national level. Two reports compared these long-form data with 1999-2001 data from the ACS test sites for selected counties and one of these compared these data at the tract level. The other report reviewed operational data from the 2001 and 2002 supplementary surveys. In most of the reports comparing long-form and ACS data, the Census Bureau identified additional work that was needed to improve the quality of the ACS estimates or to help explain differences between the two sets of data for 2000. As noted earlier, these comparisons were limited to the national level. (The seven new reports are listed in the bibliography.)

Appendix V: Current Status of Unresolved Issues

Independent Controls for Population and Housing Characteristics

According to the Census Bureau’s plans, the calculation of independent controls for population characteristics (age, sex, race, and ethnicity) and housing characteristics for the full ACS will require a significantly different methodology from that used for the ACS supplementary surveys. Controls will be needed at the same level of geographic area detail as those that were used for the 2000 Census long form and will need to reflect the new concepts of residence and reference period underlying the ACS.

For the annual ACS supplementary surveys, these characteristics were used from ICPE as the independent controls. ICPE uses Decennial Census short-form data as benchmarks and administrative record data to interpolate between and extrapolate from the census benchmarks. The program provides “official” annual estimates of population and housing characteristics at the county level, and for some subcounty levels, as of July 1 of each year, using the usual residence concept for seasonal residents. The program also provides annual estimates of total population and housing units for all areas of general-purpose government, such as cities, villages, towns, and townships. Table 3 shows information on the calculation of the independent controls used for the 2000 Census long form, the ACS supplementary series, and the fully implemented ACS through 2012.

Using ICPE for the ACS supplementary surveys, the Census Bureau prepared controls for counties, or combinations of counties. As shown in table 3, for the residence concept, controls from the 2000 Census and ICPE, which were based on the usual residence concept, were used. The reference period for the ACS test program for all years except 2000 was for July 1; for 2000, it was for April 1. (Controls for the 2000 Census long form also were for April 1.)

For the full ACS, the Census Bureau will use controls based on the current residence concept. According to the Census Bureau, the current residence concept recognizes that the place of residence does not have to be the same throughout a year, so that the current residence concept allows the ACS data to more closely reflect the actual characteristics of each area. The Census Bureau will use the current residence concept because the ACS is conducted every month and produces annual averages rather than point-in-time estimates, as the Decennial Census does. Also, because the ACS data are collected monthly, it will be necessary to use independent controls that define the reference period as the average for the year using a July 1 reference period.

To produce ACS estimates for the full sample, the Census Bureau will need new methodologies for calculating independent controls. For the first annual estimates, for 2005, a methodology will be needed to provide ACS-defined controls for all places with population of 65,000 or more, including those for which intercensal population estimates are not available. For the 2005–07 estimates, which will be used to calculate the first multiyear averages, a methodology for controls for geographic areas with populations between 20,000 and 65,000 will be needed. For the 2008–12 estimates, a methodology for controls down to the geographic levels used for the 2000 Census long form will be needed. Finally, when the population and housing characteristics data from the 2010 Census short form become available and are incorporated into the ICPE estimates, another new methodology will be needed to revise the ACS controls for 2010.

The Census Bureau also has reported that it is not planning to revise earlier years’ ACS data for consistency with revised 2010 estimates unless the inconsistencies between the 2010 ICPE and 2010 Census characteristics were significant. Table 4 shows the differences between population estimates at the county level for 2000 using ICPE based on the 1990 Census and the corresponding data from the 2000 Census. In 2000, the population estimates for almost 20 percent of the counties differed by more than 5 percent. For counties whose population was smaller than 20,000, almost 25 percent had similar differences.

Census Bureau staff had long recognized the need for new methodologies to develop independent controls for the ACS. For example, a 1995 paper by Love, Dalzell, and Alexander, discussing issues related to evaluating the 1996 test site results, expressed concern about independent controls and residence rules, as well as the need for consultation with users. In 1998, the Census Bureau sponsored a conference on the quality of ACS data for rural data users. In the final report on this conference, the Westat authors concluded that the Census Bureau needed to continue and expand its contacts with stakeholders and to conduct additional research on several issues, including independent controls. Alexander and Wetrogan also discussed this issue at the 2000 Joint Statistical Meetings when they reviewed possible methods for using ICPE estimates. They also noted the need to consult with users on how to present information on the differences in ACS controls and ICPE in ACS publications.

Census Bureau staff also recognized that the new ACS would create differences between (1) ACS population and housing characteristics data and the corresponding “official” data from the Decennial Census and (2) ACS population and housing characteristics data and the “official” ICPE population estimates, which are benchmarked to Decennial Census data. They also recognized that the creation of new controls for the ACS would result in inconsistencies between ACS data and data from federal household surveys, such as the CPS, whose population and housing characteristics are also based on the Decennial Census and ICPE estimates. Such differences might hinder the use of ACS data to expand and improve small geographic area estimates based on the other surveys. (CPS provides official national estimates of labor force information, such as the unemployment rate and income estimates used to calculate the number of persons in poverty.)

In March 2003, the Census Bureau announced that it did not have a final methodology and that such methodologies would not be established for several years. In March 2003 in American Community Survey Operations Plan, Release 1, the Census Bureau identified research projects to be completed in preparation for full implementation of the ACS. One of these projects, “weighting and estimation,” covered the methodology for calculating the independent controls for the ACS; a second, “program of integrated estimates,” covers the calculation of these controls from the ICPE. This plan also reported that the results of these research projects would not be available in 2004 to begin implementing them with the start of the full ACS. Instead, the Census Bureau said it would use interim procedures and that it would take “extensive long-term investigation and experimentation” to develop final procedures.

For the weighting and estimation project, the Census Bureau reported that it would be using an interim adjustment to adjust the intercensal population and housing characteristics estimates to the ACS residence and reference period concepts. This project would include research to examine the need to achieve agreement between the estimates of occupied housing units, households, and householders at all geographic levels. The Census Bureau reported that work on the project to revise and simplify the weighting methodology began in early 2003, that preliminary papers documenting the revisions might be available by summer 2004, and that research would continue for several years.

For the program of integrated estimates project, the operations plan discussed the need for more research to introduce improvements to the ICPE estimates using information from the 2000 Census, more current ACS distributions of population characteristics, and administrative records to produce improved population and housing unit estimates for all areas, including small geographic areas. The plan also discussed improving the housing characteristics. ACS distributions of local area vacancy rates and household characteristics can be incorporated into statistical models that use distributions of housing unit characteristics to better estimate subcounty populations. No time schedule was provided for completing the research.

In October 2003, Census Bureau staff presented a paper at the Census Advisory Committee meetings that described the options being considered to convert the ICPE estimate to the current residence concept. The paper described options for determining controls for ACS implementation in fall 2004 but did not indicate that research was under way to determine the options’ effects. A second paper at the same meetings reported on research to improve the intercensal estimates by using ACS data for two “high priority” areas—international migration and internal migration. This work was being conducted as part of the Program of Integrated Estimates. “Given the newly benchmarked intercensal estimates, the following question arises regarding the use of the 2010 Census data in the ACS: Should ACS estimates continue to be controlled to 2010 Census data at the county or county group level and differences between the ACS and census population counts and characteristics allocated proportionately to the tract or block group levels? Or should ACS estimates be controlled to 2010 Census data at the tract and block group level, as would have been the case with a long form?”

All the experts agreed that the ACS should be controlled to the decennial census, but several noted that they had not thought about the issue and had not heard anything from the Census Bureau on the issue. (The experts are listed in app. I.)

Operational Issues

The Census Bureau has identified operational issues with the ACS test programs, primarily from information from evaluation studies on the 2000 Decennial Census and Census Bureau staff research papers on comparisons between data collected in the ACS 2000 supplementary survey and the 2000 Decennial Census long form. These issues include problems with questionnaire design, nonresponse followup, and data capture, as well as coverage of persons living in group quarters.

In January 2004, the Census Bureau released the results of a key evaluation study of 2000 Decennial Census long-form data, using a reinterview survey. The study identified problems with long-form questions, which are the same as those used for the ACS, and proposed several research efforts based on a statistical evaluation of the quality of the responses to each question. For questions identified as having significant quality problems, the study recommended research on the design of the form and placement of the questions and suggested using cognitive experts in testing revised questions. The study also recommended that the Census Bureau and BLS work on the ACS employment and unemployment questions to ensure that they would complement the BLS local area unemployment statistics program.

The Census Bureau also conducted a study to evaluate the design of the ACS questions that are needed to implement the residence concept and reference period for the ACS. The study suggested that additional testing was needed for the questions about multiple residences (currently, the last set of questions in the housing section). It noted “that asking these questions on a person basis may produce different and probably better data than asking them on a household basis.” The study was limited in scope and did not assess how accurately ACS respondents assign persons associated with the household to a current residence.

In the ACS, the Bureau uses “In the past 12 months . . .” whereas the Census Bureau used “In 1999 . . .” for the long form. Because the reference date is not fixed, it is important for a respondent to supply the date that the ACS questionnaire filled out. Otherwise, it cannot be determined whether there is an inconsistency in an ACS questionnaire received in late April 2004 that lists a resident aged 10 with a birthdate of April 15, 1993.

Census Bureau staff also discussed operational issues in research papers, based on evaluations of comparisons between 2000 Decennial Census long-form and ACS 2000 supplementary survey data for selected items presented at the 2003 Joint Statistical Meetings. A paper on income data identified the new question on the reference period as a potential source of problems, even though an additional instruction had been added to the ACS questionnaire in 1999. The authors expressed concern that some ACS respondents may misinterpret the question on “income in the past 12 months” as a request for monthly income instead of income during the previous year. The paper also included recommendations for additional research on the effect of the data capture methods. For the 2000 long form, all data items were entered with an automated optical character recognition procedure; data from the ACS will be manually keyed.

Another paper, presented at the same 2003 meetings, that evaluated differences in the data on disabled persons found large and significant differences at the national level and also recommended that new questions be tested. Additional areas were identified for further research, based on evaluations of questions such as educational enrollment, ancestry, and grandparents caring for grandchildren. These areas included specific facets of the mailout-mailback system and nonresponse followup. For example, nonresponse follow-up for the 2000 long form was conducted for all nonrespondents, but for the ACS test program and for the full ACS, nonresponse follow-up will be conducted for a sample of one-third on all nonrespondents.

The Census Bureau also has discussed issues with the expansion of ACS coverage to include persons living in group quarters—for example, nursing homes, prisons, college dormitories, military barracks, institutions for juveniles, homeless shelters. In October 2002, it informed its advisory committee members of the formation of a special planning team to address issues on the definition of group quarters and duplication in the address file. From the minutes of this meeting, it appears that this team will focus on group quarters in the context of the 2010 Census short form. In the ACS March 2003 operations plan, the Census Bureau reported on a new project to cover group quarters in the full ACS. The Census Bureau reported that the special project was needed because of the special challenges of developing an updated address list; in the past, such a list had been updated only once a decade. According to the Census Bureau, tests on the new list were to be completed in time for use in the full ACS in January 2005. In addition, an internal planning document issued in October 2003 identified group quarters (and residence rules) as special problems and instructed staff to provide recommendations on the collection of data on them in January 2004. Usually, the Census Bureau tests new questions. According to recent Census Bureau decisions, those tests would have to be completed so that new questions could be incorporated into the 2008 ACS questionnaire.

Valuation and Presentation of Dollar-Denominated Data Items

The Census Bureau has adjusted all dollar-denominated items from the ACS testing programs, such as incomes, housing values, rents, and housing-related expenditures, for inflation. For example, ACS data for 2001 and 2002 released in September 2003 for median household income are expressed in 2002 dollars. This practice means that when each added year of ACS data is released, all dollar-denominated items for prior years will be revised. The Census Bureau makes a similar adjustment for the annual income data collected in the CPS. Unlike the ACS, the Census Bureau releases annual CPS data without the adjustment. In addition, the annual values collected in the ACS were adjusted to the calendar year. It will be using the CPI for the annual and monthly adjustments for all geographic areas.

A report prepared for HUD found problems with the adjustment, including (1) the lack of a “trending” adjustment in the calculation of annual averages, (2) the use of the adjustment for multiyear averages, (3) the adjustment for cost of living for data items other than income, and (4) the lack of the unadjusted annual data that would enable HUD to use alternative methodologies. In addition, research by Census Bureau staff questioned the adjustment for incomes when they found that it was a probable source of difference between income data from the supplementary survey and corresponding data from the CPS and the 2000 Census long form. “Making an inflation adjustment is not the same as trending. The cost of living adjustment assumes that the purchasing power measured at any point in the data collection period remains constant throughout the period. For example, assume that the cost of living rises by 3 percent a year. If a household reports an annual income of $50,000 in January, a cost of living adjustment to the end of the year would increase this income to $51,500, the amount needed in December to equal the purchasing power of $50,000 in January. A trending adjustment makes no assumption about purchasing power. It attempts to track movements in dollar income. Assume that dollar income is growing at 5 percent a year.

Then a trending adjustment to the end of the year would increase the $50,000 reported in January to $52,500 in December.” “The Census Bureau plans to report income in constant dollars. Income information collected in the various months will be adjusted for inflation so that all collected income will be expressed in dollars with the same purchasing power, presumably the purchasing power of dollars in December of the survey year. For moving average tabulations, all income information will be adjusted for changes in purchasing power over the period used to calculate the moving average. In other words, income reported by a respondent in the first month of a five-year moving average will be adjusted for almost five years of inflation.” “The standard Census Bureau tables for areas over 65,000 will tabulate the rents reported by respondents over the twelve months during which data were collected. A unit reporting a contract rent of $800 in January might actually be paying $850 in December. The standard table would record this unit as having a rent of $800. The standard Census Bureau tables for areas under 20,000 will tabulate rents reported by respondents over a sixty-month period. A unit reporting a contract rent of $800 in the January of the first year might actually be paying $1,070 in December of the fifth year. The standard table would record this unit as having a rent of $800.”

Such changes would not be captured with an adjustment based on the all- items CPI. “The ACS will generate income distributions comparable to those from the decennial census, but the distributions will have a feature that will complicate the use of income data from the ACS in APP measures. Whereas the decennial long form measures money income, the ACS reports average purchasing power.”

The report thus recommended that HUD use the unadjusted data—data that the Census Bureau had not planned to publish—in order to make the changes needed for HUD. “If no CPI adjustment had been made to the dollars reported on either Census 2000 or C2SS/ACS, the difference between medians at the U.S. level would have been smaller than the 4.6 percent shown in Table 3 . Instead, the difference would have been 2.5 percent. Since adjustment clearly played a role in determining the size of the difference between Census 2000 and C2SS/ACS estimates, it would be worthwhile to examine the costs and benefits of adjusting C2SS/ACS incomes as well as the choice of factors used to adjust them.”

The authors summarized their findings by concluding that “it is clear that we are just at the beginning stages of understanding why Census 2000 and C2SS income figures differ.” They noted that the income comparisons are most critical because these Census Bureau data are used in the calculation of the number of people in poverty.

In a December 2003 research paper, Census Bureau staff examined concerns about the absence from the official poverty measures of adjustment for geographic differences in cost of living. Like the ACS, for which the Census Bureau is assuming that the cost of living is the same throughout all geographic areas, the poverty measures are based on the same assumption. The authors concluded that the use of a poverty measure that takes into account geographic differences in housing costs, would significantly change the poverty measures in many states.

Evaluations of ACS, Long-Form, and CPS Data Comparisons

One of the Census Bureau’s major justifications for the ACS test programs has been its comparing data collected in these programs, and corresponding data from the 2000 Decennial Census short and long forms, to identify operational problems. Another major justification for the ACS test programs has been the use of these comparisons, and comparisons with corresponding data from the CPS, to inform users in making the transition from the 2000 long form to the ACS. “to make a transition from the Census 2000 long form to collecting long-form data throughout the decade, we will begin ACS data collection in 1,203 counties. This data collection will allow for comparison of estimates from Census 2000 with estimates from the ACS for all states, large cities, and population subgroups, and will help data users and the Census Bureau understand the differences between estimates from the ACS and the Census 2000 long form.” “These data will also contribute to a comparison with data from Census 2000 that is necessary because there are differences in methods and definitions between the census and the ACS. Moreover, decision makers will want to compare an area’s data to those from Census 2000. Comparisons using data from the operational test and from the 31 sites are essential to determine how much measured change between Census 2000 and future years of the ACS is real and how much is due to operational differences between the ACS and the census.”

When the Census Bureau began in 2001 to report on full implementation of the ACS, its first report focused on the operational feasibility of conducting the ACS. Its second report in 2002 focused on differences in operational characteristics of the ACS and the census long form, such as response rates and the extent of imputations.

The 2002 report stated that three reports evaluating differences between the ACS and census long form would be published at the end of 2003. The Census Bureau repeated this schedule in March 2003 when it released another official report on ACS plans. In September, we were told by one of the ACS experts that consultants had been hired to conduct evaluations for 4 of the 31 test sites. The reports on comparisons with long-form items and for the test sites were published in May, June, and July 2004.

The results of these comparisons are similar to comparisons and evaluations of long-form data items previously prepared by Census Bureau staff, BLS, and GAO. “These comparisons showed large national differences for key items that did not appear to be accounted for by coverage differences between the two surveys. For example, at the national level, the largest differences were for these items: (1) for the number of housing units lacking complete plumbing facilities, with the long-form estimate 27 percent higher than the estimate from the supplementary survey, and (2) for the number of unpaid family workers, with the long-form estimate 59 percent lower. . . . We also found a great degree of variation in the state differences between the long form and the supplementary survey.” “We found that at the national and state levels, there were small differences for the unemployment rate and for the poverty rate for all individuals. In contrast, comparisons of these rates for the CPS with these two surveys showed larger differences. The national unemployment rate, according to the CPS, was 4.0 percent, compared with 5.8 percent for the long form and 5.4 percent for the supplementary survey. The national rate for individuals in poverty for the CPS was 11.3 percent, compared with 12.4 percent for the long form and 12.5 percent for the supplementary survey.”

Given these results, we recommended that the Census Bureau expand the scope of evaluation studies to develop supplementary survey estimates for states and large places consistent with the 2000 long form and that it include in its evaluations comparisons of year-to-year changes for 2001 and 2002, using data from the supplementary surveys and the CPS at the national and state levels for key economic and housing items. “Relative to the CPS, the ACS consistently generates lower estimates of the labor force and employment but higher estimates of unemployment. These patterns are present in each of the years 2000, 2001, and 2002. They are repeated in nearly all state-level data as well.”

He made a series of recommendations for additional research, some requiring additional information from the Census Bureau. “provided a summary of the major differences between the two income data sources, in terms of data collection, capture, and processing, and provided very preliminary assessments of the possible role these differences may have played.”

The authors reported that additional work was needed to understand the differences and offered recommendations for further research.

Another paper presented at the same meetings examined differences between the national estimates for people aged 5 or older with a disability—48.9 million was the 2000 Census long-form estimate, 39.7 million the C2SS estimate. The author did not determine which estimate was more reliable but did find that the wording of some questions might explain the overall difference. In addition, the author reported that more work, such as additional analysis of currently available data and testing of new questions, was needed to clearly identify the reasons for the difference. The differences in disability data were also the subject of a National Council on Disability position paper, which recommended changes to the questions on disability. “The Census Bureau should carry out more research to understand the differences between and relative quality of ACS estimates and long-form estimates, with particular attention to measurement error and error from nonresponse and imputation. The Census Bureau must work on ways to effectively communicate and articulate those findings to interested stakeholders, particularly potential end users of the data.” “The Census Bureau should make ACS data available (protecting confidentiality) to analysts in the 31 ACS test sites to facilitate the comparison of ACS and census long-form estimates as a means of assessing the quality of ACS data as a replacement for census long- form data. Again, with appropriate safeguards, the Census Bureau should release ACS data to the broader research community for evaluation purposes.”

Information on Key Properties of Multiyear Averages

One of the major differences between the ACS and the long form it will replace is that the ACS will provide data for geographic areas with populations smaller than 65,000 in terms of multiyear averages. Because of the statistical properties of these averages and users’ unfamiliarity with them, we and many other stakeholders have identified these averages as a major challenge for users, including federal agencies. The Census Bureau has recognized the need for such guidance on the averages but has not made public plans for the topics to be discussed or when the guidance will be published. “In discussing this issue, a number of the participants thought that averages were particularly problematic for those areas in which change is irregular. For example, the question was raised as to the meaning of ‘average poverty’ over a 5-year period in which poverty rose and fell from one year to the next and, thus, the average would have no obvious meaning.”

The report made similar comments with regard to such characteristics as unemployment and income. Although the conference participants had generally agreed with these concerns, the report pointed out that annually updating the 5-year averages “will provide some insight into trends, although turning points will be difficult to discern precisely, as will short- term trends.” “Although a 5-year moving average will generally provide reasonably reliable cross-section statistics for all areas, including very small communities, some care will have to be exercised in choosing time periods for which changes in population or their characteristics are measured. With 5-year averages, four-fifths of the data in a pair of neighboring years will be identical. The change being measured will then be one-fifth of the difference between the most recent year and the first year of the earlier time period. The sampling errors of the differences will thus be based on annual sample sizes, not 5-year averages, and will generally be too large to make useful inferences for small areas. The two 5-year averages that are being compared should generally be discrete and non-overlapping periods, e.g., 2003–2007 and 2008–2012. These comparisons will have about the same reliability as changes between two censuses using data collected in the Census long form.” “encourage analysts to use the same length of cumulation when comparing areas of different sizes . . . . For example, we would use one year for comparing states, but would recommend 5 years for all the counties in a table comparing large and small counties.”

Alexander noted that this approach differed from that of Kish, the developer of the concept of a “rolling sample,” who would “let us use tables of counties with one-year estimates for large counties, 3-year averages for medium-sized ones, and 5-year averages for small ones.” He concluded this section of the paper by saying, “It will be interesting to see what practices data users will adopt in this regard.” “If there is little change in the population over the time covered by the average, the interpretation is about the same as that of a point-in-time estimate with the advantage that the ACS estimate is more current than the historical decennial census long-form estimate.”

The paper provided examples with “naive” assumptions about how users extrapolate between censuses to show that multiyear averages “work.” By implication, under other conditions, users will need guidance on when multiyear averages can be used. The paper also did not discuss the interpretation of changes in the multiyear averages, as in the 1999 Westat conference report or multiple estimates, which Alexander had discussed in his paper for the 2001 Statistics Canada conference.

In September 2002, two reports focused on issues related to the statistical properties of multiyear averages. We published a report on several aspects of the ACS, including the selection of questions and the feasibility of conducting the ACS as a voluntary survey, and HUD released a report prepared for its staff on the use of the ACS for HUD programs. “Analyze and report on differences between year-to-year changes for 2001 and 2002, using the data—from ACS special supplements and the CPS at the national and state levels—for key economic and housing characteristics, such as the unemployment and poverty rates, to determine the stability of the annual ACS data.”

We also discussed the need for additional information on the characteristics of the multiyear averages to help federal agencies make the transition to the ACS. We specifically noted the need for information on the selection of ACS data for geographic areas with populations larger than 20,000 for which there will be multiple estimates. On this issue, we stated that, “In addition, we found that the ACS development program did not cover information about different ways to integrate the annual data for states and large counties and the 3- and 5-year averages for smaller counties.”

For example, federal agencies that need state data can choose to use the annual data, multiyear averages of the annual data, or 3-year or 5-year ACS averages. Federal agencies that also need county data can choose to use the most recent annual data for large counties and adjust the averages of the smaller counties to agree with annual data. Alternatively, they can choose to use various combinations of multiyear averages. As many federal agencies, as well as state and local governments, will be using the ACS data for allocating funds, Census Bureau guidance would reduce the inconsistent use of the multiple estimates.

HUD is a major user of Decennial Census long-form data for various program applications. Its contract with ORC Macro to review how the ACS will affect HUD programs that previously relied on the Decennial Census long form for geographic area data resulted in a report that made two points about the multiyear averages, in addition to raising the previously discussed issues on the inflation adjustment to income. One of these issues related to interpretations of changes in the multiyear averages and their stability; the other related to the availability of multiple estimates for the same area.

The ORC Macro report noted that year-to-year stability is important and needs to be addressed. It warned that the “differences in the precision of estimates or year-to-year changes in estimates can create problems for HUD applications.” The report used eligibility and level of benefits as an example of what could vary because of the effect of sampling variability on these changes. ORC Macro also stated: “The ACS will report data using different reporting periods for different sized areas. Inconsistent or multiple reporting periods can create problems for HUD applications.” ACS data for many geographic areas will be available in terms of annual estimates and 3- and 5-year averages, and the annual and 3-year averages (for larger areas) will be available before estimates for smaller areas. As a result, HUD will have to choose from multiple measures for some geographic areas. The study noted that HUD might decide to (1) continue to use 2000 long-form data until 2008, when the first 5-year average data will be available for all levels of geography, or (2) use the most recently available data in all cases.

ORC Macro’s report also expressed concern about the amount of annual ACS data that the Census Bureau will release for areas with populations smaller than 65,000, whose accuracy the Census Bureau has found does not meet publication standards. According to the study, the Census Bureau informed HUD that beginning in 2008, it would provide researchers and planners a “research file” containing annual ACS data for areas of all sizes, including census tracts. ORC Macro recommended that if the Census Bureau does release these data, HUD consider using these “unofficial” research file results in some of its applications. The study noted, however, that if HUD decided to use these unofficial data but other agencies decided not to use them, there would be no standardization across government programs in funding allocation where the same ACS items were used. “is a smoothed estimate; by averaging a particular time period’s data observation with those within a particular time window, the resulting estimate is meant to follow the general trend of the series but not be as extreme as any of the individual points. The ramifications of this basic concept emerge when moving average estimates are entered into sensitive allocation formulas or compared against strict eligibility cutoffs. A smoothed estimate may mask or smooth over an individual year drop in level of need, thus keeping the locality eligible for benefits; conversely, it may also mask individual-year spikes in activity and thus disqualify an area from benefits. It is clear that the use of smoothed estimates is neither uniformly advantageous nor disadvantageous to a locality; what is not clear is how often major discrepancies may occur in practice.” “It is incorrect to use annual estimates based on moving averages over several years when assessing change since some of the data are from overlapping time periods and hence identical. At the least, the results will yield incorrect estimates of the variance of the estimates of change. Therefore, users should be cautioned about this aspect of the use of moving averages.”

External Consultation

During the past decade’s development of the ACS, the Census Bureau has had many opportunities to consult with and take account of input from stakeholders and users in making key decision on the programs. It has (1) sponsored NAS panels, (2) held user conferences, (3) hired consultants to organize two conferences, (4) met regularly with its advisory committees and other user groups, and (5) encouraged its staff to present reports at ASA meetings and meetings of similar professional organizations. In the past several years, we and other federal agencies have reported on the ACS and provided recommendations to the Census Bureau. It established the ACS Federal Agency Information Program in 2003, responding to a recommendation we had made. It also announced last year that it was looking into establishing a partnership with the Congress and its oversight entities. “Eight years later, faced with the task of offering advice on making the vision of continuous measurement a reality in the 2010 census, the similarity between the arguments then and now is uncanny. Similar, too, are the points of concern; the current panel is hard-pressed to improve upon the basic summary of concerns outlined by our predecessors. We are, however, much more sanguine that a compelling case can be made for the ACS and that it is a viable long-form replacement in the 2010 census.”

The Census Bureau has neither responded to the panel’s first interim report in 2000 nor indicated that the recommendations were being adopted. The Census Bureau also has not responded to recommendations and issues raised by HUD and BLS. For example, it has not responded to HUD’s recommendations on the ACS adjustments to dollar-denominated items or to BLS’s recommendations on the ACS labor force questions. (On the issue of dollar-denominated items, we found no indication that the Census Bureau had ever consulted outside experts about either the methodology or the implementation.)

Census Bureau summaries of discussion about the ACS at its Advisory Committee meetings from October 2000 to April 2003 also indicate a lack of responsiveness. During this period, committee members raised concerns about the ACS. In particular, they made recommendations about many of the issues we have discussed in this report, including the evaluations of ACS and long-form comparisons, the new residence rules, independent controls, ICPE, group quarters, and Spanish-language questionnaires. At the April 2003 meeting, ASA committee members also requested a change in the structure of the Advisory Committee meetings, asking the Census Bureau to spend less time on update sessions and more time on sessions devoted to gathering more detailed input, commentary, and recommendations on topics the Census Bureau needs and wants advice on. Although the Census Bureau has addressed issues related to ICPE and Spanish-language questionnaires, the meeting summaries do not report that it followed recommendations in other areas.

Appendix VI: Comments from the Department of Commerce

Appendix VII: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

Additional staff who made major contributions to this report were Heather Von Behren, Penny Pickett, Mitchell Karpman, Michael Volpe, Andrea Levine, Patricia Dalton, and Robert Goldenkoff.

Bibliography

The first section in this bibliography lists documents on the history of the long form and mid-decade census. The remaining works are divided between numerous types of Census Bureau reports and papers, Association of Public Data Users papers, congressional hearings and testimony, and other reports and papers. Recent reports from the National Academy of Sciences are discussed in appendix II. Related GAO Products are listed in a separate section at the end of this report.

History of the Long Form and Mid-Decade Census

Alexander, Charles H. “Still Rolling: Leslie Kish’s ‘Rolling Samples’ and the American Community Survey.” In Proceedings of Statistics Canada Symposium 2001: October 16–19. Ottawa: Statistics Canada, 2002.

Anderson, Margo J., ed. Encyclopedia of the U.S. Census. Washington, D.C.: CQ Press, 2000.

House of Representatives, Committee on Post Office and Civil Service, Subcommittee on Census and Population. Review of Major Alternatives for the Census in the Year 2000. Serial 102-25. Washington, D.C.: August 1, 1991.

House of Representatives, Committee on Post Office and Civil Service. Census Confidentiality/Mid-Decade Sample Survey Bill. Report 93-246. Washington, D.C.: June 4, 1973.

House of Representatives, Committee on Post Office and Civil Service. Mid-Decade Censuses of Population, Unemployment, and Housing. Report 780. Washington, D.C.: August 12, 1965.

Salvo, Joseph, and Arun Peter Lobo. The American Community Survey: Quality of Response by Mode of Data Collection in the Bronx Test Site. Presented at 2002 Joint Statistical Meetings, New York City, August 14, 2002.

Census Bureau ACS Reports

American Community Survey Operations Plan, Release 1. Washington, D.C.: March 2003.

Meeting 21st Century Demographic Data Needs: Implementing the American Community Survey. Report 1. Demonstrating Operational Feasibility. Washington, D.C.: July 2001.

Meeting 21st Century Demographic Data Needs: Implementing the American Community Survey. Report 2. Demonstrating Survey Quality. Washington, D.C.: May 2002.

Meeting 21st Century Demographic Data Needs: Implementing the American Community Survey. Report 3. Testing the Use of Voluntary Methods. Washington, D.C.: December 2003.

Meeting 21st Century Demographic Data Needs: Implementing the American Community Survey. Report 4. Comparing General Demographic and Housing Characteristics With Census 2000. Washington, D.C.: May 2004.

Meeting 21st Century Demographic Data Needs: Implementing the American Community Survey. Report 5. Comparing Economic Characteristics With Census 2000. Washington, D.C.: May 2004.

Meeting 21st Century Demographic Data Needs: Implementing the American Community Survey. Report 6. The 2001-2002 Operational Feasibility Report of the American Community Survey. Washington, D.C.: May 2004.

Meeting 21st Century Demographic Data Needs: Implementing the American Community Survey. Report 7. Comparing Quality Measures: The American Community Survey's Three-Year Averages and Census 2000's Long Form Sample Estimates. Washington, D.C.: June 2004.

Meeting 21st Century Demographic Data Needs: Implementing the American Community Survey. Report 8. Comparison of the American Community Survey Three-Year Averages and the Census Sample for a Sample of Counties and Tracts. Washington, D.C.: June 2004.

Meeting 21st Century Demographic Data Needs: Implementing the American Community Survey. Report 9. Comparing Social Characteristics With Census 2000. Washington, D.C.: June 2004.

Meeting 21st Century Demographic Data Needs: Implementing the American Community Survey. Report 10. Comparing Housing Characteristics With Census 2000. Washington, D.C.: July 2004.

Census Bureau Advisory Committee Presentations

The presentations in this section were made by the Census Bureau’s Decennial Census Advisory Committee, Professional Association Advisory committees, and Race and Ethnic Advisory Committees.

The ACS: Data Products to Meet User Needs. Race and Ethnic Advisory Committee meeting, Washington, D.C., March 14, 2001.

Alexander, Charles, Alfredo Navarro, and Deborah Griffin. Update on ACS Evaluations. Decennial Census Advisory Committee meeting, Washington, D.C., November 5, 2001.

Gordon, Nancy. The American Community Survey. Joint Meeting of the Census Bureau Advisory Committees, Washington, D.C., July 28, 2000.

Gordon, Nancy. The American Community Survey. Decennial Census Advisory Committee meeting, Washington, D.C., September 21–22, 2000.

Gordon, Nancy. The American Community Survey. Race and Ethnic Advisory Committees meeting, Washington, D.C., November 2, 2000.

Gordon, Nancy. American Community Survey Update. Decennial Census Advisory Committee meeting, Washington, D.C., May 2, 2002.

Griffin, Deborah. An Overview of the Research and Evaluation Program for the American Community Survey. Decennial Census Advisory Committee meeting, Alexandria, Virginia, October 2–4, 2002.

Griffin, Deborah H. Comparing Characteristics from the American Community Survey and Census 2000: Methodology. Census Advisory Committee of Professional Associations meeting, Washington, D.C., April 10–11, 2003.

Navarro, Alfredo. American Community Survey: Use of Population Estimates as Controls in the ACS Weighting. Census Advisory Committee of Professional Associations meeting, Washington, D.C., October 23, 2003.

Navarro, Alfredo. A Discussion of the Quality of Estimates from the American Community Survey for Small Population Groups. Census Advisory Committee of Professional Associations meeting, Washington, D.C., October 2–3, 2002.

Weidman, Lynn, and Signe Wetrogan. Enhancing the Intercensal Population Estimates Program with ACS Data: Summary of Research Projects. Census Advisory Committee of Professional Associations meeting, Washington, D.C., October 23, 2003.

Census Bureau Continuous Measurement Series

The memorandums listed here from the 20 in the Continuous Measurement series are those most directly related to topics we review in this report.

Alexander, Charles H. A Continuous Measurement Alternative for the U.S. Census. CM-10, October 28, 1993. CM-11 summarized this paper at the 1993 annual meeting of the American Statistical Association, San Francisco, California, August 10, 1993.

Alexander, Charles H. Further Exploration of Issues Raised at the CNSTAT Requirements Panel Meeting. CM-13. Internal Census Bureau memorandum, Washington, D.C., January 31, 1994.

Alexander, Charles H. A Prototype Continuous Measurement System for the U.S. Census of Population and Housing. CM-17. Presented at the annual meeting of the Population Association of America, Miami, Florida, May 5, 1994.

Alexander, Charles H. Some Ideas for Integrating the Continuous Measurement System into the Nation’s System of Household Surveys. CM-19A. Internal Census Bureau memorandum, Washington, D.C., January 6, 1995.

Census Bureau Internal Reports

2004 Census Test Operational Plan. Washington, D.C.: September 29, 2003. 2010 Census Decision Memorandum Series No. 5, Finalizing Content for the 100 Percent Items in the 2010 Census and the American Community Survey. Washington, D.C.: June 3, 2004. 2010 Census Planning Memorandum Series No. 24, Action Plan: 2010 Research and Development Planning Group on Race and Ethnic Data Collection, Tabulation, and Editing. Washington, D.C.: June 9, 2004. 2010 Census Planning Memorandum Series No. 26, Action Plan: 2010 Research and Development Planning Group on Special Places/Group Quarters Development and Testing. Washington, D.C.: March 8, 2004.

ACS-2010 Consistency Review Plan. Washington, D.C.: October 1, 2003.

American Community Survey Development Report Series Program Plan. Washington, D.C.: rev. June 12, 2002.

Census 2000 Evaluation Reports

Abramson, Florence. Special Place/Group Quarters Enumeration. Census 2000 Testing, Experimentation, and Evaluation Program, Topic Report No. 5. U.S. Census Bureau, Washington, D.C., February 2004.

Adlakha, Arjun, J. Gregory Robinson, Kirsten West, and Antonio Bruce. Assessment of Consistency of Census Data with Demographic Benchmarks at the Subnational Level. Census 2000 Evaluation O.20. U.S. Census Bureau, Washington, D.C., August 18, 2003.

Clarke, Sandra, John Iceland, Thomas Palumbo, Kirby Posey, and Mai Weismantle. Comparing Employment, Income, and Poverty: Census 2000 and the Current Population Survey. Census 2000 Auxiliary Evaluation. U.S. Census Bureau, Washington, D.C., September 2003.

Palumbo, Thomas and Paul Siegel. Accuracy of Data for Employment Status as Measured by the CPS-Census 2000 Match. Census 2000 Evaluation B.7. U.S. Census Bureau, Washington, D.C., May 4, 2004.

Schneider, Paula. Content and Data Quality in Census 2000. Census 2000 Testing, Experimentation, and Evaluation Program, Topic Report No. 12. U.S. Census Bureau, Washington, D.C., January 22, 2004.

Census Bureau 2003 JSM Staff Papers

Bureau staff presented many ACS-related papers at the August 2003 Joint Statistical Meetings in San Francisco, California. We reviewed the papers in this section in detail because they were related to comparisons between ACS estimates and 2000 Census results.

Boggess, Scott, and Nikki L. Graf. Measuring Education: A Comparison of the Decennial Census and the American Community Survey. Presented at Joint Statistical Meetings, San Francisco, California, August 7, 2003.

Dye, Jane Lawler. Grandparents Living with and Providing Care for Grandchildren: A Comparison of Data from Census 2000 and 2000 American Community Survey. Presented at Joint Statistical Meetings, San Francisco, California, August 7, 2003.

Love, Susan, and Deborah Griffin. A Closer Look at the Quality of Small Area Estimates from the American Community Survey. Presented at Joint Statistical Meetings, San Francisco, California, August 4, 2003.

Posey, Kirby G., Edward Welniak, and Charles Nelson. Income in the American Community Survey: Comparisons to Census 2000. Presented at Joint Statistical Meetings, San Francisco, California, August 7, 2003.

Raglin, David A., Theresa F. Leslie, and Deborah H. Griffin. Comparing Social Characteristics between Census 2000 and the American Community Survey. Presented at Joint Statistical Meetings, San Francisco, California, August 3, 2003.

Stern, Sharon M. Counting People with Disabilities: How Survey Methodology Influences Estimates in Census 2000 and the Census 2000 Supplementary Survey. Presented at Joint Statistical Meetings, San Francisco, California, August 7, 2003.

Other Census Bureau Staff Research Papers

Alexander, Charles H. American Community Survey Data for Economic Analysis (October 2001). Presented at the Census Advisory Committee of the American Economic Association meeting, Washington, D.C., October 18–19, 2001.

Alexander, Charles H. Recent Developments in the American Community Survey. Presented at the 1998 Joint Statistical Meetings, Dallas, Texas, August 12, 1998.

Alexander, Charles H., Sharon Brown, and Hugh Knox. American Community Survey Data for Economic Analysis (December 2001). Presented at the Federal Economic Statistics Advisory Committee meeting, Washington, D.C., December 14, 2001.

Alexander, Charles H., Scot Dahl, and Lynn Weidman. Making Estimates from the American Community Survey. Presented at the 1997 Joint Statistical Meetings, Anaheim, California, August 13, 1997.

Alexander, Charles H., and Signe Wetrogan. Integrating the American Community Survey and the Intercensal Demographic Estimates Program. Presented at the 2000 Joint Statistical Meetings, Indianapolis, Indiana, August 14, 2000.

Butani, Shail, Charles Alexander, and James Esposito. Using the American Community Survey to Enhance the Current Population Survey: Opportunities and Issues. Presented at the 1999 Federal Committee on Statistical Methodology Research Conference, Arlington, Virginia, November 15-17, 1999.

Davis, Mary Ellen, and Charles H. Alexander, Jr. The American Community Survey: The Census Bureau's Plan to Provide Timely 21st Century Data. Missouri Library World, Spring 1997.

DeMaio, Theresa J., and Kristen A. Hughes. Report of Cognitive Research on the Residence Rules and Seasonality Questions on the American Community Survey. U.S. Bureau of the Census, Statistical Research Division, Washington, D.C., July 2003.

Love, Susan, Donald Dalzell, and Charles Alexander. Constructing a Major Survey: Operational Plans and Issues For Continuous Measurement. Presented at the 1995 Joint Statistical Meetings, Orlando, Florida, August 16, 1995.

Nelson, Charles, and Kathleen Short. The Distributional Implications of Geographic Adjustment of Poverty Thresholds. U.S. Bureau of the Census, Housing and Household Economics Statistics Division, Washington, D.C., December 2003.

Posey, Kirby G., and Edward Welniak. Income in the ACS: Comparisons to the 1990 Census. Presented at the American Community Survey Symposium, Suitland, Maryland, March 1998.

Salvo, Joseph, and Arun Peter Lobo. The American Community Survey: Quality of Response by Mode of Data Collection in the Bronx Test Site. Presented at the 2002 Joint Statistical Meetings, New York, August 14, 2002.

Smith, Amy Symens. The American Community Survey and Intercensal Population Estimates: Where Are the Crossroads? Technical Working Paper 31, U.S. Census Bureau, Population Division, Washington, D.C., December 1998.

Association of Public Data Users Papers

Davis, Mary Ellen. The American Community Survey Data Products, Alexandria, Virginia, October 20, 2003.

Gage, Linda, State of California, Department of Finance. American Community Survey: Research by the Data User Community. Alexandria, Va.: October 20, 2003.

Petroni, Rita. How Do 3-Year Averages from the ACS Compare to Census 2000 Data? (Preliminary Results). Alexandria, Va.: October 20, 2003.

Salvo, Joseph, City of New York, Planning Department. American Community Survey: Research by the Data User Community. Alexandria, Va.: October 20, 2003.

Scarr, Harry A. Deputy Director, Census Bureau. Continuous Measurement. Association of Public Data Users, Washington, D.C.; October 16, 1994.

Congressional Hearings and Testimony

Barron, William Jr., Acting Director, U.S. Bureau of the Census, before the U.S. House of Representatives, Committee on Government Reform, Subcommittee on the Census. The Census Bureau’s Proposed American Community Survey (ACS), Serial 107-9. Washington, D.C.: June 13, 2001.

Kincannon, Charles Louis, Director, U.S. Bureau of the Census, before the U.S. House of Representatives, Subcommittee on Technology, Information Policy, Intergovernmental Relations, and the Census. The American Community Survey: The Challenges of Eliminating the Long Form from the 2010 Census, Serial 108-97. Washington, D.C.: May 13, 2003.

Prewitt, Kenneth, Director, U.S. Bureau of the Census, before the U.S. House of Representatives, Committee on Government Reform, Subcommittee on the Census. House Hearing on ACS July 20, 2000. The American Community Survey: A Replacement for the Census Long Form? Serial 106-246. Washington, D.C.: July 20, 2000.

Other Reports and Papers

Kalton, Graham, and others. The American Community Survey: The Quality of Rural Data, Report of a Conference. Rockville, Md.: Westat, June 29, 1998.

Nardone, Thomas, and others. Examining the Discrepancy in Employment Growth between the CPS and the CES. Washington, D.C.: FESAC, October 17, 2003.

National Council on Disability. Improving Federal Disability Data. Washington, D.C.: January 9, 2004.

ORC Macro. The American Community Survey: Challenges and Opportunities for HUD. Calverton, Md.: September 27, 2002.

Vroman, Wayne. Comparing Labor Market Indicators from the CPS and ACS. Washington, D.C.: Urban Institute, September 2003.

Westat Inc. The American Community Survey: A Report on the Use of Multi-Year Averages. Rockville, Md.: April 30, 1999.

Related GAO Products

2010 Census: Cost and Design Issues Need to Be Addressed Soon. GAO-04-37. Washington, D.C.: January 15, 2004.

Medicaid Formula: Differences in Funding Ability among States Often Are Widened. GAO-03-620. Washington, D.C.: July 10, 2003.

Formula Grants: 2000 Census Redistributes Federal Funding Among States. GAO-03-178. Washington, D.C.: February 24, 2003.

Major Management Challenges and Program Risks: Department of Commerce. GAO-03-97. Washington, D.C.: January 1, 2003.

The American Community Survey: Accuracy and Timeliness Issues. GAO-02-956R. Washington, D.C.: September 30, 2002.

Legal Authority for American Community Survey. B-289852. Washington, D.C.: April 4, 2002.

Medicaid Formula: Effects of Proposed Formula on Federal Shares of State Spending. GAO/HEHS-99-29R. Washington, D.C.: February 19, 1999.

Decennial Census: Overview of Historical Census Issues. GAO/GGD-98-103. Washington, D.C.: May 1, 1998.

Poverty Measurement: Adjusting for Geographic Cost-of-Living Difference. GAO/GGD-95-64. Washington, D.C.: March 9, 1995.

Status of the Statistical Community after Sustaining Budget Reductions. GAO/IMTEC-84-17. Washington, D.C.: July 18, 1984.