The American Community Survey Design Issues And Initial Test Results

Charles H. Alexander, U.S. Bureau of the Census[^1]

The American Community Survey (ACS) will be a large continuing survey of
the U.S. population using a Arolling sample@ design of the type
described in Kish (1990). It is a mail survey with follow-up by
telephone and personal visit of a sample of nonrespondents. The Census
bureau=s ongoing Master Address File operation provides the frame. The
ACS collects information about the same topics as the census Along form@
content sample questionnaire. After a period of testing and comparison
to the 2000 census long form, the ACS sample will increase to about
three million mailouts each year starting 2003, leading to replacement
of the content sample in the 2010 census. This paper describes the ACS
survey design, objectives, and estimation procedures along with the
considerations that led to them. The major methodological issues in
conducting the survey and assessing the quality of the data are
outlined. Initial results from the 1996 tests in four counties are
presented, along with plans for future research and testing.

Key words: Rolling sample, small area, survey coverage

INTRODUCTION

The American Community Survey (ACS) is a rolling sample survey being
developed by the U.S. Bureau of the Census as an eventual replacement
for the decennial census Along form@ survey that provides the detailed
economic, social, and housing characteristics of communities throughout
the U.S. The ACS will cover the same topics as the census long form, but
instead of contacting about 17,000,000 addresses at one time, the ACS
will mail to about 3,000,000 addresses each year throughout the decade.

The ACS design has two main estimation objectives:

a\) provide descriptive profiles for communities of all sizes with mean
squared error (MSE) generally similar to the census long form estimates,
but updated throughout the decade;

b\) provide a time series of annual estimates for communities well below
the State level, to measure changing local conditions.

After Census 2000, the ACS will replace the census long form sample as
the source of detailed estimates of the characteristics of small areas.
There will still be a decennial census to get a population and housing
Acount.@

There has been a long-standing interest in updating the census
descriptive profiles (Melnick, 1991 or Sawyer, 1993) in part because of
their role in allocating Federal funds to local areas. The interest in
tracking changes for sub-state areas has increased because of recent
political developments in the U.S. sometimes referred to as Adevolution@
of decision-making to state and local governments.

To meet these two objectives, the ACS uses a rolling sample design
suggested by Kish (1990, 1981) with what Kish calls Aasymmetrical
cumulation@ of the survey data, i.e. cumulating different numbers of
years of data for different geographic levels or different uses. For the
first objective, to produce estimates comparable in variance and bias to
census long form estimates, 5 years of data would be cumulated to get
sufficient sample sizes. For the second objective, to track changes for
states and sub-state areas, single-year estimates would be used with the
understanding that the larger sampling variance would mean that only
very large changes could be detected for the smallest domains such as
census tracts. The ACS questionnaire will initially cover the same
topics as the 2000 census long form. After 2002, additional topics might
be added. Participation in the ACS by sample households is required by
law.

The Census Bureau views the ACS as part of a larger Acontinuous
measurement@ program, including benefits for the statistical programs of
other Federal agencies. Besides the direct ACS data collection, this
includes:

a\. use of ACS information to improve the Bureau=s intercensal
demographic estimates, which are in turn used in weighting the ACS;

b\. use of the ACS field staff to help update MAF/TIGER, keeping it
complete enough for an intercensal survey frame;

c\. use of the ACS data and sampling frame to improve the estimation
from various household surveys conducted by the Census Bureau, such as
the Current Population Survey (CPS), the Survey of Income and Program
Participation (SIPP) or the National Health Interview Survey (NHIS);

d\. development of model-based small-area estimates for specific
characteristics using data from existing household surveys,
administrative records and eventually the ACS; the Census Bureau=s Small
Area Income and Poverty Estimates (SAIPE) program is an example, as is
the Bureau of Labor Statistics Local Area Unemployment Statistics (LAUS)
program (Brown, 1997).

The ACS is being introduced in stages to allow review of the data,
refinement of the operations, and a comparison with the Census 2000 long
form before proceeding with a full introduction of the new survey. The
stages are as follows:

1996-1998 Demonstration and Testing Period (four sites in 1996, eight in
1997, nine in 1998)

1999-2001 Thirty-seven Acomparison sites@ with 5% annual sample

2000-2002 National comparison sample with overall rate of 0.7% annually

2003-on Full introduction (three million addresses per year, including
all counties).

ACS DESIGN AND OPERATIONS

[Sample Design:]{.underline} The ACS uses a Arolling@ sample design with
each address being interviewed at most once in a 5 year period. Each
year=s sample addresses will be spread evenly across the 12 months of
the year and, starting in 2003, across the entire nation. The sample is
in general not clustered, although there may be some exceptions in areas
with unusually high travel costs and in group quarters.

The sampling frame will be the Census Bureau=s Master Address File
(MAF). The MAF is being created for the 2000 census, but is being
created early for the ACS test sites. It starts with the 1990 census
Address Control File, which is linked to the TIGER geographic database.
This is updated using Postal Delivery Systems Files (DSF) in areas where
the DSF addresses can be geocoded based on a Acity-style@ house number
and street name. In other areas, the MAF must be created by physically
listing each block. After Census 2000, the MAF will be kept up to date
by regular matches to the DSF, at least every 6 months, by additional
listing in rural areas of high growth as identified by administrative
records systems. The sample will be selected as a systematic sample from
the MAF, including only addresses not selected in the previous four
years.

[Data Collection:]{.underline} For each monthly sample panel, the ACS
starts by mailing a questionnaire to each address about 10 days before
the start of the Amailout month.@ There is a Apre-notice@ letter, an
initial mail questionnaire, and a reminder card, one week apart. After
about 3 weeks, a replacement questionnaire is mailed if no response has
been received.

At the beginning of the following month, nonresponding addresses are
assigned to telephone nonresponse follow-up. For addresses where the
telephone number can be obtained from commercial directories, a
telephone interview is attempted. Mail returns continue to come in
during this second month; in 1996 about 19% of the telephone nonresponse
follow-up group were removed from follow-up because of a late mail
return.

The third month, any addresses still not interviewed are eligible for
personal-visit follow-up. This includes addresses where no telephone
number could be located, as well as addresses where there was a number
but no interview could be obtained. One-third of these addresses are
selected for follow-up by personal visit. Note that this includes most
of the vacant addresses.

As an example, the March mailout panel has telephone follow-up in April,
and personal-visit follow-up in May. In general, a new area introduced
in a particular year starts with mailouts in November and December of
the previous year, so that the normal pattern of follow-up work in
January and February is in place by the time the year starts.

Roughly speaking, based on the results of the 1996 and 1997 tests, we
expect a National average of about 70% of the sampled addresses to be
completed by mail or telephone and about 1/3 of the remaining 30% to be
selected for personal visit followup.

The mail returns undergo a clerical edit which includes determining

1\) if the form is missing enough responses to require a callback;

2\) if an initial write-in entry giving the number of persons at the
address was inconsistent with the number actually included in the
questionnaire;

3\) if more than 5 people were listed as living at the address, since
the form only collects characteristics for 5 people.

The first condition is referred to as the Acontent edit@ and the last
two as the Acoverage edit.@ If the form fails either edit, then there is
a telephone callback that attempts to fill in all missing data and
straighten out any coverage problems. The exception is that if the
content edit finds that the form is completely blank, the case continues
to nonresponse follow-up as if no form had been returned. In 1996, about
half of the mail returns failed one or more edits; of these 96% gave a
telephone number, and 92% of these had some further resolution from
telephone followup.

[Residence Rule and Reference Period]{.underline}: The residents of any
sample address, and their characteristics, will be determined as of the
time of data collection. This refers either to when the mail form is
filled out or when the nonresponse follow-up interview takes place. The
ACS currently uses a A2-month@ rule to determine who is a Acurrent
resident@ of an address. Anyone staying at the address more than two
months is included as a current resident. People staying two months or
less would also be included, unless they usually live somewhere else.

ACS ESTIMATION METHODS

[ACS Weighting]{.underline}: The 1996 ACS estimates were weighted using
fairly conventional survey methods combining features of the census
long-form weighting and household surveys such as the Current Population
Survey (CPS). The details are given in Alexander, Dahl, and Weidman
(1997).

[Edit and Imputation]{.underline}: The editing and imputation for the
1996 ACS was similar to that for the 1990 census long form, although
there are minor improvements for specific data items, mainly associated
with minor questionnaire changes that are also being considered for the
2000 census.

[Variance Estimation]{.underline}: The sampling variances for ACS
estimates have been estimated using replication methods similar to those
used for the CPS, with reweighting of each replicate to account for the
effect of population and housing controls. For some items, the variance
was also estimated using the random group method used for the 1990
census. The results of the two methods are still being compared, but an
initial review showed the results to be generally similar. As with the
1990 census long form, the ACS variance estimates do not include
variance due to imputation of missing data. This will be remedied as
soon as possible. The ACS variances will mainly be reported to the users
in the form of Ageneralized variance functions@ (GVF), which approximate
the variance as a Adesign factor@ times the corresponding variance from
a simple random sample. Different groups of characteristics have
different design factors. Work on the 1996 GVFs is still underway.

RELATIONSHIP TO OTHER FEDERAL GOVERNMENT SURVEYS

The ACS will provide a valuable Astatistical infrastructure@ to improve
the estimates and operations for other Federal government household
surveys. The ACS data can be used in weighting and sampling for these
surveys, as census long form data have traditionally been used, but the
ACS will be more up-to-date. The sample for other surveys can be
supplemented with housing units having specific demographic or economic
characteristics taken from recently interviewed ACS sample units. When
the survey samples are redesigned after the 2000 census, the MAF will
provide more flexibility for drawing additional sample between censuses.
The ACS will also provide auxiliary variables to improve small area
models such as those used for the Local Area Unemployment Statistics
(LAUS) or Small-Area Income and Poverty Estimates (SAIPE) programs.

However, the ACS will not replace the need for the CPS to measure
unemployment, the SIPP to measure the dynamics of income and poverty, or
other special-purpose surveys such as the NHIS or the National Crime
Victimization Survey (NCVS). The subject matter of these surveys
requires specialized questions that are too complicated for a
self-response mail questionnaire. The ACS design is not suitable for
measuring month-to-month changes as does the CPS, nor for following
people over a period of time as does the SIPP.

There has been concern about possible confusion between the more
accurate national and state labor force estimates from the CPS, which is
designed especially to measure labor force status and the corresponding
ACS estimates. The Census Bureau and Bureau of Labor Statistics are
working together to avoid such confusion (Brown, 1997). The ACS will not
produce monthly unemployment estimates. For annual estimates, the ACS
will provide a set of Aadjusted@ unemployment and civilian labor force
(CLF) responses that will give unemployment and CLF estimates agreeing
exactly with CPS at the national level, and conforming more closely to
the CPS estimates at the state level.

The potential benefits of the ACS for other Federal government surveys,
and its limitations, are discussed in more detail in Brown (1997).

WHY THIS DESIGN?

[The Alternative of Expanding the CPS]{.underline}. An alternative to
having a separate mail survey like the ACS to produce intercensal
small-area data would be to expand the CPS sample and have one large
personal-visit survey to produce both annual small-area data and
estimates of short-term change in labor force characteristics. A rolling
sample design with approximately 2 million addresses per year (without
sampling for nonresponse followup) could in theory serve both
objectives. This is about four times the current monthly CPS sample, but
would not dramatically reduce the variance of monthly change estimates
because the high correlation between monthly estimates with the current
CPS rotating panel design would be lost.

The unit cost of this design would be substantially higher than the
current CPS. The CPS now uses a cluster sample, conducts most interviews
by telephone using the phone numbers obtained on the first of eight
interviews at each address, and has a shorter interview than a combined
ACS/CPS survey would require. We think that a survey of this design
would cost several times as much as the combined cost of the current CPS
plus the projected \$75 million annual cost of the ACS operations.

To make a compromise CPS/ACS design affordable, it would be necessary to
give up on some of the objectives, sacrificing either top-quality
monthly measurement of unemployment, or Along form@ data for small areas
such as census tracts. A review of uses of census data (Edmonston and
Schultze, 1995) showed that the long-form small-area estimates were
necessary to meet legislative requirements, and the importance of the
monthly unemployment estimates for economic decision-making is well
established.

[The Alternative of a Mid-Decade Census]{.underline}. With the failure
of a mid-decade census to be funded for 1985 or 1995, this alternative
was not extensively considered in the ACS development. For the purpose
of updating census profiles for small areas, a mid-decade sample census
is arguably as effective as the five-year averages proposed for the ACS.
However, a quinquennial Asnapshot@ is not effective for monitoring
year-to-year changes, the second major use of the ACS. This new use
seems to have made the difference in obtaining support for the ACS.

[Alternatives Relying Mainly on AIndirect@ Estimation]{.underline}.
Another alternative would have been to rely less on Adirect@ estimation
from a large survey and more on Aindirect@ model-based methods combining
information from administrative records and smaller surveys. The
Asmaller surveys@ could be a modification of CPS and existing surveys,
or they might include a smaller mail survey such as a much reduced ACS.

As mentioned in Section I, research on such methods is part of the
Continuous Measurement program. As these methods develop and become
accepted by data users, we hope some of the ACS sample will gradually be
replaced by information from statistical models. However, the
development of these methods is not far enough along to eliminate the
need for large samples to produce estimates of a variety of
characteristics for very small areas.

[Why This Data Collection Design]{.underline}? The uniform spread of the
sample was needed to provide comparable estimates for all levels of
geography each year. A precursor of the ACS with different areas in
different years was previously explored (Herriot, Bateman and McCarthy,
1989), but was rejected because of the difficulty in making comparisons
across areas.

The choice of a mail survey with followup was based on experience with
this design in the census. The particular multiple-mailings approach was
selected based on research following the 1990 census. The limitations of
the questions that can be asked by mail are not a barrier since the ACS
objectives involve the topics covered by the census long form survey,
which is also done by mail.

We decided not to have any clustering of the sample, rather than to use
small Aultimate clusters@ of, say, four adjacent addresses as does CPS.
The unclustered design is more efficient for the mail survey. The
relatively high mail response rate means that a large initial cluster
would be needed to have an expected four, or even two, in a cluster for
followup. We are still considering clustering of followup cases in
remote areas by reassigning the followup cases in a particular area to
the same month of interview.

The data collection procedures for the telephone and personal-visit
followup interviews were adapted from those used for CPS and other
Census Bureau household surveys. The subsampling rates were chosen
considering the relative costs per interview for the mail, telephone,
and personal-visit modes using the rule that the allocation should be
inversely proportional to the square root of cost but rounding to
whole-number sub-sampling rates (no subsampling for telephone and 1 in 3
for personal visit).

VARIANCE AND BIAS OF THE ACS ESTIMATES

The ACS was designed with certain tradeoffs in mind between sampling
error, frequency of updating the data, various sources of measurement
error, and issues of interpreting and using the data.

The intended tradeoffs for the smallest communities are as follows:

1\) Sampling error: standard errors 1.25 times as large as the long form
design for Atypical@ estimates for small areas;

2\) Frequency of updating: annual rather than decennial;

3\) Issues of interpreting and using the data: 5-year average rather
than point-in-time;

4\) Other nonsampling errors: roughly equivalent, with each design
having relative strengths and weaknesses.

For larger domains, where 5 years of ACS sample is more than enough for
many purposes, annual estimates or shorter averages can be used. In this
case, sampling error and the interpretation issues concerning the
multi-year averages are less important, and other nonsampling errors are
relatively more important.

In presentations of the ACS plans, these intended quality tradeoffs have
been described to potential ACS data users. Now that the preliminary
1996 data are available, we can begin to verify the first statement
about the standard errors and the fourth about nonsampling errors. The
remainder of this section gives a first look at the preliminary results.
All these conclusions must be regarded as very tentative, since they are
based on preliminary data, and a small non-probability sample of test
areas.

[Sampling Error: ACS Compared to 1990 Long Form]{.underline}. A
preliminary comparison of the ACS standard errors for tracts, to the
corresponding A1990 census standard errors@ computed using the 1990
generalized variance function (GVF) assuming a 1-in-6 sampling rate, is
generally consistent with the anticipated 1.25 ratio. However, the
results show that a simplistic rule like A1.25 times as large@ is only a
general guide. The ratio is higher for items that are concentrated in
the Anonresponse universe@\--that portion of the population that would
not respond by mail or telephone\--because of the ACS=s use of sampling
for nonresponse followup. The extreme case is for vacant units, which
are almost all collected by personal visit. We are still working on
summarizing this variance comparison and extrapolating to what can be
expected from the 2003 ACS.

[Evidence From the 1996 Test About Nonsampling Errors.]{.underline}
Rather than attempt to quantify the net bias in ACS estimates in a
Atotal error@ analysis, we adopt the less ambitious approach of looking
for evidence of specific quality problems and quantifying them
separately, in some cases with only indirect measures.

The final weighted unit response rates were quite high, running at 98.2%
in the 1996 test areas and 98.7% in the 1997 areas except Houston, from
January through July 1977, and 97.2% in Houston, TX, which has many
hard-to-enumerate areas. By contrast, long-form data were collected for
only 91.5% of sample units in the 1990 census, although all the rest had
the basic census count information collected. The ACS final response
rates in Rockland, NY were uniformly high for all tracts although mail
return rates varied dramatically. (See Salvo and Lobo, 1997.)

Salvo and Lobo (1997) argue that a more valid comparison is to look at
what proportion of households complete a specific questionnaire item,
i.e., they look at the combined effect of Aunit@ and Aitem@ nonresponse.
In the Rockland, NY test site, they found that item response rates for
cases assigned to follow-up were uniformly at least as high for the ACS
as for the 1990 long form and were substantially higher for some items.
This was expected, as a product of having a permanent field staff.

We are not prepared to draw a conclusion about whether the ACS suffers
from the same kinds of overall undercoverage of persons relative to
census-based intercensal demographic estimates that is seen for the CPS
and many other household surveys. Comparisons of ACS weighted population
estimates prior to post-stratification come within a few percent of the
intercensal population estimates for the test counties. The population
estimates include an adjustment for undercount in the 1990 census (Table
1, row 1). However, it is possible that these results are overly
favorable if the vacancy rate is under-estimated, as discussed below.

There is still reason for concern about differential undercoverage of
non-white persons and persons of Hispanic origin. Results were mixed on
this, with some sites showing undercoverage of these groups and others
showing good coverage and in one case overcoverage (Table 1). The latter
anomaly suggests that the race/Hispanic-origin breakdown in the
population controls may not have fully captured changes since the
census. (Alexander, Dahl, and Weidman, 1997). If so, this demonstrated a
situation where the ACS can provide information useful in improving the
population controls. This analysis is complicated by differences in the
race/origin categories between the ACS and the 1990 census, another
reason that we are not prepared to draw conclusions without further
study.

Table 1

Ratio of ABefore PPSF@ Estimate

Divided by AAfter PPSF@ Estimate

  ---------------- -------------- -------------- ------------ ------------
  Site                                                        

                   Rockland       Multnomah      Brevard      Fulton
                   County         County         County       County

  Total Persons    .975           .987           .958         .941

  Race = ABlack@   1.044          .781           .861         \*

  Race = AOther@   .974           .985           .898         \*

  Hispanic Origin  .930           1.162          .849         \*
  ---------------- -------------- -------------- ------------ ------------

\* Sample too small for a reliable estimate

Comparisons of the 1996 ACS estimates with 1990 census estimates at the
county level showed few differences that suggested methodological
problems. The most salient concern is that the ACS had noticeably lower
vacancy rates. Somewhat lower rates are expected because the residence
rules reduce the number of Avacant-usual residence- elsewhere@
situations, but there is a possibility that the vacancy rate could be
underestimated because of the long period allowed for ACS follow-up,
with units that are vacant at the time of mailout becoming occupied by
the time of follow-up. This is being studied further.

Income, poverty, and other economic data were not ready in time for this
anaylsis. There is concern that asking the income questions Afor the
last 12 months@ throughout the year may give less accurate recall than
asking for Athe last calendar year@ in April. A small test comparing
Alast 12 months@ and Alast calendar year@ is being conducted at the end
of 1997, and may give some insights about this issue.

Examination of changes for individual tracts has just begun. One
dramatic error was found in a small tract in the Rockland test site
where a geocoding error on the MAF caused the number of addresses to
drop dramatically between1990 and 1996. This illustrates the need to
feed information about address problems from the ACS data collection
process back to the MAF updating process. The ability to detect such MAF
errors is a potential benefit of the ACS, but we do not yet have this
system in place.

A fundamental question is how funds can be allocated equitably based on
the most recent data when the best estimate for large cities may be for
the previous year while for small places it may be for the previous 5
years. Two solutions have been suggested. The first is to use the
longest average (5 years in most cases) for all areas. The second is to
allocate funds to large areas using one year data and then within the
large areas based on multi-year averages. The large areas could include
collections of small rural counties in addition to large cities or
metropolitan areas. This question also needs to be widely discussed.

[Annual Average Data.]{.underline} The differences between the ACS and
the long form survey associated with collecting data all year with a
moving reference date, rather than in the few months after the census
using a fixed reference date, may actually be more important than the
use of multi-year averages. However, these differences have generated
less concern among users, perhaps because annual averages are more
familiar from other household surveys such as CPS. The 1999-2001
comparison sites have been selected to represent areas in which various
sources of differences are expected to be especially important and we
expect our understanding of these differences to grow as time goes on.

REFERENCES

Alexander, C. H., Dahl, S. and Weidman, L. (1997). AMaking Estimates
from the American

> Community Survey.@ Presented at the 1997 Annual Meetings of the
> American Statistical Association.

Brown, S. (1997). APotential Uses of Continuous Measurement in BLS Labor
Force Programs.@

> Presented at the 1997 Annual Meetings of the American Statistical
> Association.

Center for Study of Social Policy (1995). AMaking Decisions Count: How
the Census Bureau=s

> New Survey Could Transform Government.@ 1250 Eye Street, NW,
> Washington, DC.

Chand, N. and Alexander, C.H. (1997). AAchieving Agreement in the
American Community

> Survey and the Current Population Survey.@ Presented at the 1997
> Annual Meetings of the American Statistical Association.

Edmonston, B. and Schultze, C. (1995), editors. Modernizing the U.S.
Census. National

> Academy Press, Washington, DC.

Herriot, R.A., Bateman, D.B. and McCarthy, W.F. (1989). AThe Decade
Census Program\--A

> New Approach for Meeting the Nation=s Needs for Sub-National data.@
> Proceedings of the American Statistical Social Statistics Section, pp.
> 351-355.

Kish, L. (1981). APopulation Counts from Cumulated Samples.@ In Using
Cumulated Rolling

> Samples to Integrate Census and Survey Operations of the Census
> Bureau, U.S. Government Printing Office, Washington, DC., June 26,
> 1981, pp. 5-50.

Kish, L. (1990). ARolling Samples and Censuses.@ Survey Methodology, 16,
1, pp. 63-79.

Melnick, D. (1991). AThe Census of 2000 A.D. and Beyond.@ In Review
Major Alternatives for

> the Census in the Year 2000, U.S. Government Printing Office,
> Washington, DC, August 1, 1991, pp. 60-74.

Ott, K., Parmer, R.J., Loudermilk, C., McMillan, Y., Reilly, B., and
Coughlin, T. (1997).

> AEvaluation of the Census Bureau=s Master Address File Using National
> Health Interview Address Listings.@ Presented at the 1997 Annual
> Meeting of the American Statistical Association, Survey Methods
> Research Section.

Salvo, J and Lbob, P. (1997). AThe American Community Survey:
Nonresponse Follow-up in the

> Rockland County Test Site.@ Presented at the 1997 Annual Meetings of
> the American Statistical Association.

Sawyer, T.C. (1993). ARethinking the Census: Reconciling the Demands for
Accuracy and

> Precision in the 21st Century.@ Presented at the Research Conference
> on Undercounted Ethnic Populations, Bureau of the Census, May 7, 1993.

U.S. Department of Transportation, Bureau of Transportation Statistics
(1996). *Implications of*

> *Continuous Measurement for the Uses of Census Data in Transportation
> Planning.* Washington, DC, April 1996.

/media/pszemraj/PeterTrove/project-files/DATA/ms-word-files/big-DOC/187170.doc

[^1]: Charles H. Alexander, Bureau of the Census, Room 3705-3, Suitland,
    MD 20746. This paper reports the general results of research
    undertaken by Census Bureau staff. The views expressed are
    attributable to the author and do not necessarily reflect those of
    the Census Bureau.