Variance Reduction for Sub-County Estimates

in the American Community Survey

Prepared for the Spring 2007 meetings of the

Census Advisory Committee of Professional Associations

Robert Fay

The views expressed on statistical, methodological, technical, or
operational issues are those of the

author and not necessarily those of the U.S. Census Bureau.

Variance Reduction for Sub-County Estimates

in the American Community Survey

Robert Fay

**Questions:**

1.  In your positions both as researchers and as members of the public,
    are you concerned about this potential use of administrative records
    in the ACS?

2.  The initial evidence is that, although this approach reduces the
    variance of many estimates, the benefit to income statistics such as
    median income and poverty rates appears minimal. The data used so
    far are exclusively demographic: age, sex, race, and Hispanic
    origin. In fact, the extract files used in the research contain no
    income data. If the administrative sources were expanded to include
    income data, it is possible that ACS income estimates could be
    improved. Would you see this expanded use of administrative records
    as beneficial or too sensitive to undertake?

3.  The weighting of the Canadian census long questionnaire in 1991 and
    subsequent censuses is a clear precedent for this work. Are there
    other applications that could guide the remaining research on this
    application to ACS?

4.  There are other calibration estimators that are alternatives to
    generalized regression estimation (GREG). Do any seem to have
    practical advantages over GREG that would be helpful in this
    application?

5.  Are there appropriate model-assisted estimators that could improve
    on the calibration approach used here? In weighting the Canadian
    census sample, calibration with complete count estimates is a
    primary goal. In this application, the primary goal is variance
    reduction, and the calibration goal is secondary because the
    administrative data will not be published. Does this approach lead
    to the appropriate optimization? By dropping calibration goals,
    would an alternative estimator reduce variance further?

# Introduction

Since 1999, the American Community Survey (ACS) has been implemented in
34 test counties at rates roughly comparable to the sampling rates used
in the full ACS production sample, which started in 2005. In fact,
during the period 1999-2001, the rates in the 34 counties were set high
enough to provide roughly the same sample size as expected during a
5-year period in ACS production. Consequently, the test counties
provided data down to the tract level for comparison to Census 2000.
(Census tracts average roughly 4000 persons.) External collaborators
examined the test ACS data and reported their findings. Among them, Paul
Voss and his colleagues (Van Auken et al. 2004) noted that while ACS
county-level variances were close to what was expected given the
sampling rates, ACS tract-level variances were disappointingly higher
than the corresponding Census 2000 variances. They observed this
difference in the two Wisconsin counties they studied, but Census Bureau
staff soon replicated their finding across the set of 34 counties
(Starsinic 2005).

The reasons for this shortfall in reliability was clear in hindsight.
Projections of ACS reliability were based on sample-size relationships
with the decennial long form. But weights used in decennial estimation
are controlled to 100% census counts in each weighting area; census
weighting areas were often, although not always, coterminous with census
tracts. ACS estimates are controlled only at the county level and above.
Thus, decennial tract-level estimates are advantaged by variance
reductions unavailable to the ACS.

The goal of this research is to achieve variance reductions at the tract
level and other sub-county levels by combining administrative records
and model-assisted estimation. The preliminary results are quite
encouraging, but the research is still ongoing.

# The Approach

The estimation approach for the decennial census uses 100% counts
available at a fine geographic level. The usefulness of these
geographically detailed counts necessarily fades as the decade
progresses.

Instead, administrative record data provide an alternative that, unlike
the census, reflects change during the decade. The Census Bureau has an
ongoing program for administrative record data and, among its products,
produces an annual series of merged administrative data that present a
census-like picture of much of the population. Most but not all of the
administrative records have been matched on the basis of address to the
Master Address File (MAF). The MAF is also the sampling frame for the
ACS. The estimation approach uses an extract from the administrative
record database that contains basic demographic information (age, sex,
race, Hispanic origin) and the MAFID linking the administrative data to
the MAF. The underlying premise of the approach, which has been so far
empirically supported, is that the administrative record data for
demographic characteristics have fairly high correlations with the
corresponding ACS characteristics at the household level.

The basic approach combines the administrative record data with
model-assisted estimation---specifically generalized regression
estimation (GREG)---to achieve variance reductions without introducing
significant bias. As a class, model-assisted estimators are
design-consistent, and their biases, if any, are generally of lower
order than their sampling variances. Essentially, they are unbiased
estimators. This property is in contrast to many small-area estimation
approaches that are instead model-based. In a frequentist sense, the
bias component of a model-based estimator contributes a significant
fraction, often the majority, of the overall error, unless the model is
perfectly true. Assessment of model-assisted estimators is far
easier---gains, if any, can be measured in terms of variance reduction.

The approach inserts a new step in the ACS estimation process, early in
the estimation steps (just after the noninterview adjustment but before
any ratio or raking steps). The basic approach can be summarized (Fay
2006) as follows:

1.  Link administrative records to the ACS sampling frame, dropping
    administrative records that cannot be linked.

2.  Form unweighted totals of the linked administrative record
    characteristics at the tract level.

3.  Apply ACS sampling weights at the housing unit level to the linked
    administrative record data that fall into the ACS sample. The
    weighted estimates at this step represent unbiased (or essentially
    unbiased) estimates of the unweighted totals in step 2.

4.  Using generalized regression estimation (GREG), calibrate the ACS
    sample weights so that the weighted administrative totals from the
    sample match the unweighted totals from step 2. (The number of
    constraints is allowed to vary with the size and other
    characteristics of each tract.)

5.  Use the new weights in subsequent stages of the ACS weighting, which
    includes ratio and raking/ratio estimation. Although the new weights
    are adjusted in subsequent estimation steps, the argument is that
    most of the variance reduction at the tract level will be retained
    in the final weights.

All linking is performed on the basis of MAFID. No comparison is made of
the composition of the household represented by the administrative
record data and the ACS person-level data. In fact, the entire
calculation is performed without any reference to the ACS person-level
data.

Until recently, the focus of the research was on improving tract-level
estimates, and presumably other sub-county estimates, for ACS 5-year
period estimates. The first such national set will be published in 2010
for the period 2005-2009. There is now evidence, not yet presented
publicly, that this approach can be applied at the place level for
3-year or even 1-year estimates.

Empirical results (Fay 2006) for tracts based on this research strategy
produced promising results. Average tract-level variance reductions were
about 91% for total housing units, 67% for total population, 25-50% for
more specific age/sex demographic cells, and close to 50% for the
Hispanic and Black population counts. (The results were preliminary
because the analysis compared variances before and after the
implementation of GREG but without full implementation of other
weighting steps.)

Since October, the research effort has focused on imbedding the
estimation step in the Multi-Year Estimates Study, which has the overall
goal of producing 3-year and 5-year estimates for the same set of 34
test counties. The study examines the 3-year periods 1999-2001,
2000-2002, 2001-2003, 2002-2004, and 2003-2005, and the 5-year periods
1999-2003, 2000-2004, and 2001-2005. The ACS frame and sample for 1999
were matched to administrative record year 2000 because the 1999
administrative data are not readily available. For all other years, the
ACS frame and sample were matched to the corresponding administrative
record year. Thus the new study affords an opportunity to assess the
performance over time of the approach.

# Is this a new idea?

On the one hand, similar applications are hard to find in the U.S. If
implemented in the ACS, to my knowledge this application will be the
largest application in the county. I previously reviewed a set of
references that I could find on U.S. applications (Fay 2005b), and I
need to repeat the effort to look for new developments. Some
applications are at high geographic levels only, whereas the ACS
application proposes GREG at the tract level for 5-year periods and at
sub-county levels for 3-year and potentially 1-year periods.

On the other hand, the application should be almost obvious to many
others. There is a rich theoretical literature on model-assisted
estimation. I suspect that many graduate students have been exposed to
"the yellow book," (as I have heard it called)---Särndal, Swensson, and
Wretman, 1992, *Model Assisted Survey Sampling.* U.S. researchers,
including Wayne Fuller, have contributed to this literature. The late
Charles Alexander, one of the original architects of the ACS, considered
properties of GREG estimation in another context, but to my knowledge
did not recognize its application to ACS.

Statistics Canada has led government statistical agencies in application
of GREG. In 1991, Statistics Canada converted from a raking/ratio type
estimator, similar to the ones implemented for the last few U.S.
censuses, including Census 2000, with GREG implemented at a geographic
level roughly the size of a tract in the U.S. In Fay 2005, I reviewed
some of the evolution of the Canadian approach for the 1991, 1996, and
2001 censuses. Presumably, methodological enhancements for the census in
2006 will be worth considering. Many of my Canadian colleagues are quite
familiar with generalized regression estimation and some have
contributed to the literature. Knowledge of the GREG estimator and its
properties are incorporated into many of the specific guidelines
provided in the Fourth Edition of *Statistics Canada Quality Guidelines*
in Section 2.10, Estimation (pp. 45-51). Thus, there is considerable
precedent for the application of GREG estimation.

Overall, then, the concept of using GREG estimation to reduce variance
is not new. A few aspects of the ACS application are somewhat atypical:
particularly the effort to construct appropriate unweighted totals to
use in the estimation by linking the administrative records to the ACS
sampling frame. In this application, the GREG step is simply inserted as
one of several estimation steps rather than replacing the rest of the
ACS estimation strategy. In most other applications, GREG estimation is
the primary estimation method.

*Status of the research:*

The overall research strategy has been to identify a simple version of
the estimation strategy to yield a substantial portion, perhaps 80%, of
the gains that are possible from the most carefully crafted estimator.
We expect the Multi-Year Estimates Study will produce a rich set of data
for analysis and further improvement in the GREG estimation.

References:

Fay, R.E. (2005a), "Model-Assisted Estimation for the American Community
Survey," *Proceedings of the 2005 Joint Statistical Meetings on CD-ROM,*
American Statistical Association, pp. 3016-3023 (available as a .pdf
from the author).

\_\_\_\_\_\_ (2005b), "Potential Applications of Model-Assisted
Estimation to Demographic Surveys in the U.S.," paper presented at the
Federal Committee on Statistical Methodology Research Conference,
available from
[www.fcsm.gov/05papers/Fay_IIIC.pdf](http://www.fcsm.gov/05papers/Fay_IIIC.pdf).

\_\_\_\_\_\_ (2006)," Using Administrative Records with Model-Assisted
Estimation for the

American Community Survey," paper to appear in the *Proceedings of the
2006 Joint Statistical Meetings on CD-ROM,* American Statistical
Association. (available as a .pdf from the author).

Starsinic, M. (2005), "American Community Survey: Improving Reliability
for Small Area Estimates," *Proceedings of the 2005 Joint Statistical
Meetings on CD-ROM,* American Statistical Association*,* pp. 3592-3599.

Van Auken, P.M., Hammer, R.B., Voss, P.R., and Veroff, D.L. (2004),
"American Community Survey and Census Comparison, Final Analytical
Report, Vilas and Oneida Counties, Wisconsin; Flathead and Lake
Counties, Montana," unpublished report dated March 5, 2004, available at
http://www.census.gov/acs/www/AdvMeth/acs_census/lreports/vossetal.pdf.