# Presentation: 277853

## Methodological Issues in Local Area Application of the American Community Survey

- JSM 2002
- August 14, 2002

## The Difficulties in Comparing Quality Measures between the Census and the American Community Survey

- Susan Love
- Discussant
- Census Bureau

## PROCEED AT YOUR OWN RISK!

- Comparing two data sets purported to measure the same concept should be approached with the same care you would use in a mine field
- No matter what you do someone is bound to say
- you are comparing apples and oranges
- And if you are trying to compare data quality someone will say
- you are not being fair!

## Do Your Homework

- What you don’t know can cost you “big time”!
- Understand the sources of the data
- Identify where similarities exist
- Determine what differences are insurmountable
- Learn the jargon
- Some comparisons should never be made!

## Stuff Happens

- Even when two surveys are designed to measure the same things using the same questions and similar methods, the results are not always comparable

## ACS versus Censusmore like Granny Smiths and MacIntoshes

- The questions are the same, but the reference dates are different
- The mailout phase is similar, but the followup  phases are different
- The criteria used to decide if a sample response is usable are different, but the difference can be measured
- The content edits are similar, but imputation rates reflect differences in data collection operations and processing

## Common Comparison Pitfalls 

- Search for the true meaning of “Response”
- Mail Response vs Mail Return rates
- Self-response vs Enumerator response rates
- Survey Response vs Interview rates
-  And its opposite
- Nonresponse vs Noninterview rates
- Non-mail vs Followup rates

## Making “Self-Response” Comparablebetween the ACS and the Census Long Form

- Using the proportion of self-response contributing to the final data overcomes operational differences
- Weighting responses by sampling fraction reciprocals corrects for differences in the sampling rates
- Areas with larger Census samples may have lower mail return rates due to lower Long Form response
- ACS has adopted the Census differential sampling, but the impact on mail return rates is not known

## Public Cooperation in the Bronxproportion of households interviewed by self-response collection modes

**1990 Census Long Forms**

- 56%

**2000 Census Long Forms**

- 53%

- 36%

**2000 ACS Test**

-   self-response households
-   nonresponse households

## Self-Response Reality

- The ACS will never reach the level of self-response accomplished by the Census in most areas
- In hard-to-enumerate areas like the Bronx, constant emphasis will have to be placed on a quality nonresponse followup

## More Common Pitfalls

- All “Unit Nonresponse” is not the same
- Know your denominators!  Are all units interviewed, or only occupied units?
- Are proxy interviews accepted, or must all interviews be with the sample household?
- Unit noninterviews occur in all censuses and surveys, even if not officially reported

## ACS Unit Nonresponse

- What is an ACS Noninterview?
- An “interviewed” household record that has less than the minimum data required to be edited and treated as an interview
- A personal visit case that should have been interviewed but was not (a “Type A” noninterview)
- Both of these are routinely measured

## Census Sample Unit Nonresponse

- What is a Census Long Form Noninterview?
- A Long Form household record that has less than the minimum data required to be in the Census sample
- A proportion of the Census units in an area for which no data were collected equal to the area’s Long Form sampling rate
- this component of Long Form noninterviews has not been measured

## Minimum Census Sample Data Criteria

- For Census sample data processing, each Long Form household record is checked to see if it has AT LEAST one person record with two 100% items AND two sample items
- Long Forms that don’t meet this minimum criteria are not “Sample Data-defined” and are not placed in the Census sample

## Minimum ACS Data Criterion

- An Acceptability Index is computed for each “interview” household record
- This index is the sum of all basic (“100%”) items, with AGE counting as two, divided by the number of people in the household
- Households with an Acceptability Index of less than 2.5 are considered noninterviews, in addition to the CAPI Type A noninterviews

## Percent “Not Sample Data Defined” Bronx Records, by Collection Mode Census minimum data criteria

- Nonresponse Followup

- Self-Response

- Percent of Households

## Percent “Unacceptable” Bronx Records, by Collection ModeACS minimum data criterion

- Nonresponse Followup

- Self-Response

- Percent of Households

## Comparing the Bronx Sample Composition

- Self-response households are over-represented in the Census samples - their Long Form mail returns are nearly always sample data-defined
- Nonresponse followup households are under-represented in Census samples - their Long Forms are frequently dropped
- This tendency to over-represent self-response is not as evident in the ACS Test sample

## Interesting Observation

- If the ACS acceptability criterion had been used in the census, many more Long Form nonresponse followup records would have been placed in the Bronx samples of both censuses
- If the Census sample data-defined criteria had been used in the 2000 ACS Test, slightly more self-response records would have been treated as noninterviews in the Bronx

## One Last Pitfall

- Beware comparing item nonresponse or imputation rates when
- item universes differ
- definitions of “missing” answers differ
- edit imputation methods differ
- treatments of “unit nonresponse” components differ
- weighting processes differ

## Item Nonresponse

- The Census and ACS use the same editing methods to impute characteristics when individual items are not answered, but
- the universes of sample records may be  different - only “interview” records provide data and are edited
- Records identified as “noninterviews” are adjusted for by weighting that mirrors the characteristics of the edited interviews.

## Comparing ACS and Census Item Imputation Rates

- The sample records are weighted only by the initial weights
- This minimizes the effect of the differences in the Census and ACS weighting methodology by removing
- the noninterview and control adjustments on the ACS results, and
- the adjustments from weighting the Census sample records to the 100% counts on the Census results

## Imputation Rates for Selected Items Bronx Total2000 Census and 2000 ACS Test

## Inputation Rates for Selected Items Bronx Self-Response2000 Census and 2000 ACS Test

## Imputation Rates for Selected ItemsBronx Nonresponse Followup  2000 Census and 2000 ACS Test

## Item Imputation by Mode Reflects ACS and Census Operational Differences

- Item imputation in the Bronx has tended to be higher in the Censuses than in the ACS Test
- Emphasis is on data collection in ACS, while it is on the count in the Census
- ACS nonresponse followup collects more complete information through the use of computerized instruments and permanent staff
- ACS mail returns undergo telephone followup for missing items, an operation done in the 1990 Census but not in the 2000 Census