# Presentation: 285566

## ACS – Quality Questions

- Prepared for the October 11, 2006
- Census Information Center, State Data Center, and Business and Industry Data Center Training Conference

## Question 1 - Tracking the sample

- How do we get from the initial sample selected to the count of final completed interviews?

## Factors that reduce the sample size from the initial sample

- Survey nonresponse
- Identification of sample addresses that are ineligible for interviewing
- CAPI subsampling
- Availability of “good” phone numbers

**Notes:**
Many factors can impact the count of final interviews 
Survey nonresponse
Initial sample addresses determined to be nonexistent or otherwise ineligble for interviewing
CAPI subsampling – The number of cases subsampled out (not selected for the CAPI subsample) is a function of mail response, mailability (addresses that we conclude are complete enough to mail), deliverabilty (addresses that the USPS returns as undeliverable), success in obtaining phone numbers

The variable subsampling rates used in 2005 were successful in reducing the effect of CAPI subsampling on the final count of interviews

To explain the affect of some of these factors I will walk you through 2 examples
The goal of these slides is to demonstrate the methods used in the ACS to calculate response rates and to also show the sensitivity of the final count of interviews to factors such as mailability
The examples are simulations based on reasonable assumptions – not real 2005 data.
First time we have tried to explain this to users so be patient – and ask questions – if its not clear.  I’d also appreciate your advice on how this explanation might be improved

## Slide 4

**Notes:**
Think of the circle (or pie) as the initial sample for an area – that initial sample canbe split into two parts based on the format and content of the addresses
For this area of 10,000 sample addresses, 10% were determined to be unmailable (In the US, mailable includes addresses with either a complete city-style address or a complete rural route address, differs slightly for PR)
Complete city-style = HN/SN, ZIP
Complete rural-style = RR#, box#, ZIP
PR also requires urbanization name

So the blue area shows the mailable addresses, the purple area the unmailable addresses

## Slide 5

**Notes:**
The unmailable addresses are not eligible for mail or CATI operations
This slide shows a possible outcome for the mailable addresses – the 90% is split into 3 slices - 45% resulted in a mail interview and 9% in a CATI interview – leaving 36% of the initial sample mailable but with nonresponse after mail and CATI – that nonresponse is shown here in pale blue

## Slide 6

**Notes:**
Now lets look at the affect of subsampling for CAPI.  
Two universes are subsampled at different rates
The unmailables (purple) are subsampled at 2-in-3 or 67% so in this example, of the 10% unmailables, about 7% are subsampled for CAPI – leaving 3% not sampled (or subsampled out)
Mailable addresses are subsampled at different rates 50%, 40% and 33%.  For this example I used a 33% rate, so the 36% of the initial sample that remained nonresponse after mail and CATI efforts (in pale blue) is subsampled with 12% going to CAPI and 24% subsampled out
So, only the subsampled cases continue into the CAPI operation –where there are 3 possible outcomes – an interview, a noninterview or the identification that the sample unit should be deleted (an ineligible unit)

## Slide 7

**Notes:**
I realize this slide is busy but here is what I want you to focus on –
First, lets look at the mailable universe that was eligible for CAPI (it is shown in pale blue)
The 24% that wasn’t sampled is still there but the 12% slice is now split into 3 parts - with about 11% interviews and under 1% each of deletes and nonresponse.
Similarly for the unmailables, in purple – The not sampled universe is still at about 3% but the 7% that went to CAPI is split into 3 parts – interviews, deletes and nonresponse
Because the CAPI sample is a representative sample, we can use the results (dispositions) from the sample cases that went to CAPI to weight up to the full universe – covering the two pieces of this pie that were subsampled out.

This slide – yes, with lots of information, shows the full set of disposition codes for all of the initial sample cases – we use all of this information to weight the results and to calculate the response rates

Lets look at weighting first...

## Slide 8

**Notes:**
This final pie shows the impact of that weighting.  The subsampled out pieces are now accounted for by the results of the CAPI interviewing. Note that the purple area of unmailables is now only 3 slices – the subsampled out piece is gone – similarly for the mailables – 3 slices.
These are the final weighted results.  Its important to acknowledge that the subsample is a representative sample allowing this weighting to be of high quality
Note that each outcome or disposition is weighted up – not just the interviewsAll of these final weighted pieces are used to calculate the response rates.

## Summary

|  | Unweighted | Weighted |
| --- | --- | --- |
| Initial sample | 10,000 | 10,000 |
|    Deletes | 93 | 230 |
|    Nonresponse | 93 | 230 |
|    CAPI interviews | 1679 | 4137 |
|    Mail interviews | 4500 | 4500 |
|    CATI interviews | 900 | 900 |
| Total completed interviews | 7079 | 9537 |
| Response rate | ----- | 97.6% |

**Notes:**
In the example we just walked through – I started with a sample of 10,000
The response rate was over 97% and the rate of final interviews to initial sample was close to 71%.
These two rates describe 2 different things – the 97% response rate describes the % of the population directly represented in the sample – we use well developed models to account for nonresponse

The 70.8% is a measure of the % of the initial sample that could be used in the production of estimates – this is the key measure that impacts the reliability of the estimates – the MOEs – the size of the confidence intervals.

The MOEs of the estimates are a function of the # of completed interviews

You can see that nonresponse, deletes and subsampled out addresses all contribute to a reduction in the sample size from the intial sample selected to the completed interviews that are used to produce the survey estimates

## Slide 10

**Notes:**
This graph shows a distribution of state-level ratios of final interviews to the initial sample 

US level – in 2005 was 66% - of the initial sample of about 3 million, about 2 million were interviewed

PR – in 2005 of the about 36K selected initially, about 22K interviewed (60%)

Rates vary at state (and similarly will vary across geographic areas) – most states in the range of 60-70%

Outliers include – AK, LA, MS, NM, TX – low
CT, IO, MN, NE, OH, WI - high

Factors influencing this rate include 
low response rates by mail and phone (these in turn due to factors such as cooperation, mailability of addresses, ability to obtain good phone numbers, language or literacy barriers)
Nonresponse during CAPI
High levels of invalid/ineligible housing units (e.g., LA and MS in 2005)

## Question 2 –Language program

- What are the plans to ensure high quality data for Asian Americans with limited-English proficiency?

## Current

- Call centers employ interviewers who speak Spanish, Chinese, Russian, Vietnamese, Korean, Arabic, Italian, Tagalog, German
- Regional Offices employ interviewers who speak all of the above languages plus French, Polish, Greek, Japanese, Portuguese, Urdu, Navajo, Creole

## Current

- Regional Offices use interpreters when they encounter language needs not supported
- Noninterviews due to language barriers in the 2005 ACS were less than 0.1%
- Of these, nearly 50% were Spanish
- No Regional office had a rate in excess of 0.3%
- We continue to monitor these levels and identify areas of need

## Current

- Cognitive testing of materials in Spanish, Chinese, Korean, and Russian
- Development of glossary of translated terms for ACS and 2010 materials

## Future

- Proposed development and testing of a multilingual assistance brochure as a mail insert
- Researching development of language guides as interviewer tools in Chinese, Korean, Vietnamese and Russian

## Questions?

- _[Deborah.H.Griffin@census.gov](mailto:Deborah.H.Griffin@census.gov)_
- (301) 763-2855