# Presentation: 730366

## The Use of Random Digit Dialing in Household Surveys:  Challenges and Changes

**Chris Chapman**
**2008 IES Research Conference**
**Washington, DC**
**June 11, 2008**
**chris.chapman@ed.gov**

**Notes:**
Good  afternoon, my name is Chris Chapman and I will be talking with you about some of the work we have done to address growing challenges for our Random Digit Dial telephone surveys which are collectively referred to as the National Household Education Surveys or NHES for short.  

## Overview 

- Two primary challenges in the use of traditional RDD collections for household surveys
- Declining response rates in traditional Random Digit Dialing (RDD) sample methodology
- Declining coverage rates in traditional RDD sample design studies
- Analyses of alternative approaches for government RDD surveys
- Use of pre-phone contact mailed information materials
- Use of prepaid monetary incentives
- Study of staged, multimode collection methods
- Expanding coverage

**Notes:**
More specifically, I will go over some of the work we have done to improve response rates and study the effects of declining response rates followed by a short summary of work we did to study coverage rates, but that we have not released yet, and a discussion of NHIS work on the topic.

I will talk about premailing and incentive strategies that we have tried along with a staged multimode strategy we recently employed primarily to study potential biases in the data.

Then I will talk about some of the coverage issues we are facing.

## National Household Education Surveys (NHES)

- The analyses discussed today will be in context of work primarily done with NHES
- NHES is a repeating household survey conducted by NCES
- NHES has relied on RDD sampling approaches since its inception in 1991 through the latest collection in 2007
-        Sample is drawn from existing 100 banks starting with
-        130,000 to 475,000 numbers
-        Sample is restricted to non-business numbers and
-        excludes cells phones

**Notes:**
As I mentioned, much of this work was done in context of NHES.  Is anyone here familiar with these collections?

For those of you who are not familiar with NHES, it is a repeating cross sectional set of surveys that were initially fielded in 1991.  We just wrapped up the latest collections late in the spring of 2007.

The first few years of the study, we used the Mitofsky-Waksberg sampling approach, but then shifted to a listed assisted method to reduce clustering effects and increase data collection efficiency.

We build the sample off of working 100 banks with residential numbers.  Basically start with all known area codes in the country with residential numbers.  Exchanges in these area codes that have been set aside for cell phones are then removed.  Then, we pull all prefixes in those area codes with listed telephone numbers.  Then, within those prefixes we check white pages listings to see which 100 banks have listed residential numbers included in them.  The 100 banks are the next 2 digits after the prefix (digits 7 and 8 of a telephone number)
	- Within working 100 banks, we then randomly assign the last 2 digits to the working 100 bank and use those as the basis of the sample.

We then cross check the numbers against yellow page and white page listings to remove business only numbers.

## Topics Addressed

- Early childhood education and nonparental care for infants, toddlers, preschoolers and students in early elementary grades
- Parent and family involvement in the education of K-12 students
- 	After school programs and activities for elementary and middle school children
- 	Adult education for all civilian, non-institutionalized adults
- 	Civic education and participation for K-12 students, parents and adults
- 	School safety and discipline for K-12 students

**Notes:**
This next slide shows the substantive content of the collections included in NHES.  Generally, we use NHES to sample people from birth through adulthood to collect data on issues ranging from enrichment activities for infants and toddlers to post-retirement personal interest courses taken by adults at local community colleges.

## National Household Education Surveys (NHES)

- Typically relies on two stages of interviews
- 		1.  Screener interview to determine household and 	within	household eligibility – generally need 45,000-	60,000	completed screener interviews per collection
- 		2.  Extended interviews collects detailed information 	about person(s) sampled within the household
- Significant nonresponse problem in the Screener
- 		1.  Approximately 80% in the early 1990s
- 		2.  Approximately 50% now

**Notes:**
To get from the telephone number to sampling people, we need to call households and roster household members.  We do the rostering through screening interviews.

After screening, we run sampling algorithms on household members and, depending on the content of NHES during that year, sample individual household members.  We have typically needed 45,000 to 60,000 completed screener interviews to get large enough sample for the follow-up, or extended substantive interviews

The screener is where we have first contact with households and where we take the biggest response rate hit.

## Increasing Response Rates with RDD

- Use of premailing
- 	Conducted experiments showing that sending letters to households before calling helped improve first stage response rates
- Use of monetary incentives
- 	Use of small monetary incentives mailed to households before calling helped improve first stage response rates
- In-person follow-up
- 	During recent bias study, found that in-person contact with phone interview refusal cases significantly boosted response rates

**Notes:**
To help boost our declining screener response rate, we have done several different kinds of work.  
Mailing options
Incentives
In person follow-ups

## Premailing with RDD

- Letters about the purpose of the study and where to get information about it help boost first stage response rates
- In the 2001 NHES, the
- 		response rate for matched households sent a letter was 75%
- 		response rate for matched households not sent a letter was 70%
- 		response rate for households that could not be matched was 55%
- Challenge is matching randomly generated phone sample to mailing addresses – approximately 50 percent
- See http://nces.ed.gov/pubs2005/2005071_2.pdf

**Notes:**
See 2003 monetary incentive report pages 2-3 for summary
See 2001 methodology report page 62 and page 113 for details.

## Prepaid Incentives with RDD

- In 2003, we tested monetary incentive options
- Required that we had a mailing address – again had a 50% match rate
- Tested no incentives against combinations of $2 and $5 incentives sent before the interview and as a refusal conversion technique
- http://nces.ed.gov/pubs2006/2006066.pdf

**Notes:**
In 2003, we then tested the use of small, prepaid monetary incentives in combination with different mailing and refusal conversion approaches.  

This strategy, like the mailing strategy itself required that we have mailing addresses to work.  Again, we were able to match about 50% of telephone numbers to addresses.

We tested not sending incentives against sending combinations of $2 and $5 incentives at precontact and refusal conversion stages.

## Prepaid Incentives with RDD

- adfafasfas

- adfafasfas

**Notes:**

This shows results by experimental group.  Groups were determined by mailing status – references to 1st and priority mean first class and priority mailing respectively, amount of incentive sent, and when incentives were sent – before first call, after first refusal, or both.

What is not shown here are results for households for which we had no address information.  There response rate was 49% and there ever refusal rate was 55%.

## Prepaid Incentives with RDD

- Optimal approach for cost v. response rate effect was no initial incentive with a $5 incentive for refusal conversion
- Next best approach was $2 initial incentive with a $2 conversion incentive
- Both boosted first stage response rates over no incentives by approximately 5 percentage points
- Tested use of priority and regular mail for initial and refusal conversion mailing - mailing did not have an independent effect beyond incentives for the first stage of data collection

**Notes:**
We found the best balance between cost and response rate was to send households who refused a $5 incentive to complete the survey when we called back.

After that, it was a $2 and $2 combination.  

As shown, we also tested mail format options, but found that this did not have an independent effect in this experiment above and beyond the monetary incentives.

## Study of Potential Bias and Multimode Approach	

- Conducted an extensive bias test in 2007
- Key components included:
- 	1.  In person follow-up for telephone nonrespondents – 	used to study potential nonresponse bias
- 	2.  In person contact for households not in the RDD 	frame – important to understand potential coverage bias

**Notes:**
In 2007, we conducted an extensive bias analysis to study both potential nonresponse and potential coverage bias in the data.  

We worked with a special sample of about 7,500 cases that were not part of the main data collection.

Unlike the main study, the bias study sample was clustered in PSUs selected in proportion to population size and the sample was built from mailing addresses.  The mailing addresses were then sent to various commercial vendors to obtain telephone numbers.  

We oversampled from addresses with matching telephone number to facilitate studying households that refused the telephone interview or for which we could not establish telephone contact.

Sample sizes were set to allow detection of 5 percentage point differences from the main study in key estimates typically generated from NHES.

## Multimode Work

- Results not released yet but some results are generally clear
- If budget permits, in-person follow up can significantly increase response rates if used to help with telephone refusal conversion
- If not corrected, surveys based on RDD samples of land line phones alone can incure coverage bias in studies of prekindergarten through high school students and their families

**Notes:**
We have not published the results yet, but generally speaking, it does appear that in-person contacts with households after a phone refusal can significantly increase response rates.  This is expensive however since you have to pay for travel of field staff and we provided monetary incentives to complete the screener after it was completed by the household.

We also found that there might be undercoverage bias if a landline sampling frame is used.  Without corrections through weighting procedures, undercoverage biases can exist.

## Frame Adjustments

- NHIS indicates that in 2007 16% of households were cell only
- Approximately 14% of children live cell phone only households
- Approximately 15% of adults live in cell phone only households
- Can correct for some of coverage bias through weighting
- Preferable to incorporate cell phones into frames though
- This may reduce overall response rates
- Challenging for calculating proper weights
- http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless200805.pdf

**Notes:**
The National Health Iinterview Survey has been used to study cell phone-only and cell-phone mostly households compared to landline households.  The most recent published data are for late 2007 so are quite timely.

What they found was that by using landline only sampling frames, you exclude about 1 in 6 households and similar proportions of children and adults from your frame.  

As I mentioned, some of the coverage bias can be corrected through various weighting strategies, but it would be preferable to avoid excluding so many households from the outset of a study.

We can incorporate cell phone numbers instead of excluding them as we currently do.  We have a pretty good sense of what these numbers are.  

However, this is not a solution that can be utilized without careful consideration.  One is that cell phone interviews tend to have lower response rates than landline only interviews so this will not correct issues related to nonresponse.

Probability of selection adjustments in weights can be harder to estimate because we do not yet have a way to identify cell phone numbers in the same household and households may have a cell phone per person.