The National Survey of Fishing, Hunting, and Wildlife-Associated Recreation 

Cell Phone and Debit Card Test 

A Cost -Benefit Analysis 

Aniekan Okon 

Elke Mclaren 

Denise Pepe 

U.S. Census Bureau 

March 16, 2012 

Abstract 

The 2011 National Survey of Fishing, Hunting, and Wildlife-Associated Recreation (FHWAR) is an 

address-based sample survey with interviews conducted by Computer-Assisted Telephone Interviewing 

(CATI) and Computer-Assisted Personal Interviewing (CAPI). We conducted a cell phone and debit card 

test to research alternative survey designs that might increase the number of less expensive CATI 

interviews completed, and do so in a cost-effective manner. This test consisted of 1,411 households, for 

which no telephone number was available, that were randomly assigned to one of three treatments. The 

first treatment was sent an advance letter with a cell phone. We sent a postcard a week before the cell 

phone mailout informing the sample that they would be receiving a package in the mail with a cell phone 

in it. The second treatment received an advance letter with a $25 debit card incentive. The third treatment 

received only an advance letter. The letters for all three treatments emphasized the cost savings of 

conducting a telephone interview and requested that the respondent call the telephone center to complete 

an interview. We compared the results from the cell phone and debit card test to our production CATI 

and CAPI samples to evaluate whether either of these test options are viable alternatives to collecting 

data through a personal visit when we have an address but no telephone number. The results show that 

the incentive (debit card) got a better response rate than the cell phone treatment and, when it comes to 

determining if it could be used in production, the results indicate that it is possible but would be very 

expensive to do. 

Introduction 

How do we reach our survey sample in a cost effective way, maintain survey response rates, and avoid an 

increase in survey bias? These are among a great number of questions being asked by researchers every 

day, especially with response rates decreasing for surveys (De Leeuw and DeHeer 2002; Atrostic, Bates, 

Burt, and Silberstein, 2001; Baruch, 1999; Deheer, 1999; Steeh, 1982) and an increasing proportion of the 

population relying on cell phones as their main source of communication. In the first half of 2010, for 

example, more than one in four American households (26.6%) had only wireless telephones— an 

eightfold increase over just 6 years (Blumberg, and Luke 2010) and a number that is sure to increase each 

year. This high number of wireless-only households might offer an explanation as to why phone 

cooperation is on the decline, as cell phones are not listed in public directories or vendor lists that use 

reverse-address searches to locate phone numbers. Even reachable by a cell phone, depending on the 

nature of the survey, respondents might not be as forthcoming as they would have been if they were in the 

confines of their home using a landline, which can affect the quality of the data (AAPOR Cell phone Task 

Force 2010). Surveys are also competing for the same pool of participants. In light of this, researchers are 

trying to come up with ideas that would make their survey stand out among the many so as to yield a high 

response rate. 

1 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The National Survey of Fishing, Hunting, and Wildlife-Associated Recreation (FHWAR) is sponsored by 

the U.S. Fish and Wildlife Services. It collects information regarding fishing, hunting, and other wildlifeassociated activities. FHWAR is a multistage probability sample of households with coverage in every 

state. The survey is conducted every 5 years and is done in 3 waves. Wave 1 is household-based and 

consists of a screener with the possibility of detailed interviews asking about a person’s hunting, fishing 

or wildlife-watching activities. Wave 2 and Wave 3 are person-based, detailed interviews where 

respondents are selected for sample based on data collected during the Wave 1 screener. FHWAR uses 

Address-Based Sampling (ABS) methods to select the initial Wave 1 sample. ABS involves drawing 

addresses randomly from a frame of addresses. The primary advantage to using ABS is that it provides a 

very high level of coverage, with some estimates placing frame coverage of U.S. residential postal 

households in the mid-90 percent range. With appropriate tools, ABS cases can be associated with 

telephones numbers for a telephone survey, or ABS cases can simply be used as is to conduct personal 

visit interviews. In some instances, the kind of survey being conducted dictates what interview mode to 

use, but cost constraints are often of paramount importance. It is a well established fact that personal visit 

interviews tend to cost considerably more than telephone interviews. 

In 2006, the FHWAR survey was conducted in CAPI. In 2011, the interviews were done in both CATI 

and CAPI. The initial plan was for the interviews to start in CATI and then, after a few weeks, transfer 

cases that could not be contacted to CAPI. Faced with increasingly high costs of the personal visit 

interview component of the survey, research was conducted during the survey’s 2011 implementation to 

explore possible methods for increasing the number of CATI interviews. Of course, costs could be 

reduced by sampling cases for CAPI follow-up at a lower rate. A major downside of such a strategy, 

however, would be reduced precision and increased variance of survey estimates. The 2011 study 

examined two somewhat costly but potentially beneficial new strategies for increasing the proportion of 

CATI interviews among the non-telephone portion of the sample; send respondents either a cell phone or 

debit card incentives with an advance letter requesting that they call us to complete an interview. 

Much research has examined how incentives affect response rates in telephone, mail, and personal visit 

surveys (e.g. Dillman 1978; Singer et al 1999; Willimack et al 1995; Armstrong 1975; and Gunn and 

Rhodes 1981). However, the research base is scarce regarding the effects of sending a pre-paid cell phone 

or Smartphone on telephone survey response rates. A very small NORC study [2009 NIS Methodology 

Report] may constitute the only relevant research to date. This study sent pre-paid phones to 40 subsampled households in both Arizona and Michigan to try and obtain an interview by telephone. In 

Q1/2008 for both Michigan and Arizona a combined total of 8 interviews were completed, but none were 

completed in Q2/2008. 

The purpose of our study was to compare the CATI response outcomes for the three treatments (cell 

phone treatment, $25 debit card incentive treatment, and letter only treatment), and, in the event of 

positive impacts on response, to assess the success of the experimental treatments from a cost-benefit 

perspective. 

Methodology 

The sample for the study came from the Master Address File (MAF). The MAF is a file that is strictly for 

the use of the Census Bureau, covers all geographical areas in the country, and includes all housing units. 

It is updated every six months. Only sample addresses without a phone number were eligible for selection 

for this study. 

2 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

All treatments were sent a similar advance letter explaining that personal interviews are expensive and 

that we were trying to reduce costs by conducting the interview over the phone but did not have a 

telephone number for them. We further requested they call a toll-free telephone number to participate in 

the 2011 FHWAR survey. For the cell phone treatment, we mailed the participants the advance letter 

with the cell phone informing them that we were sending them a cell phone in hopes of connecting with 

them via the phone. Prior to mailing, we sent a postcard informing the participants that they were going 

to receive a cell phone in the mail. The debit card treatment received the advance letter with the $25 debit 

card enclosed. The control treatment received the advance letter only. 

The cell phones sent to the cell phone treatment were Verizon pre-paid phones, costing $19.99, with $30 

worth of minutes (160 minutes of talking time) included, for a total cost of $50. Each cell phone was 

assigned a phone number associated with the central time zone (CST), in accordance with instructions for 

the Census interviewers to call between the hours of 2pm-9pm EST in order to limit their calls to 

reasonable hours no matter what area of the country was being called. For the debit card treatment, 

respondents were informed in the advance letter that they would be given the PIN (personal identification 

number) to activate the card upon completion of the survey. 

This study was comprised of two phases. In the first phase, letters were sent to 1,411 no-phone cases, of 

which more than half from each treatment came back as Undeliverable As Addressed (UAA’s).

1 We 

conducted a second phase where we selected an additional 220 cases per treatment from the MAF.2 For 

the purpose of this paper, only the results from the second phase will be discussed as it is more 

comparable to our production-sampling scheme. 

Results 

Table 1. Response Rate 

Panel N Completed Response Rate 

Cell phone 220 30 13.6% 

Incentive 220 33 15% 

Letter 220 11 5% 

The results from the second phase show that the debit card incentive treatment had the highest response 

rates. The response rate for the debit card incentive was 15%, cell phone was 13.6% and the control was 

5%. When comparing the panels for statistical significance we find that the difference between the debit 

card incentive panel and the cell phone panel did not differ significantly, but both were significantly 

higher than the letter only panel. 

1 The Office of Management and Budget (OMB) required that the study only use no-phone sample address for 

which the phone number was also blank in the 2010 Census. Moreover, the MAF, from which the sample was 

drawn for the first phase, had not been updated with permits, constructions, or vacants. We believe these two 

factors contributed to the unusually high number of UAAs. 

2 OMB lifted the さミo-phoミe saマple addヴessざ ヴeケuiヴeマeミt foヴ the secoミd phase of the study. In addition, the MAF 

had been updated to incorporate permits, constructions, and vacants. 

3 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Cost Benefit Analysis 

As mentioned earlier, FHWAR production consists of two modes, first CATI and then CAPI, for cases for 

which a phone interview cannot be completed. The study was conducted in CATI with no follow up 

interviews. So, in doing the cost-benefit analysis, comparisons had to be done in two parts; calculating the 

cost of doing the interviews in CAPI without follow-up and then estimating what the cost would have 

been if the study was done in mixed mode, like the production survey. The cost-benefit analysis 

concentrates on the debit card incentive panel because the response rate shows that this incentive panel 

outperformed both the more expensive cell phone panel and the control panel. 

For cost-benefit calculation purposes, we assume that the average cost to conduct a personal interview P 

(i) is $250, including the cost of non-interviews (Census Internal Budget Memo 2011). . The cost to 

conduct an interview by phone T (i) is assumed to be $67 (Census Internal Budget Memo 2011). Noninterviews by telephone T (ni) cost 1/3 as much as completed interviews. Using those numbers we try to 

get a baseline for the three treatment panels. 

Based on the personal visit cost of $250 per case, sending all 220 cases directly to CAPI would have cost 

$55,000 (i.e., 220*250). As seen in Table 1, the $25 dollar debit card yielded 33 CATI interviews, 

leaving 187 cases to be sent to CAPI. Using the cost assumption above, the 33 interviews completed in 

CATI cost $3,036 (i.e. 33*(67+$25) ($25 = cost of the card)). The 187 cases sent for CAPI interview 

would cost $46,750 (187*250), resulting in a total cost for this treatment of $49,786. Using the incentive 

in a mixed mode design results in a cost savings of approximately 10% compared to what it would have 

cost to send all cases to CAPI for interviewing. 

Conclusion 

We carried out this study to investigate alternative survey designs that might increase the number of 

CATI interviews completed, and do so in a cost-effective manner. Do the results show enough evidence 

that an alternative survey design, namely this one, could be used in future production surveys? The 

answer is inconclusive and we provide an explanation below. 

One of the best measures to judge how well any survey is doing is by looking at the response rates. And, 

yes, response rates were low. However, the debit card incentive panel outperformed the other two 

treatments. This is in line with Dillman’s theory of social exchange which states that people who 

participate in surveys will be compensated in a way that meets their needs (Dillman 2000). As the debit 

card incentive panel outperformed the other panels, we could infer that it could work on a large scale 

survey, with further research. 

Another measure to evaluate a study like this is to look at coverage. Were we able to expand our coverage 

to include any demographic groups that had been missing or underrepresented in past rounds of the 

FHWAR survey? In comparison to historical FHWAR survey data, it does not appear so. In production, 

African Americans are underrepresented. The sample underrepresentation occurred with this study. We 

see from the debit card treatment and the other two treatments that a majority of the completed interviews 

were from white respondents. Could this be a result of the way the sample was drawn or is it just the 

nature of the survey? We do not know, but more research needs to be conducted to answer such questions. 

4 
 

 

 

 

 

 

 

 

The next and probably more important measure is the cost of conducting this study. Is the cost of doing 

this study worth it? For instance, can the $25 debit card incentive, which had a higher response rate than 

the other treatments, be used in actual production when dealing with a large sample size? We believe that 

this study shows that it can be done in actual production but would be costly. 

There are still areas that could be looked into for future research. For example, is there a difference in 

reported expenditures between the three treatment panels and between the treatment panels and 

production? Also is there a difference in the frequency with which respondents hunted or/and fished 

between the treatment panels and between the treatment panels and production? Because the sample size 

for this study was small, we could not report on the results with any confidence. 

5 
 

 

 

 

 

 

References 

American Association for Public Opinion Research. 2010. ―New Considerations for Survey 

Researchers When Planning and Conducting RDD Telephone Surveys in the U.S. With 

Respondents Reached via Cell Phone Numbers.‖

Armstrong, J. Scott (1975), "Monetary Incentives in Mail Surveys," Public Opinion Quarterly, 

39, 111-16. 

Atrostic, B. K., Bates, N., Burt, G., and Silberstein, A. ―Nonresponse in U.S. Government 

Household Surveys: Consistent Measures, Recent Trends, and New Insights.‖ Journal 

of Official Statistics, 2001, 17(2), 209–226. 

Baruch, Y. ―Response Rate in Academic Studies: A Comparative Analysis.‖ Human 

Relations, 1999, 52(4), 421–438. 

de Heer, W. ―International Response Trends: Results of an International Survey. Journal 

of Official Statistics, 1999, 15(2), 129–142. 

De Leeuw, E.D. & De Heer, W. (2002). Trends in household survey nonresponse: A longitudinal 

and international comparaison. In: R.M. Groves, D.A. Dillman, J.L. Eltinge, & R.J.A. Little 

(Eds). Survey nonresponse. New York: Wiley, pp. 41-54. 

Dillman, D.A. (1978). Mail and Telephone Surveys: The Total Design Method. New York: 

Wiley & Sons. 

Gunn, Walter J. and Isabelle N. Rhodes (1981), "Physician Response Rates to a Telephone 

Survey: Effects of Monetary Incentive Level," Public Opinion Quarterly, 45, 109-15. 

Singer, E., N. Gebler, T. Raghunathan, J. Van Hoewyk, and K. McGonagle. 1999. ―The Effects 

of Incentives on Response Rates in Personal and Telephone Surveys; results from a Meta 

Analysis.‖ Journal of Official Statistics, vol. 15 

Steeh, C. G. ―Trends in Nonresponse Rates, 1952–1979.‖ Public Opinion Quarterly, 1981, 

59, 66–77. 

Willimack, D. K., Schuman, H., Pennell, B. E., & Lepowski, J. M. (1995). Effects of a prepaid 

nonmonetary incentive on response rates and response quality in a face-to-face survey. Public 

Opinion Quarterly, 59, 78-92. 

6