EXPERIENCES WITH INCENTIVES IN A PUBLIC 

ATTITUDES SURVEY USING RDD 

Ronald S. Fecso, 

Division of Science Resources Statistics, National Science Foundation

Keywords: response rate, survey closeout 

The Survey of Public Attitudes Toward and Understanding of Science and Technology is a 

National Science Foundation (NSF), Division of Science Resources Statistics (SRS), survey 

that has been conducted since the early 1970s, generally every two years. In early years, the 

survey was conducted by a grantee. More recently, a contractor winning a competitive bid 

process conducted the survey. Historically, the survey had a 70-percent response rate, using 

random digit dialing (RDD) telephone data collection from a central location. Maintaining the 

70 percent rate has become increasingly difficult, a growing problem with RDD surveys

throughout the survey research community. In anticipation of continued problems with the 

response rate, the survey contract contained an option for the contractor to propose the use of 

financial incentives in an effort to achieve the highest possible response rate for the survey. 

The decision to proceed with incentives in the two most recent surveys was based on a desire 

to maintain response rates. The conduct of the study and the proposed incentive experiment led 

to negotiations between SRS staff and the 1999 contractor (with appropriate consultation with 

Office of Management and Budget, OMB, staff) aimed at assessing the value of incentive 

offers. By changing the incentive approach used during the survey, information was collected 

related to questions of potential interest to Federal survey organizations and other RDD survey 

methodology users. (e.g.: Do the incentives work? What payment level is most effective?) 

In the recently completed 2001 survey, there were continuing response rate problems. 

Incentive efforts, differing from those tried in 1999, were used, primarily in an attempt to 

increase the response rate at the end of a much shorter survey-fielding period than in recent 

past surveys. This paper presents the resulting response rates and other related information 

from the 1999 incentive approaches. The results from the 2001 survey were not available in 

time to meet the deadline for submitting this paper. In reporting on the results of the 1999 

incentive approaches, this research provides an addition to the literature on survey closeouts 

and the impact of some incentive approaches on response rates. 

The 1999 Survey 

SRS desired an effective sample size of approximately 2,000 individuals using random digit 

dialing telephone survey methodology. The 1999 contractor (the Chicago Academy of 

Sciences, with NORC as the data collection subcontractor) purchased the original random digit 

dial sample from Survey Sampling, Inc. The sample of 7,200 telephone numbers (an 

oversample reflecting historic experiences with out-of-scope rates) was grouped into 32 

replicates of 200 cases and two replicates of 400 cases. The 32 replicates were released from 

the start of data collection on March 27 through the end of the regular data collection period, 

April 17, 1999. (The two 400 case replicates were held in reserve and not released.) 
In addition, addresses, when available, were purchased for approximately 1/3 of the sample. 

An advance letter was sent to these cases (with mailing information) explaining the study and 

providing an 800 number to request more information or to make an appointment for an 

interview. Analysis of this effort is not included in this report. 

During the regular telephone interviewing period, from March 27 to May 12, no respondent 

fees were offered. At the end of this regular survey operation period there were 1,606 

completes. The contractor reported that regular survey activities had reached the point where it 

was becoming increasingly difficult to obtain cooperation from potential respondents and that 

regular survey activities would not be likely to achieve the desired 70 percent response rate. 

(Note that SRS recognized that achieving a prespecified response rate level is not an indicator 

of overall data quality. The effort here was to attempt to maintain a level of response that users 

found acceptable in their basic evaluation of the survey.) Therefore, the contractor exercised 

the contract option to use an incentive approach. At that point, the response rate was under 60 

percent -- well below the desired 70 percent response rate. Before starting the incentive 

approach, calls which were being sidelined in the “maximum calls” disposition (meaning ring, 

no answers and 17 attempts) were released on May 7th (to the 9th) for a final attempt before 

being declared “unlikely households” and moved out of pending cases. On May 13, 15 and 16, 

1999, the first of several “tests” of various monetary incentive levels was conducted. 

The 1999 Incentive Approaches 

The incentive study progressed into three phases. The incentive experiment had two initial 

phases and a third closeout phase. The phases were: 

PHASE 

(1) This phase began after the contractor ceased regular survey activities. All active 

cases, except for previously scheduled callbacks, were eligible for this phase. 

These cases were “randomly” divided into four payment groups. Each group had 

a different incentive payment level. The members of each group were to be 

offered the payment level assigned to that group, either $0, $10, $20, or $30, if 

contact was made. This experiment was conducted for three days, overlapping 

the May 15 - 16 weekend. 

(2) A modification of the incentive approach was developed after completing phase 1. 

During this phase, interviewers were instructed to continue contacting the 

remaining sample, except for those already scheduled from prior contacts, and to 

offer $10 to gain the cooperation of respondents who initially refused to 

participate. 

(3) The final incentive approach, as was the case in phase 1, attempted to create 

experimental conditions to allow testing of the effects of incentive approaches. In 

this phase, remaining cases, not already scheduled for callback, were assigned to 

two new treatment groups using randomization instruction supplied by SRS. The 

two groups were offered of $0 and $20 respectively. 
As each phase was completed, SRS staff held telephone conferences with the contractor to 

discuss the results and variations in protocol. The contractor was asked to provide data 

regarding outcomes from each approach. There was a need to complete the survey quickly and 

with the highest response rate possible, but also a need to create information about how 

incentives might be useful for the survey in the future. Thus, after each phase, the protocol for 

the next phase was developed, in a short timeframe, using the information provided by the 

contractor to the extent possible to help assess (1) the value of using an incentive (Is it a costeffective way of increasing the response rate?), and (2) which protocol works best to increase 

response rate and survey timeliness. Therefore, we attempted to structure the incentive 

approaches in as much of an experimental manner as possible in order to provide results about 

incentives that could be of use to SRS and to other organizations. 

Phase 1 

Note that while we analyze the data as though cases were randomly assigned, in the actual 

assignment protocol created by the contractor, payment levels were assigned to interviewers. 

Each incentive amount was randomly assigned to an interviewer across the three days. Each 

interviewer was assigned a group of cases each day and only that interviewer worked on those 

cases that day. As interviewers made contact with an eligible respondent, an offer was made to 

the respondent and that case was assigned to the payment group based on the offer amount 

made. For example, interviewer A was assigned to offer $20 for cooperation on the first day. 

Interviewer A called 100 numbers on that day in which A was allowed to offer the $20 

incentive. Of the 100 numbers called, interviewer A made contact with 20 eligible respondents 

in which the cash incentive was offered. These 20 respondents became part of the Phase 1 $20 

group. (Note: SRS has made no formal assessment of how this randomization scheme has 

impacted the generalizability of the “experiment,” but wishes to note the variation from case 

randomization so that readers use appropriate cautions when considering the results.) 

The first payment incentive phase lasted for three days. At the end of the three-day period, 175 

cases had been contacted and offered an incentive while 61 cases had been contacted but not 

offered an incentive (Table 1). The effort resulted in 50 additional respondents from those 

offered incentives (34 completes and 16 appointments later completed). All other cases 

available to phase 1 sampling were not reached during this weekend experiment. The number 

of completes increased by almost 100, up to 1,705 at the completion of phase 1. At this point, 

the contractor anticipated having to pay 80 percent of the remaining 295 cases needed to reach 

the goal of 2,000. 

The results of phase one of the experiment indicated that approximately 15 percent of 

respondents who were personally contacted by an interviewer completed the interview 

immediately, without the offer of an incentive (see Table 1). An additional 41 percent of 

contacted respondents were willing to make an appointment for a later date, but it is important 

to recognize that many of these appointments were not kept. Nearly 20 percent of respondents 

who were contacted personally by an interviewer and offered some financial incentive for 

participation agreed to do the interview immediately and an additional 36 percent of this group 

made an appointment for a later date. The 19.4 percent completed among all those offered a 

cash incentive is not statistically significant from the 14.8 percent completed in the 
nonincentive group. The appointment rates do not differ significantly either. Refusal rates 

were remarkably consistent among groups. 

The weekend experiment results found in Table 1 show that the percent of respondents who 

continue to refuse is not influenced by the offer of an incentive. But the number of immediate 

completions is slightly higher and the number of deferrals to a latter date is slightly lower for 

respondents offered an incentive, considering the three levels of incentive in the aggregate. 

In the Phase 1 (weekend) experiment, respondents were assigned to one of four groups 

randomly. Group One received no incentive offer. Group Two received an offer of $10 for 

participation. Group Three received an offer of $20 for participation. Finally, Group four 

received an offer to $30 to participate in the study. The results – although not statistically 

significant due to small cell populations – suggested that $30 might be most effective, with 24 

percent of the contacted respondents agreeing to be interviewed immediately (see Table 1). 

Phase 2 

For the two weeks after the weekend experiment, the contractor proposed that uncooperative 

respondents be offered $10 for participation initially, followed by a $20 incentive if they still 

refused, and capped by a “final” $30 offer for participation. The contractor believed that, given 

the results of the weekend experiment, it appeared likely that this procedure would yield the 

highest number of completed cases (and the highest response rate) in the shortest period of 

time. Further, the contractor believed that the approach would cost approximately the same 

amount that would be required for a longer field period with no incentive or with a lower 

incentive. Given the time frame in which a decision needed to be made and insufficient data to 

support the case, SRS would not accept this approach. 

Many of the remaining households in the study had been contacted numerous times in attempts 

to contact the selected respondent, but the respondent was unavailable. Often, indicating that 

the selected person is not available is simply a way of declining the interview without directly 

saying so. Yet, some respondents have work schedules that keep them away from their 

telephone most evenings and weekends, including workers in the retail trade, food services, 

travel, and recreation industries. When this survey was conducted in 1997, the field period was 

extended for nearly a month to obtain 2,000 interviews and a response rate of 70 percent. On 

the basis of the experience with the survey to date and available data, SRS and the contractor 

negotiated to reach an agreement that data collection continue after the weekend experiment, 

using a $10 incentive. 

For the two-week (May 17 to June 1) phase 2 collection period, the contractor continued to 

make calls and interview respondents, offering a $10 incentive to all individuals (not already 

scheduled for call back) who refused to participate in the study. (During this period those 

pending cases from the original experiment weekend continued to be worked and their original 

incentive was offered if contact was made.) During this two-week period, interviewers were 

able to make contact with the selected respondent in 34 percent of the outstanding cases. 

Interviews were completed with 18 percent of the contacted respondents Appointments were 
made for an additional 32 percent of the cases (Table 1). This effort brought an additional 72 

respondents. 

The small number of cases and the lack of a control group in phase 2 largely precluded 

obtaining statistically significant information from phase 2 data. Still, one might argue that 

there is some slight increase in completions. Further, this group, already prone not to 

cooperate, may be inclined to make a faster decision of final refusal using the phase 2 incentive 

(49 versus 44 percent). 

Table 1: Offers and Dispositions for the Incentive Approaches and Additional Conversion Efforts, 1999 

Survey of Public Attitudes Toward and Understanding of Science and Technology 

Type of 

Incentive 

Total 

N of 

Cases 

Number 

of R’s 

Offered 

Incentive 

Number Percent 

Comp. 

Interview 

Appointment Refusal Comp. 

Interview 

Appointment Refusal 

Phase 1 - Weekend Experiment 

None 325 61 9 25 27 14.8 41.0 44.3 

$10 325 52 10 19 23 19.2 36.5 44.2 

$20 325 65 10 26 29 15.4 40.0 44.6 

$30 325 58 14 18 26 24.1 31.0 44.8 

Offers 

combined

 975 175 34 63 78 19.4 36.0 44.6 

Total 1300 236 43 88 105 18.2 37.3 44.5 

Phase 2 - Additional Two Weeks of Refusal Conversions 

$10 1176 395 72 127 196 18.2 32.2 49.6 

Phase 3 – Close-out Two-level Experiment 

None 280 278 23 NA 96 8.3 NA 34.5 

$20 529 529 79 NA 129 14.9 NA 24.4 

NA – not applicable, survey closeout 
At the end of the second phase of the experiment, 115 respondents had been offered and 

accepted financial compensation. After completing the survey, a few respondents changed 

their minds because they did not want to give out their mailing addresses (to receive their 

checks), so the actual number getting compensation was less than 115. The total amount given 

out was $1,110. The incentive option was budgeted at $2,000, and while the contractor did not

go over that budget, the budget for interviewing time was exceeded. 

The contractor was encouraged by the results. The contractor believed that a three to four 

percentage point improvement might be obtained in the immediate completion rate, which 

would save significant interviewer costs in the follow-up and tracking of these cases. 

Furthermore, if this pattern were to continue, the study would be completed sooner and there 

would be a smaller chance of any unforeseen events occurring during the field period. Thus, 

the contractor requested permission to continue data collection after phase 2 with an offer of a 

financial incentive of at least $10 for participation in the study. 

Phase 3 

After phase two of the incentive activities, all data collection for this survey was halted until 

sufficient data was available to justify an approach to a continuation of the survey and the 

supplemental funds requested by the contractor. SRS requested more analysis to assess the 

need for the incentives to complete the survey and achieve the desired response rate. 

The first additional information requested concerned the disposition of the 88 appointments 

reported by the contractor from the original weekend experiment. An additional 17 

completions were obtained from this group during the following two weeks. The contractor 

did not report the number of hard refusals, but the project manager reported that virtually all of 

the remaining cases were still eligible to be tried again. 

The additional completions made a much stronger case for the value of the incentive offers. 

Only one additional completion was obtained from the cases to which no incentive offer had 

been made, while 16 additional completions were obtained from respondents to which some 

level of prior offer had been made. 

Next, SRS inquired into the number of the 72 completions during the two weeks of phase 2 

that required an incentive. The contractor reported that all of the 72 respondents were offered a 

$10 incentive, but that only 67 actually provided a name and address for the mailing of a 

check. The other five declined to provide that information, or concluded that a check for $10 

was not worth the effort. 

Agreement to fund and continue the incentive experimentation was reached. The continuation 

of data collection conformed to the following adjustments to the experimental design: 

A. All outstanding cases for which an appointment was scheduled remained in the 

experimental group for which the appointment was last discussed. Experimental 

groups to that point include Phase 1 and 2. These carryover appointments would 

not be part of the phase 3 incentive experimental cases. 
B. The experimental units, the remaining cases from the experiment not having 

been offered one of the experimental "treatments" (1,176 minus the 395 phase 2 

contacts minus some extraneous units, were randomly assigned to one of two 

additional experimental groups as follows: 

1. 1/3 of the cases, but not less than 280 cases were to have no incentive 

offered. 

2. 2/3 or the balance of the cases if the minimum of 280 was necessary in 

group 2 would have a fixed incentive of $20 offered. 

C. Increasing incentive levels would not be offered. 

D. Cases from groups 2 and 3 above would be randomly assigned to interviewers. 

The final respondent fee experiment was initiated on June 30 (Phase 3) and was stopped on 

August 5. Assignment to the two new treatment groups, those offered $0 and those offered 

$20 were assigned per randomization instruction supplied by SRS. Table 1 show that the phase 

3 $20 incentive was more likely to result in a completed interview than no offer (15 versus 8 

percent). 

Conclusions 

The experimentation with incentives during this survey indicates that a monetary incentive 

resulted in at least a small increase in completion rates during the closeout period of this 

survey. The incentive may also help refusals give that indication earlier in the sequence of 

callbacks, thus saving recall resources. Before basing decisions to implement similar incentive 

approaches in future surveys, SRS also needs to study the call back counts and other cost 

information. Since data quality is not based solely on response rate, but depends upon the 

nonresponse adjustment methods used, the incremental responses impact on quality of overall 

survey results needs to be measured against the added cost to the survey. Recognizing that 

response rates for RDD surveys are declining, SRS needs to design experiments up front for 

future surveys, rather than being forced into the position of creating ad hoc approaches and 

experiments under the pressure of survey deadlines. 

Acknowledgements 

The author thanks the following people for assistance and/or guidance during the survey and/or for providing 

information used in this report -- Mary Frase and Melissa Pollak, NSF; Daniel Warfel, OMB; Jon Miller and 

Linda Kimmel, Chicago Academy of Sciences; Mary M. Hess, National Opinion Research Center 

Disclaimer 

The information contained in this paper is the sole opinion of the author and does not necessarily reflect the 

position of the Division of Science Resources Statistics, NSF, or those contributing data or other assistance during 

the project.