United States

Department of

Agriculture

National

Agricultural

Statistics

Service

Research and

Development Division

Washington DC 20250

RDD Research Report

RDD-06-07

December 2006

The Effect of

Incentives on

Response in 2005 

ARMS Phase III

Interviews

Jaki S. McCarthy

Daniel G. Beckler

Kathleen Ott

The views expressed herein are not necessarily those of NASS or USDA. This report was prepared for 

limited distribution to the research community outside the U.S. Department of Agriculture.
EXECUTIVE SUMMARY

Phase 3 of the Agricultural Resource Management Survey (ARMS) is the one of the longest and

most detailed sample survey data collections conducted by the US Department of Agriculture’s

National Agricultural Statistics Service (NASS). For this survey, NASS collects highly detailed 

economic data covering a calendar year from agricultural producers nationwide. Previous

research with a self-administered mail-out/mail-back version of the ARMS questionnaire

(Beckler, Ott and Horvath, 2005) showed benefits of indirect monetary incentives ($20 ATM

cards). Face-to-face interviews are used to collect data from the Costs and Returns Report

(CRR) sample of the ARMS Phase 3. For 2005, use of ATM cards was continued as standard

practice with the mail-out/mail-back Core form sample and an incentive experiment was

conducted comparing several types of incentives provided to ARMS Phase 3 CRR sampled

operations.

Five treatment groups (including a control group) in the CRR sample were used for this incentive

experiment. All experimental groups received a pre-survey letter and were then contacted for a

face-to-face interview. Treatment groups received either 1) a standard pre-survey letter with no

incentive, 2) pre-survey letter with a prepaid indirect cash incentive (in the form of $20 

automated teller machine (ATM) cards), 3) pre-survey letter with a promised individual financial

profile comparing the operation with aggregated information about other operations, 4) presurvey letter with a NASS logo wall clock or 5) pre-survey letter with a pre-paid $20 ATM card

and a promised individual financial profile. The entire ARMS Core version was provided with a

$20 incentive but was not included in this experiment. Information for the Core form sample is

provided along with the experimental groups.

Response rates for the CRR incentive groups were slightly higher than the control group.

However, the differences were not significantly different, with response rates for all groups in 

the low 70 percents. Each incentive tested required an additional cost for the incentives

themselves and also for processing and delivery of the incentive.

Used as tested, these incentives do not appear to be effective in increasing response rates on the

face-to-face ARMS CRR sample. This is contrary to our prior evaluation of their use with the

mail-back Core Version of the form where response rates were increased. For the Core form

sample, incentives raised response rates and were also cost effective since they increased mail

response and reduced costs from face-to-face interviews. In addition, the number of cards

cashed was low, and the Core Version non-respondents cashed their ATM cards at a much lower

rate than respondents (both in the previous research and in the current year). In the current

research, Phase 3 CRR interview respondents and non-respondents cashed the ATM cards at

comparable rates. This suggests that incentives, while increasing costs, are of limited

effectiveness when combined with a face-to-face interview mode. Careful consideration should

be made regarding how incentives are used in the future as they do not appear to be universally

effective.
RECOMMENDATIONS

1. Incentives (both monetary and non-monetary), as tested in this experiment, should not be

used for ARMS Phase 3 samples which use only face-to-face interviews. This is because

they raise costs but do not significantly increase response rates.

2. A small subsample control group should be used to monitor the continued effectiveness

of incentives with the mail-back Core version of ARMS 3.
2

The Effect of Incentives on Response in 2005 ARMS Phase 3 Interviews

Jaki S. McCarthy, Daniel G. Beckler, and Kathleen Ott

1/

Abstract

The United States Department of Agriculture’s National Agricultural Statistics

Service conducts the annual Agricultural Resource Management Survey (ARMS) in 

several phases. Phase 3 of this survey collects detailed economic information with

a lengthy Costs and Returns Report (CRR) questionnaire administered in face-toface interviews. An additional sample uses a sixteen-page Core questionnaire with 

mail-out/mail-back data collection and face-to-face nonresponse follow-up. Based 

on positive results of incentive use with the 2004 ARMS Core form, both prepaid

and promised incentives were tested on the CRR face-to-face interview sample of

the 2005 ARMS in order to increase response rates. Incentives included $20 ATM

cards, NASS logo wall clocks, and Individual Farm Analyses.

The results showed that incentives, as implemented, did not significantly increase

the overall response rates. Due to the increased cost and processing required to 

provide incentives, their use is not recommended for future ARMS face-to-face

interview samples.

Key Words: Incentives; nonresponse; response rate; refusal conversion.

1. INTRODUCTION

The Agricultural Resource Management

Survey (ARMS) is an annual survey

conducted by the United States Department

of Agriculture’s (USDA) National

Agricultural Statistics Service (NASS) and

co-sponsored by USDA’s Economic

Research Service (ERS). The ARMS

consists of three phases: Phase 1 screens for

potential samples for Phases 2 and 3; Phase

2 collects data on cropping practices and

agricultural chemical usage; and Phase 3

collects detailed economic information 

about the agricultural operation as well as

the operator’s household.

ARMS Phase 3 has been problematic

because, compared with other NASS

surveys, its response rates are low, and its

data collection costs are high. All ARMS

Phase 3 data are collected by face-to-face

enumeration because of the length and

complexity of the questionnaire.

The ARMS Phase 3 contains several distinct

subsamples. The Cost and Returns Report

(CRR) sample provides information for the

financial analysis of farm businesses, farm

households and costs associated with

producing agricultural commodities. In

addition, cost of production and expenditure

data are obtained with specific individual

commodity subsamples. The CRR and

commodity subsamples are all conducted

with face-to-face interviews. The CRR

1/ Daniel G. Beckler and Kathleen Ott conducted this research while mathematical statisticians with the USDA’s

National Agricultural Statistics Service - Research and Development Division (RDD), Jaki McCarthy is also a

statistician in the RDD, located at Room 305, 3251 Old Lee Highway, Fairfax, VA 22030.
2

questionnaire is over 30 pages long and

interviews average about 11⁄2 hours in length.

A much shorter self-administered Phase 3

“Core” questionnaire developed for the 2003 

survey was used for part of the sample for

mail-out/mail-back data collection.

Since the utility of a self-administered

ARMS Phase 3 Core form was demonstrated 

in 2003, NASS sought ways to implement

its use and increase the overall response rate

as well as contain data collection costs.

Offering potential respondents incentives is

a proven technique to increase response

rates on a variety of surveys conducted by 

several agencies and companies (Church,

1993; James and Bolstein, 1992; James and

Bolstein, 1990; Singer, 2002). An incentive

experiment conducted with the 2004 ARMS

Phase 3 Core indicated that monetary 

incentives increased response rates for the

mail-out/mail-back sample of this survey 

while being cost effective (Beckler, Ott and 

Horvath, 2005).

Based on the promising results of the use of

incentives with the mail-out/mail-back Core

form, it was hypothesized that incentives

could also boost response rates in the ARMS

face-to-face interviews. Therefore, an

experiment comparing incentive use with

the CRR face-to-face interview sample of

the 2005 ARMS Phase 3 was conducted.

2. METHODS

The 2005 ARMS Phase 3 Costs and Returns

Report (CRR) sample size was 11,625

including list and area frame samples. Ten 

thousand list frame sample operations were

randomly assigned to 5 treatment groups.

The total sample was stratified by state, the

ARMS farm value of sales (as maintained 

on the sampling frame), and the type of

operation (also as maintained on the

sampling frame). Then, five sub-samples,

each of size 2,000, were systematically

selected. The sub-samples were drawn such 

that each was equally represented by the

strata. Once the sub-samples were drawn,

NASS Field Offices had the opportunity to

remove operations from the samples with 

whom they had previous data collection 

agreements. This resulted in each subsample being slightly less than 2,000.

In addition, the Core sample size was 16,498 

farm/ranch operations across 15 states. The

15 states were those with the highest

agricultural value of sales and included 

Arkansas, California, Florida, Georgia,

Illinois, Indiana, Iowa, Kansas, Minnesota,

Missouri, Nebraska, North Carolina, Texas,

Washington, and Wisconsin.

For the 2005 ARMS Phase 3 sample, data

collection could begin January 30 and ended

in mid April. All data for the 5 treatment

groups were collected via face-to-face

interviews.

The National Processing Center (NPC) in 

Jeffersonville, IN printed all of the presurvey letters and assembled and mailed all

of the pre-survey materials to the treatment

groups. They also assembled and mailed the

Core sample questionnaires and ATM cards.

Each of the 5 treatment groups were mailed

a pre-survey letter, as is standard data

collection methodology. In addition,

treatment groups received incentives with

their pre-survey letters in the form of $20

automated teller machine (ATM) cards, a

non-monetary incentive in the form of a

NASS logo wall clock, and the promise of a

Farm Profile for their operation. The Farm

Profile or “Individual Farm Analysis” was a

report that displayed the percentile within

which the operation’s data fell compared to

an overall group estimate for each of 16 
3

expense categories. An example appears in

Appendix A, Figure A3. Generic profiles

without individual operation data were

generated for non-respondents. The Farm

Profiles were generated individually by staff in

NASS’s Statistics Division, printed in

Headquarters, and sent to the appropriate

NASS field offices in September 2006 for

distribution to sampled operations.

Combinations of these stimuli were

administered to four of the five sub-samples

mentioned above; a fifth sub-sample

received no stimuli (except the standard presurvey letter) and served as the control

group for this project. Collectively, these

five sub-samples formed the five treatment

groups used for this project. Table 1

contains descriptions of the treatment

groups.

All treatment groups received pre-survey 

letters that included some uses of the ARMS

data. The letters also included a reference to

the type of incentive the operation received

($20 ATM card, NASS logo clock, financial

profile) as an advanced thank you for

participation. The decision to use $20 ATM

cards as incentives was based on their

previous successful use with the ARMS

Core form. Wall clocks were chosen as a

prominent way to display the NASS logo to

respondents. The letters and examples of

the incentives are shown in Appendix A1-

A6.

The sub-samples in the incentive treatment

groups all received pre-survey letters that:

(1) explained the incentive was a “thank

you”, and; (2) described the uniqueness of

the ARMS. Because all interviews were to 

be completed via personal interview, there

was no statement justifying the ATM card 

being used as a cost saving measure. (This

statement was included in the cover letter

for the Core forms which were to be

completed and returned via mail but were

not part of this experiment.)

The actual ATM card incentive was mailed

with the pre-survey letter to recipients via

United States Postal Service and was affixed

to a standard 81⁄2 inch x 11 inch sheet of

paper that reiterated a “thank you” and

included instructions on how to use the card.

This is shown in Appendix A, Figure A6.

The wall clocks, with a pre-survey letter,

were delivered via Federal Express.

The $20 ATM cards were supplied by

Table 1: ARMS Phase 3 Version 1 Treatment Groups

Treatment

Group

First Contact Interview Contact

1 (Control) Pre-Survey Letter

No Incentive

Face-to-Face Interview

No Incentive

2 Pre-Survey Letter

$20 ATM Card Incentive

Face-to-Face Interview

No Incentive

3 Pre-Survey Letter

Mention Individual Financial Profile Incentive

Face-to-Face Interview

Promise Individual Financial Profile Incentive

4 Pre-Survey Letter

Non-Monetary Clock Incentive

Face-to-Face Interview

No Incentive

5

Pre-Survey Letter

$20 ATM Card Incentive

Mention Individual Financial Profile Incentive

Face-to-Face Interview

Promise Individual Financial Profile Incentive
4

JPMorgan Chase bank and were usable in

nationwide ATM machines that displayed

the NYCE®, Pulse

®, Maestro®, or Cirrus®

logos. The cards were also usable at pointof-sale (POS) (i.e., retail) establishments

that allow the use of debit cards as payment;

however, recipients were not explicitly told

this. In addition to the $20 incentive, the

ATM cards were loaded with an extra $4 to

cover any transaction charges. The cards

were pre-activated and were immediately

usable when the recipients received them.

The personal identification number (PIN)

needed to use the card was embossed on the

front of each card after the words “THANK

YOU”.

1 The front of each card also

included the embossed message, “FOR

HELP 1-888-424-7828”; this toll-free

telephone number was answered by NASS

staff. Finally, all ATM cards expired on

May 31, 2006 (there was no provision for

extending this date). If a card recipient lost

or could not use the card, a replacement

could be requested by calling the toll-free

phone number listed on the instruction sheet.

For the few cases where this occurred, only 

the replacement card was used in our

analysis.

Interviewers were instructed to schedule

interviews using the same procedures as in

previous years when no incentives were

provided to the CRR sample. Interviewers

were informed of which CRR respondents

received incentives. However, other Federal

Agencies who have used monetary

incentives have found that when 

interviewers ask respondents about receipt

 1 There was a problem with how some card recipients were

told to use their cards. This may have impacted how many

recipients were able to cash their cards. The PIN was created

from the cards “sequence number”. For example, suppose

the sequence number was “12345”; the actual PIN was

“2345”. The initial instructions did not acknowledge this or 

how to correctly enter the PIN. A postcard was sent a week

or two after the problem was discovered alerting recipients

(with five digit sequence numbers) to only enter the last four 

digits as the PIN.

of the incentives, potential respondents often

report they did not receive the incentive and

ask for a replacement to be provided (see

McGrath, 2006). This significantly

increases costs and NASS did not have

sufficient additional ATM cards to meet

substantial numbers of replacement requests.

For this reason, interviewers were

specifically instructed NOT to mention the

incentives in their contacts with respondents.

3. RESULTS AND DISCUSSION

3.1 Response Rates

Unlike previous incentive research 

conducted with the ARMS Core mail form,

the results of this project showed that the

incentives for the CRR sample did not

significantly increase response rates. Table

2 shows the response rates by treatment

group (treatment group descriptions are

given in Table 1). While not part of the

experimental comparisons, the response rate

for the Core form (who all received $20 

ATM cards) is also shown.

As shown in Table 2, although all treatment

Table 2: Response Rates 1/ by Treatment Group

Treatment

Group

N 2/ Completes Response

Rate

Δ from

Control

1

(Control) 1,985 1,395 70.28 --

2 1,958 1,414 72.22 1.94

3 1,967 1,387 70.51 0.23

4 1,965 1,422 72.37 2.09

5 1,955 1,420 72.63 2.35

Core 16,230 11,928 73.49 --

1/ The Response Rate denominator included completes, refusals,

inaccessibles, estimated refusals, and estimated inaccessibles.

2/ Initially, all treatment groups contained 2,000 records, but field

offices removed operations with whom they had previous data

collection agreements (held in office), or otherwise did not

want them to participate in the study. The Ns given here include

only the count of operations that were provided with the

incentives (except the Control, which only excludes the held in

office). Counts are according to information obtained from

NPC.
5

groups that received an incentive had

slightly higher response rates than the

control group (Treatment Group 1), none

were significantly different. See Appendix

B for details on the significance testing.

Pairwise comparisons of the groups’

response rates are shown in Appendix C.

3.2 ATM Card Usage by CRR Sample

In previous research using $20 ATM cards

with the Core Version of the ARMS, NASS

realized a significant cost savings in large

part due to the low number of people who 

actually cashed their cards (Beckler, Ott and

Horvath, 2005.) As shown in Table 3, less

than 30 percent of ATM cards were cashed 

by the treatment groups receiving them in

this face-to-face interview sample. This is

similar to findings of other government

agencies (Kay, Boggess, Selvavel and 

McMahon, 2001). However, unlike the

Core form respondents, the proportion of

CRR sampled operations who cashed the

cards is similar for respondents and nonrespondents. Core form sampled operations

who receive ATM cards but were ultimately 

non-respondents, cashed their ATM cards at

a very low rate, both this year and in

previous research.

This seemingly curious result may have

occurred because the ATM cards were not

described as resulting in cost savings for the

CRR sample. For the Core form,

respondents who received the ATM cards

but did not respond may have felt they were

not entitled to the money or perhaps that

their cards would not be activated without

the return of their form.

In addition, for the Core form sample, the

ATM card was delivered in the same

package as the questionnaire they were to

complete. For the CRR respondents, the

incentives were delivered completely 

separate from the interviewer’s request to 

provide the data (and as mentioned 

previously, the interviewers were instructed

not to mention the incentive.) This may 

have impacted the response rates and card 

cashing rates for the current experiment.

3.3 Costs of Incentive Use

Table 3: ATM Card Usage by Treatment Group 1/

Card Use by Respondents Card Use by Nonrespondents Card Use by All Card Recipients

Treatment

Group Count

Cards Used

For $

Withdrawal

Percent of

Cards

Cashed

Count

Cards Used

For $

Withdrawal

Percent of

Cards

Cashed

Count

Cards Used

For $

Withdrawal

Percent of

Cards

Cashed

2 1,414 369 26.10 544 165 30.33 1,958 534 27.27

5 1,420 371 26.13 535 131 24.49 1,955 502 25.68

Core 11,928 3,561 29.85 4,303 215 5.00 16,231 3,776 23.26

Total 14,762 4,301 29.14 5,382 511 9.49 20,144 4,812 23.89

1/ Counts include distinct operators who withdrew money using ATM cards (also includes those who made Point of Sale (POS) purchases). A small

number of operators received multiple cards (because they requested a replacement claiming the first one did not work), and a small percent of those

actually did cash both of their cards. Counts in this table count those who cashed more than one card only once; card expenses given in Table 4 include the

expenses for all cards cashed.
6

The use of ATM cards and also nonmonetary incentives such as clocks or other

tokens of appreciation add additional costs

both in dollars and other resources. The cost

of ATM cards include the $20 dollars

provided to the respondent who uses the

card plus an additional $4 to cover any fees

that may be charged by the particular

machine used. These fees include

withdrawal and purchase fees, balance

inquiry fees and failure fees. In addition,

since the ATM cards could be used at pointof-sale (POS) debit card machines, users

could withdraw money at a variety of

locations (although recipients were not

explicitly told this.) The fees actually 

incurred by ATM recipients are shown in 

Table 4. Also shown in Table 4 is the

substantial cost of using ATM cards

operationally in the large Core form sample.

For incentives that are not monetary, the

costs of incentives include the incentive

itself, and the additional costs of packaging 

and mailing the incentives. Unlike ATM

cards whose full value is only paid if the

recipient takes action to cash the card, the

entire cost of non-monetary incentives is

spent whether the potential respondent uses

the incentive or not.

The Farm Profiles were designed and 

generated by Statistics Division staff in the

Economics Section. Operations who 

responded had profiles with their operation 

compared to similar operations; for those

operations who did not respond, a generic

profile was generated without individual

operation data

2

. All profiles were created 

some time after the survey contacts (and

were delivered to the Field Offices in

September 2006). Once created and printed,

the profiles were distributed to the

respective field offices for final distribution 

to sampled operations. (Costs associated 

with this final step of field offices

distributing the profiles to the operations

were not tracked and are not included in this

analysis.)

A token incentive, like a wall clock, is more

likely to be noticed due to its large size.

However, it costs much more to package and 

mail these types of incentives. The costs of

each of the incentives used in this study are

shown in Table 5.

 2 For sampled operations who reported incomplete

data or refused to participate, the generic profiles

included only information about farms of similar size

and type (based on list frame control data for that

operation). The cover letter for these profiles

acknowledged their incomplete data or refusal.

Table 4: ATM Card Charges by Treatment Group

1/

ATM Withdrawal 2/ ATM Withdrawal

Fee 3/ POS* Purchase 2/ POS Purchase Fee 3/ Balance InquiryFee

Transaction

Failure Fee Treatment

Group Count Cost ($) Count Cost ($) Count Cost ($) Count Cost ($) Count Cost ($) Count Cost ($)

Total

Cost

2 532 11,504.31 537 539.00 8 105.48 8 2.80 17 7.65 51 15.20 $12,174.44

5 501 10,844.05 514 514.00 5 63.69 5 1.75 19 8.55 39 11.40 $11,443.44

Core 3,772 80,994.27 3,760 3,760.00 34 514.24 34 11.90 105 47.25 238 70.45 $85,398.11

* = Point of Sale (POS), i.e., the ATM card was used as a debit card.

1/ Counts reflect the numbers of each type of transaction that were made. The sum of “ATM Withdrawal” and “POS Purchase” is slightly higher than the counts given in

Table 3 because Table 3 reflects counts of distinct operators (i.e., state/POIDs), whereas the counts in this table reflects those who used their cards for multiple

withdrawal/purchase transactions and the very few operators who received and cashed multiple cards.

2/ Includes amount of withdrawal/purchase as well as any transaction fees imposed by the ATM owner/retailer.

3/ JPMorgan Chase transaction fee. JPMorgan Chase was the issuer of the ATM cards.
7

Because the incentives cost more but did

not significantly impact response rates, the

control group who received nothing more

than the pre-survey letter was the least

expensive in both cost per sample and

average cost per completed interview. This

is in stark contrast to the results of using 

incentives with a mail-back form where

substantial cost savings can be realized by 

reducing the number of personal interviews,

even if overall response rates are not

affected. It should also be noted that the

total additional cost of incentives would be

multiplied if it extended to the entire sample

instead of a small treatment group.

3.4 Data Quality and Non-response Bias

As there were no significant differences in 

response rates across the treatment groups,

no analysis of potential non-response bias or

potential impact on data quality was done.

Even though non-response rates are

comparable across groups, it is possible that

characteristics of non-respondents differ

across groups. However, we did not

examine this.

4. Conclusion

In the research on incentives with the 2004 

Core form, there were significant increases

in response for groups receiving the $20 

ATM cards. In contrast, the current research 

did not find incentives to be effective in 

raising response rates. There are several

possible reasons for this.

First, prior results were based on incentive

use for a mail-out/mail-back form (with

face-to-face follow up). The stated 

justification to respondents for providing 

incentives in a mail survey is to save the

costs of personal enumeration. There may

be a set of operations that may only be

prompted to respond by mail, if provided 

with an incentive described this way. The

justification to save costs cannot be provided

to potential respondents in a face-to-face

interview survey.

In addition, with a mail survey, the incentive

can be delivered with the questionnaire

itself, so the incentive and the survey request

Table 5: Overall Costs by Treatment Group

Costs (In Dollars)

Treatment

Group

N Postage 1/

NPC

Printing 2/

NPC

Extra

Admin 3/

Incentive

Cost 4/

ATM

Card

Admin5/

Enumeration

Costs

6/ Total

Average

Per

Sample

Average

Per

Complete

1 (Control) 1,985 598 88 – 0 0 254,080 254,766 128 183

2 1,958 597 176 861 12,174 0 250,624 264,432 135 187

3 1,967 600 89 – 10,984 0 251,776 263,449 134 190

4 1,965 10,198 88 5,895 18,505 0 251,520 286,206 146 201

5 1,955 596 176 860 22,359 0 250,240 274,231 140 193

1/ $0.305 each for first class; $5.19 each for FedEx (used for the clocks)

2/ Pre-survey letter at $0.045 each plus (if appropriate) debit card instruction sheet at 0.045 each.

3/ $0.44 each for hand stuffing the ATM cards; $3.00 each for packaging the clocks.

4/ Includes ATM card costs or the cost of the non-monetary incentives, [Clocks at $18,505 ($16,400 for the clocks, $80 set up fee, $2,025 shipping fee).

Individual Financial Profiles at $21,900 (20,000 in estimated staff costs, $1,500 in mailing costs, and $400 in printing/paper costs)]. See Appendix D for 

more detailed description.

5/ JP Morgan did not charge NASS the standard $0.85 ATM Administrative Fee per card since the delivery of the ATM cards was delayed.

6/ 2005 Agricultural Resource Management Survey (ARMS) Phase 3 Survey Administration Analysis: US average per sample ($128) multiplied by the

number in the treatment group.
8

are strongly linked. With a face-to-face

interview survey, the incentive is delivered 

separately and not directly linked to the

survey request (and indeed, our interviewers

were explicitly told NOT to attempt to link 

the incentive to their survey contact). We

have no way of knowing if making this

connection more explicit (for example, by 

the interviewer asking about it when

attempting to schedule an interview) might

have increased response.

Incentives can be beneficial in both 

increasing response rates and decreasing 

costs, but this may be highly related to the

way in which the incentives are used. Even

those incentives that have been effective in

the past should be evaluated periodically to 

make sure that they continue to be effective.

While we assume that the use of ATM cards

with the Core form sample was beneficial in 

2005, as it had been in 2004, we have no

way of knowing this. Including a small

sample control group for comparison would

provide this information. As other

organizations employ incentives, their

novelty may wear off and decrease their

effectiveness. In addition, if inflation

changes the value of $20, this may no longer

be an effective amount to stimulate

response. Finally, if more respondents cash 

their cards, the cost of providing incentives

would increase.

5. Recommendations

Based on the results of the present study, the

following recommendations are offered:

1. Incentives (both monetary and nonmonetary), as tested in this

experiment, should not be used for

ARMS Phase 3 samples which use

only face-to-face interviews. This is

because they raise costs but do not

significantly increase response rates.

2. A small subsample control group 

should be used to monitor the

continued effectiveness of incentives

with the mail-back Core version of

ARMS 3.

6. REFERENCES

Beckler, D.G., Ott, K., Horvath, P. (2005)

Indirect Monetary Incentives for the 2004 

Phase 3 Core. Research and 

Development Division Research Report

RDD-05-05, United States Department of

Agriculture, National Agricultural

Statistics Service.

Church, A. H. 1993. “Estimating the Effect

of Incentives on Mail Survey Response

Rates: A Meta-Analysis.” Public Opinion

Quarterly 57:62-79.

James, J. M., and R. Bolstein. 1992. “Large

Monetary Incentives and Their Effect on

Mail Survey Response Rates.” Public

Opinion Quarterly 56:442-453.

James, J. M., and R. Bolstein. 1990. “The

Effect of Monetary Incentives and 

Follow-Up Mailings on the Response

Rate and Response Quality in Mail

Surveys.” Public Opinion Quarterly

54:346-361.
9

Kay, W. R., S. Boggess, K. Selvavel, and M.

F.McMahon. 2001. “The Use of Targeted 

Incentives to Reluctant Respondents on

Response Rate and Data Quality.”

Proceedings of the Survey Research

Methods Section, American Statistical

Association.

McGrath, D., 2006. “An Incentives

Experiment in the US Consumer

Expenditure Quarterly Survey”

Proceedings of the American Statistical

Association, Joint Statistical Meetings.

Singer, E. 2002. “The Use of Incentives to

Reduce Nonresponse in Household

Surveys.” In Groves, R. M., D. A.

Dillman, J. L. Eltinge, R. J. A. Little

(Eds.) Survey Nonresponse (pp. 163-177).

New York: John Wiley & Sons, Inc.

Snedecor, G. W. and W. G. Cochran. 1989.

Statistical Methods: Eighth Edition.

Ames, Iowa: Iowa State University Press.
10

APPENDIX A: Treatment Group Survey Materials

Figure A1: Pre-Survey Letter for Treatment Group 1 (Control)
11

Figure A2: Pre-Survey Letter for Treatment Group 2 ($20 ATM card)
12

Figure A3: Pre-Survey Letter for Treatment Group 3 (Financial Profile)
13

Figure A3: 4 page Farm Profile Example provided to Treatment Groups 3 and 5
14
15
16
17

Figure A4: Pre-Survey Letter and Clock for Treatment Group 4 (NASS Logo Clock)
18

NASS Logo Clock
19

Figure A5: Pre-Survey Letter for Treatment Group 5 ($20 ATM card and Financial Profile)
20

Figure A6: Presentation and Instruction Sheet for ATM Cards
21

APPENDIX B: Response Rate Significance Calculations

Multiple comparisons of the binomial parameters of interest (i.e., the proportion,

! i , of survey 

response in each treatment group) were tested using the t-test statistic, given below, for the

overall mail and overall response rates. One-sided tests were used for those comparisons that

involved Treatment Group 1 (i.e., the control group) since only positive treatment effects were of

interest (and expected). Two-sided tests were used for all comparisons not involving Treatment

Group 1. A Bonferroni adjustment was made to control the error rate for the family of ten 

significance tests conducted. Refer to Snedecor and Cochran (1989) for more details on this test.

For one-sided tests comparing treatment groups with control:

Consider 

H0 t 1 :! " ! > 0

for treatment group

t = 2,3,4,5

t-test statistic 

zW =

( ) ( )

$ $

$ $ $ $

! !

! ! ! !

t

t t

t n n

"

"

+

" #

$

%

%

&

'

(

(

1

1 1

1

1 1

Significant at α=0.05 if 

zW

>

z !

10

= 2.58

For two-sided tests comparing treatment groups with each other (excluding control group):

Consider 

H0 t i :! " ! = 0 for treatment group 

t = 2,3,4,5

and 

i = 2,3,4,5

where

t ! i

t-test statistic 

zW =

( ) ( )

$ $

$ $ $ $

! !

! ! ! !

t i

t t

t

i i

i n n

"

"

+

" #

$

%

%

&

'

(

(

1 1

Significant at α=0.05 if  | zW| >

z !

10

= 2.81
22

APPENDIX C. Pairwise comparisons of Treatment group response rates

Comparisons of Response Rates

Comparison

Response

Rate

Difference 1/

t-test

Statistic 2/p-value*

Treatment Group 2 versus Treatment Group 1 1.94 1.3457 0.0892

Treatment Group 3 versus Treatment Group 1 0.24 0.1628 0.4354

Treatment Group 4 versus Treatment Group 1 2.09 1.4522 0.0733

Treatment Group 5 versus Treatment Group 1 2.36 1.6388 0.0507

Treatment Group 3 versus Treatment Group 2 -1.70 -1.1804 0.2380

Treatment Group 4 versus Treatment Group 2 0.15 0.1049 0.9165

Treatment Group 5 versus Treatment Group 2 0.42 0.2924 0.7700

Treatment Group 4 versus Treatment Group 3 1.85 1.2864 0.1984

Treatment Group 5 versus Treatment Group 3 2.12 1.4727 0.1409

Treatment Group 5 versus Treatment Group 4 0.27 0.1878 0.8510

1/ In percentage points.

2/ Comparisons involving Treatment Group 1 (the control) are based on a one-sided ttest; comparisons not involving Treatment Group 1 (the control) are based on a two-sided

t-test. One-sided tests were used with comparisons involving the control group because it

was assumed that any effect the incentive had would be positive.

Note: originally applied a Bonnferroni adjustment to account for the 10 comparisons that

were performed. However, this resulted in several p-values exceeding one, which is not

possible, so the Bonnferroni adjustment was removed and the “raw” p-values are

reported.

*None of the comparisons is statistically significant at the α = 0.05 level.
23

APPENDIX D: Cost Calculations

NPC Extra Admin

This amount is the National Processing Center’s (NPC) best estimate for the labor cost

associated with administering the incentive program for all treatment groups. This included $.44

each for handstuffing the ATM cards and $3 each for packaging the clocks.

Incentive Costs

Incentive costs include all charges associated with ATM card usage among the recipients and the

costs of non-monetary incentives. ATM charges include: ATM withdrawals, ATM withdrawal

fees, POS purchases, POS purchase fees, ATM balance inquiry, ATM failure fees, POS purchase

failure fees. Refer to Table 5 for details on each of these individual charges. Costs for the

clocks included $16,400 for the clocks (2050 were ordered to include any necessary

replacements), $80 in set up fees, and $2,025 in shipping fees. Costs of the Farm Profiles

included $20,000 in staff time (10 weeks @ $2,000 per week), $1500 in mailing costs and $400 

in printing costs. The total cost of $21,900 was divided by the total number of Farm Profiles

produced (3922) to get a rate of $5.58 per Farm Profile. This was then multiplied by the number

of records in the treatment group.

ATM Card Admin

ATM Card Admin would normally include the amount JPMorgan Chase charged for the physical

cards. In the 2004 study, this was computed as follows:

ATM Card Admin = $0.85

ni

for i=2,4,5

where:

ni = the sample size of incentive i

NASS was charged a “per card” cost of $0.85 (appearing in the above equation).

However, due to a mistake by JP Morgan Chase the fee was waived for this year.

Enumeration Costs

Enumeration Costs include enumeration costs for face-to-face interviews. Since all states

participated in this study, the US average costs as shown in the Survey Administration Analysis

were used to compute enumeration costs. The US average was $128 per sample. This number

was multiplied by the number of records in the treatment group.

Total

Total = Postage + NPC Printing + NPC Extra Admin + Incentive Costs + ATM Card 

Admin + Enumeration Costs
24

Averages

Average per Sample = Total /

ni

Average per Complete = Total /

ri

where:

ri = total survey responses in treatment group i (Item Code 910 in (1,3,5) where 1=mail complete,

3=non-mail complete, 5=out of scope)

ni = total survey samples