i

United States

Department of

Agriculture

National

Agricultural

Statistics

Service

Research and

Development Division

Washington DC 20250

RDD Research Report

RDD-05-05

November 2005

Indirect Monetary 

Incentives for the

2004 ARMS Phase III

Core

Daniel G. Beckler

Kathleen Ott

Peter Horvath

The views expressed herein are not necessarily those of NASS or USDA. This report was prepared for 

limited distribution to the research community outside the U.S. Department of Agriculture.
i
i

EXECUTIVE SUMMARY

Phase III of the Agricultural Resource Management Survey (ARMS) is the longest and most

detailed sample survey data collection that NASS conducts. On this survey NASS collects

highly detailed economic data from agricultural producers nationwide. The ARMS Phase III

Core version is a self-administered data collection instrument that uses a mail-out/mail back

methodology. For 2004, an incentive experiment was conducted using this component of the

ARMS. The use of incentives, particularly monetary ones, in surveys has proved to be effective

at increasing response rates.

Five treatment groups, including a control group, were used for the incentive experiment.

Prepaid and promised indirect cash incentives – in the form of $20 automated teller machine

(ATM) cards – and priority mail were used as stimuli.

The prepaid incentives were the most effective at increasing response rates and decreasing data

collection costs. When delivered by priority mail, the ATM card incentive increased the mail

response rate by 13.8 percentage points over the control group. When delivered by first class

mail, a 10.7 percentage point increase in the mail response rate was achieved. The promised

incentives delivered by first class mail elicited a 7.1 percentage point increase over the control

group. All three of these response rate increases over the control group were statistically 

significant at the α=0.05 level.

At the end of the data collection period, which included face-to-face follow-up for all mail nonrespondents, all treatment groups provided with an incentive had higher overall response rates

than those groups that were not provided an incentive. Specifically, the prepaid incentive

delivered in first class mail achieved a 7.0 percentage point increase in the overall response rate;

the prepaid incentive delivered in priority mail achieved a 9.0 percentage point increase in the

overall response rate; and the promised incentive delivered in first class mail achieved a 5.0

percentage point increase in the overall response rate.

The mail response rate increases led to fewer costly face-to-face follow-up contacts being

required for all mail non-respondents per ARMS data collection protocol. In addition, only

about a third of all ATM card recipients cashed their ATM cards by the expiration date. The low

rate of cashing and the need for fewer face-to-face contacts for the prepaid incentive groups led

to a lower cost for these groups than the other treatment groups.

The combination of higher response rates and lower costs for the prepaid incentive groups

provides impetus for the use of incentives for future ARMS Phase III data collections,

particularly mail data collections.

ii

RECOMMENDATIONS

1. Indirect monetary incentives should be used on a large scale for the ARMS Phase III Core

mail-out/mail-back sample. In particular, prepaid ATM cards sent by first class or priority 

mail should be implemented on a regular basis for the Core sample.

2. Priority mail should be considered in the mail-out/mail-back methodology that is

implemented to deliver the questionnaire and other survey material to sampled operations.

When used in combination with a prepaid ATM incentive, it was an effective component of a

mail-out/mail-back data collection strategy. However, when used alone, priority mail was

not effective at increasing mail response rates.

3. Any future incentives use should be included as one piece of the larger mail data collection 

strategy. The mail-out/mail-back data collection methodology used for this research was a

comprehensive plan that incorporated many important components to ensure high response,

including an advance letter, and two mail-out packets with a reminder postcard in-between.

The number and quality of contacts is crucial to obtaining reasonable mail response rates for

mail-out/mail-back surveys.

4. Any incentives considered for other components of ARMS should be part of a research plan

designed to test incentive effectiveness. This experiment did not examine the use of

incentives with the face-to-face data collection methodology or for the other versions of the

ARMS questionnaires.

5. Prior to implementing the use of indirect monetary incentives into operational programs,

NASS should first conduct research to determine whether there are any negative wide

ranging effects of their use, especially in terms of response rates to programs not using 

incentives. This may include the creation and maintenance of new data items stored in the

NASS data warehouse.

6. Prior to any future implementation of the use of indirect monetary incentives into operational

programs, NASS should conduct research to investigate their effects on nonresponse bias. It

is not known what effects the use of indirect monetary incentives may have had or may have

in the future on nonresponse bias for ARMS Phase III.
i
1

Indirect Monetary Incentives for the 2004 ARMS Phase III Core

Daniel G. Beckler, Kathleen Ott, and Peter Horvath 1/

Abstract

The United States Department of Agriculture’s National Agricultural Statistics

Service conducts the annual Agricultural Resource Management Survey (ARMS)

Phase III. Part of this survey’s sample uses a sixteen-page core questionnaire with

mail-out/mail-back data collection and face-to-face nonresponse follow-up. Both

prepaid and promised indirect monetary incentives (i.e., $20 ATM cards) were used

on the 2004 ARMS in order to increase response rates and to reduce costly face-toface follow-up interviews.

The results showed that both the prepaid and promised incentives significantly

increased the mail and overall response rates, with the prepaid incentives

performing somewhat better. The incentives also succeeded in reducing the overall

data collection costs

Key Words: Nonresponse; response rate; refusal conversion.

1. INTRODUCTION

The Agricultural Resource Management

Survey (ARMS) is an annual survey 

conducted by the United States Department of

Agriculture’s (USDA) National Agricultural

Statistics Service (NASS) and co-sponsored 

by USDA’s Economic Research Service

(ERS). The ARMS consists of three phases.

Phase I is used for screening potential

operations for Phases II and III; Phase II

collects data on cropping practices and

agricultural chemical usage; and Phase III

collects detailed economic information about

the agricultural operation as well as the

operator’s household.

ARMS Phase III has been problematic

because, compared with other NASS surveys,

its response rates are low, and its data

collection costs are high. Prior to 2003, all

ARMS Phase III data were collected by faceto-face enumeration because of the length and

complexity of the questionnaire.

A self-administered Phase III “Core”

questionnaire developed for the 2003 survey

was used for part of the sample for mailout/mail-back data collection; this sixteen

page questionnaire collected the most critical

ARMS Phase III information with half the

number of pages as in the full questionnaire.

Face-to-face nonresponse follow-up

interviews were conducted on all mail nonrespondents. In addition, part of the 2003 

Phase III sample was field enumerated using 

1/ Daniel G. Beckler and Kathleen Ott are mathematical statisticians with the USDA’s National Agricultural

Statistics Service - Research and Development Division, located at Room 305, 3251 Old Lee Highway,

Fairfax, VA 22030. Peter Horvath is a former mathematical statistician with the USDA’s National

Agricultural Statistics Service. The authors extend a special thanks to Brad Summa, Dania Ferguson, and Jim

Vogelsang for making this research possible. The authors also acknowledge Doctors Don Dillman and Danna

Moore of Washington State University for their assistance developing the research plan for this project.
2

the Core form. These cases had not

previously received the questionnaire in the

mail. Analysis by ERS and NASS determined

that the data collected through the mail were

comparable to those collected by face-to-face

methodology 

Since the utility of a self-administered ARMS

Phase III Core form was demonstrated in 

2003, NASS has sought ways to implement its

use and increase the overall response rate as

well as contain data collection costs. Offering 

potential respondents incentives is a proven

technique to increase response rates on a

variety of surveys conducted by several

agencies and companies (Church, 1993; James

and Bolstein, 1992; James and Bolstein, 1990;

Singer, 2002).

An incentive experiment was implemented

with the 2004 ARMS Phase III Core to test

whether monetary incentives would be

effective at increasing response rates for this

survey while being cost effective.

2. METHODS

The 2004 ARMS Phase III Core sample size

was 15,900 farm/ranch operations across 15

states. The 15 states were those with the

highest agricultural value of sales and

included Arkansas, California, Florida,

Georgia, Illinois, Indiana, Iowa, Kansas,

Minnesota, Missouri, Nebraska, North 

Carolina, Texas, Washington, and Wisconsin.

The total sample was stratified by state,

ARMS farm value of sales (as maintained on 

the sampling frame), and type of operation

(also as maintained on the sampling frame);

then, five sub-samples, each of size 2,000,

were systematically selected. The subsamples were drawn such that each was

equally represented by the strata. Once the

sub-samples were drawn, NASS Field Offices

had the opportunity to remove operations from

the Core sample that had previous data

reporting arrangements with the office. This

resulted in each sub-sample being slightly less

than 2,000.

The standard mail-out/mail-back data

collection methodology for the 2004 ARMS

Phase III Core sample consisted of: (1) a

cover letter and questionnaire mailed on

December 28, 2004, (2) a post-card reminder

and “thank you” sent to the entire sample on 

January 13, 2005, (3) a cover letter and second 

questionnaire mailed to all non-respondents on 

January 31, 2005, and, finally, (4) starting

February 21, 2005, face-to-face interviews

attempted for all remaining mail nonrespondents.

In addition to the standard data collection

methodology, prepaid and promised indirect

cash incentives – in the form of $20 

automated teller machine (ATM) cards – and

priority mail were used as stimuli.

Combinations of these stimuli were

administered to four of the five sub-samples

mentioned above; a fifth sub-sample received 

no stimuli and served as the control for this

project. Collectively, these five sub-samples

formed the five treatment groups used for this

project. Table 1 contains descriptions of the

treatment groups.

All treatment groups received cover letters

that included some uses of the ARMS data.

Five such uses were included in the cover

letters used for treatment groups not receiving

incentives, and three uses were included in the

cover letters used for treatment groups that

received incentives. These unintentional

differences may have confounded our results,

as they may have contributed positively or

negatively to the response rates. However, we

expect the confounding effect to be minimal

since all letters were written to positively

impact response.
3

The sub-samples in the pre-paid incentive

treatment groups all received cover letters

that: (1) explained the incentive was a “thank

you”, (2) described the uniqueness of the

ARMS, and (3) justified the use of the

incentive by its overall cost savings to the

government.

The actual ATM card incentive was delivered

to recipients in the same packet as the first

questionnaire and was affixed to a standard

81⁄2 inch x 11 inch sheet of paper that

reiterated a “thank you” and included

instructions on how to use the card. Appendix

A contains the treatment group survey

materials and includes copies of the cover

letters used for each treatment group, a copy

of the ATM card instruction sheet, and a photo 

of the actual ATM card that was used.

The $20 ATM cards were supplied by

JPMorgan Chase bank and were usable in

nationwide ATM machines that displayed the

NYCE®, Pulse

®, Maestro®, or Cirrus® logos.

The cards were also usable at point-of-sale

(POS) (i.e., retail) establishments that allow

the use of debit cards as payment; however,

this fact was not revealed to card recipients.

In addition to the $20 incentive, the ATM

cards were loaded with an extra $4 to cover

any transaction charges. The cards were preactivated and were immediately usable when

the recipients received them. The personal

identification number (PIN) needed to use the

card was embossed on the front of each card 

after the words “THANK YOU”. The front of

each card also included the embossed

message, “FOR HELP 1-888-424-7828”; this

toll-free telephone number was answered by 

NASS staff. Finally, all ATM cards expired 

on June 30, 2005 (there was no provision for

extending this date). See Appendix A for a

photo of the actual ATM card. If a card 

recipient lost or could not use the card, a

replacement could be requested by calling the

toll free phone number listed on the

instruction sheet. For the few cases where this

occurred, only the replacement card was used

in our analysis.

The decision to use $20 ATM cards as

incentives was essentially made by default.

Actual cash was preferred by the authors;

however, NASS and USDA senior

management were concerned with

accountability when using cash. Checks were

also considered, but the U.S. Treasury 

Department (the would-be issuer of the

checks) was concerned with logistical issues

related to check usage. There were no such 

concerns from NASS, USDA, or the Treasury 

Department for the ATM cards. The decision 

to offer $20 as the incentive amount was also 

largely a result of that value being the only

viable option because many, if not most,

ATMs only dispense cash in $20 increments.

Priority mail was also used as a stimulus

because evidence from survey literature has

shown the use of priority mail increases

overall response rates, especially when used in

combination with monetary incentives (Moore

and An, 2001).
4

Table 1: ARMS Phase III Core Treatment Groups

Treatment

Group

First Questionnaire Mailing Post Card Second Questionnaire

Mailing

Face-to-Face

Follow-up

1

(Control)

First Class Mail

Cover letter

No incentive

First Class Mail

Cover letter

No incentive

2

First Class Mail

Cover letter

Prepaid $20 ATM card

First Class Mail

Cover letter

No incentive

3

Priority Mail

Cover letter

No incentive

Priority Mail

Cover letter

No incentive

4

Priority Mail

Cover letter

Prepaid $20 ATM Card

Priority Mail

Cover letter

No incentive

5 1/

First Class Mail

Cover letter

Promised $20 ATM Card Stan

dard

Post-Card

Remin

der/T

han

kYou

Used for All Treatment Grou

ps

First Class Mail

Cover letter

Promised $20 ATM Card Stan

dard

Face-to-Face Follow-U

p

Used

for All Treatment Grou

ps

1/ In several tables in this report, a distinction is made between Treatment Group 5 and Treatment Group 5'. Treatment

Group 5 includes all respondents in the treatment group who received an ATM card (whether their questionnaires were

received in the mail or by a face-to-face interview) and Treatment Group 5' includes only those respondents in the

treatment group who received an ATM card because they returned their questionnaires in the mail.

3. RESULTS

3.1 Response Rates

Similar to previous incentive research

conducted on various populations, the results

of this project showed that incentives for the

farm and ranch operator population increased

response rates. Table 2 shows the response

rates by treatment group (treatment group 

descriptions are given in Table 1). Response

rates are also broken out by the different

mailing and contacts employed as part of the

data collection methodology.

As shown in Table 2, all treatment groups that

received an incentive had higher response

rates than the control group (Treatment Group 

1). The prepaid $20 ATM incentive sent by

priority mail (Treatment Group 4) had the

highest mail and overall response rates, at 43.9 

percent and 72.4 percent, respectively. The

second highest mail and overall response rates

were achieved with the prepaid incentive sent

by first class mail (Treatment Group 2) with a

mail response rate of 40.8 percent and an

overall response rate of 70.4 percent. The

promised $20 ATM incentive response rates,

at 37.2 percent for mail and 68.4 percent

overall, surpassed the treatment groups that

received no incentives, but lagged those that

provided prepaid incentives. Priority mail

alone was ineffective at increasing response

rates.

The mail and overall response rates were

compared for statistical significance using ttests across all treatment group combinations.

All tests were done at an overall α=0.05 level

of significance (with a Bonferroni adjustment

for the ten comparisons). The details of the

statistical significance testing are described in

Appendix B.

All three treatment groups that received ATM

card incentives had significantly higher mail

and overall response rates than the control

group, with the two prepaid treatment groups

outperforming the promised incentive. The

specific differences are shown below and
5

detailed in Appendix B.

In terms of mail response, all three treatment

groups that received ATM card incentives had

significantly higher response than the control

group. Specifically:

• Treatment Group 2 had a significantly 

greater mail response rate than 

Treatment Groups 1 and 3.

• Treatment Group 4 had a significantly 

greater mail response rate than 

Treatment Groups 1, 3, and 5.

• Treatment Group 5 had a significantly 

greater mail response rate than 

Treatment Groups 1 and 3.

In terms of overall response, all three

treatment groups that received ATM card

incentives had significantly higher response

than the control group. Specifically:

• Treatment Group 2 had a significantly 

greater overall response rate than

Treatment Groups 1 and 3.

• Treatment Group 4 had a significantly 

greater overall response rate than

Treatment Groups 1 and 3.

• Treatment Group 5 had a significantly 

greater overall response rate than

Treatment Group 1.

Recall cover letter differences (regarding uses

of ARMS data) between the treatment groups

may have confounded these results.
6

Table 2: Response Rates

1/ by Treatment Group

1stMailing Returns Postcard 2nd Mailing Returns Overall Mail Returns 3/ Face-to-Face

Follow-Up Completes

Overall Returns &

Treatment Completes

Group

N 2/

Count Response

Rate Count Resp. RateIncrease Count Resp. RateIncrease Count ResponseRate Count ResponseRate Count ResponseRate

1 (Control) 1,948 137 7.0% 117 6.0% 225 11.6% 586 30.1% 649 33.3% 1,235 63.4%

2 1,941 223 11.5% 214 11.0% 240 12.4% 791 40.8% 575 29.6% 1,366 70.4%

3 1,935 146 7.6% 135 7.0% 241 12.5% 635 32.9% 616 31.8% 1,251 64.7%

4 1,952 197 10.1% 226 11.6% 307 15.7% 856 43.9% 557 28.5% 1,413 72.4%

5 1,946 193 9.9% 156 8.0% 276 14.2% 724 37.2% 608 31.2% 1,332 68.4%

1/ Response Rates were determined with item code 0910 on the questionnaire. The response rate = IC

IC

910 1 3 5

910 1 3 5 8 9

=

=

( , , )

( , , , , ) where IC0910 = 1 for mail completes, IC0910 = 3 for non-mail

completes, IC0910 = 5 for out of scope records, IC0910 = 8 for refusal records, and IC0910 = 9 for inaccessible records.

2/ Initially, all treatment groups contained 2,000 records, but field offices removed operations with whom they had previous data collection agreements.

3/ Overall Mail Returns includes all returns from the 1stand 2nd mailings and postcard in addition to all mail returns received after face-to-face follow-up started.
7

Figure 1: Mail Response Rates by Treatment Group

Figure 2: Overall Response Rates by Treatment Group
8

3.1.1. Response Over Time

Rapidity of response is an important element

in the data collection process. If an early mail

response is received, further follow-up 

contacts are unnecessary. Response rates over

time can be seen for each treatment group in 

Figures 1 and 2. Figure 1 includes only mail

response, while Figure 2 tracks overall

response for each treatment group.

As shown in both graphs, the treatment groups

provided with incentives (i.e., Treatment

Groups 2, 4, and 5) out-performed the control

group (Treatment Group 1) with respect to 

both mail and overall response rates

throughout the data collection process.

Obtaining these responses earlier makes the

entire data collection process more efficient

and cheaper by allowing processing to start

earlier and by reducing the number of costly 

face-to-face follow-up contacts.

3.1.2. Response By State

Although this experiment was not designed to 

measure state-level effects on response rates,

it is interesting to notice the different trends

across states. In most states, those operators

in the treatment groups that received ATM

card incentives tended to have the highest

response rates. However, there were some

interesting exceptions such as in Texas where

the treatment group with the highest rate of

response was the Control Group! Figure 3

illustrates state-level response rates by

treatment group. Appendix C contains more

detailed information on state-level response

rates.

Figure 3: Overall Response Rates by State and Treatment Group

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Arkansas

California

Florida

Georgia

Illinois

Indiana

Iowa

Kansas

Minnesota

Missouri

Nebraska

North

Carolina

Texas

Washington

Wisconsin

O

verall Resp

o

nse R

ate

Treatment Group 1 Treatment Group 2 Treatment Group 3

Treatment Group 4 Treatment Group 5
9

3.1.3. Response by Farm Type and Size

Also of interest is the incentive effect on

different farm types and sizes; however, the

sample was not selected in such a way as to

allow statistical testing by these attributes.

Some groups have very few records, making 

the data for those groups unreliable. These

data are shown for information only and 

cannot be used to make any general inferences

about the response patterns for any specific

group or the impact of incentive use on its

response pattern. Figure 4 shows the

breakdown of response rates by farm type and 

Figure 5 shows the value of sales categories.

See Appendices D and E for more details on 

the response rates by farm type and size.

3.1.4. A Word on Nonresponse Bias

This study did not address the potential effect

on nonresponse bias of offering ATM card

incentives for the ARMS Phase III Core.

Nonresponse bias arises when survey

nonrespondents are systematically different

from respondents with respect to the data

collected on the survey. Because ATM card

incentives significantly increased response

rates, it is possible that they also changed the

operator/operation characteristics composition 

of remaining nonrespondents. If the make-up

of the nonrespondent pool changed, incentive

use may have either reduced or exacerbated

nonresponse bias, dependent upon whether

incentive use made nonrespondents less alike

or more alike. Although Figures 3, 4, and 5 

provide some insight into how the incentives

affected response from sub-populations of the

ARMS Phase III Core sample, the sample

sizes for this study’s treatment groups were

insufficient to properly assess nonresponse

bias.

Figure 4: Overall Response Rates by Farm Type and Treatment Group

0

10

20

30

40

50

60

70

80

90

100

Grains, Oilseeds, Etc.

Tobacco

Cotton

Vegies, Melons

Fruits, Nuts, Berries

Nursery, Greenhouse

Christmas

Trees

Other

Crops

Hogs

Milk, Dairy

Cattle

Sheep, Goats

Equine

Poultry, Eggs

Aquaculture

Other

Animals

Other

O

verall Resp

o

nse R

ate

Treatment Group 1 Treatment Group 2 Treatment Group 3 Treatment Group 4 Treatment Group 5
10

Figure 5: Overall Response Rates by Value of Sales and Treatment Group

0

10

20

30

40

50

60

70

80

90

Less Than

$20,000

$20,000 - $39,999 $40,000 - $99,999 $100,000 -

$249,999

$250,000 -

$499,999

$500,000 Or

More

O

verall Resp

o

nse R

ate

Treatment Group 1 Treatment Group 2 Treatment Group 3 Treatment Group 4 Treatment Group 5

3.2. ATM Card Use

Prior to survey year 2004, indirect monetary 

incentives were not used at NASS. Therefore,

it was unclear what to expect when mailing

almost 6,000 ATM cards to potential survey

respondents. Other government agencies have

used ATM cards on a large scale and found

that a sizable percentage of card recipients do

not cash their cards (Kay, et al., 2001).

3.2.1. ATM Card Use: How many 

recipients cash them?

Using a $20 ATM card incentive to attempt to

boost response could be costly if everyone

were to cash his/her card. The cover letter

emphasized that the ATM card was provided

with the purpose of passing on some of the

savings of mail data collection over a face-toface interview. As it turned out for the ARMS

Core, not nearly all recipients of ATM cards

used them. Table 4 shows the percentages of

recipients who used their ATM cards for

monetary withdrawal by treatment group.

Just over a third (38.6 percent) of all card 

recipients cashed their ATM cards. This rate

differed greatly by incentive treatment group

and response. The most striking difference is

that between respondents and nonrespondents. Respondents cashed their cards

at an average rate of 47.6 percent, while nonrespondents cashed them less than five percent

of the time. It could be that non-respondents

never saw the card because they threw the

packet in the trash without opening it. Also,

non-respondents may have felt that they did

not deserve the money because they did not

complete the questionnaire.

Also noteworthy is that mail respondents

cashed their cards at a higher rate than face-toface interviewed respondents (this can be

concluded from Table 4). The cover letter

emphasized that the ATM card was provided

with the purpose of passing on some of the
11

Table 4: ATM Card Usage by Treatment Group

1/

Card Use Among Mail Respondents Card Use Among Overall Respondents 2/ Card Use By Overall NonRespondents Card Use By All Card Recipients 3/

Treatment

Group 5/ Number of

Mail

Respondents

Cards Used

For $

Withdrawal

Percent of

Cards

Cashed

Number of

Overall

Respondents

Cards Used

For $

Withdrawal

Percent

of Cards

Cashed

Number of

NonRespondents

Cards Used

For $

Withdrawal

Percent

of Cards

Cashed

Number of

Card

Recipients

Cards Used

For $

Withdrawal

Percent of

Cards

Cashed

2 791 478 60.4 1,366 561 41.1 575 27 4.7 1,941 588 30.3

4 856 504 58.9 1,413 576 40.8 539 29 5.4 1,952 605 31.0

5 724 517 71.4 1,332 818 61.4 614 19 4/ 3.1 4/ 1,365 837 61.3

5' 724 517 71.4 1,332 818 61.4 614 0 0.0 724 484 66.9

Total (Trts.

2,4,5) 2,371 1,499 63.2 4,111 1,955 47.6 1,728 75 4.3 5,258 2,030 38.6

1/ Numbers include distinct operators who withdrew money using ATM cards.

2/ Includes mail and face-to-face respondents.

3/ Includes respondents and non-respondents.

4/ Operators in Treatment Group 5 should not have received an ATM card unless they responded to the survey. However, some returned partially completed forms that were determined to be

inadequately completed after the card had been mailed out.

5/ Treatment Group 5 includes all respondents in the treatment group who received an ATM card (whether their questionnaires were received in the mail or by a face-to-face interview) and

Treatment Group 5' includes only those respondents in the treatment group who received an ATM card because they returned their questionnaires in the mail.
12

savings of mail data collection, instead of

requiring a face-to-face interview. Assuming 

they read (and remembered) the cover letter,

those respondents who completed the survey

with a face-to-face interview may have felt

that they were not entitled to the money or that

the card was not valid unless the questionnaire

was mailed back.

Respondents in the two prepaid incentive

treatment groups (Treatment Groups 2 and 4)

cashed their cards at a rate of about 41 percent

while respondents in the promised treatment

group (Treatment Group 5) cashed their cards

at a rate of over 60 percent. This could be

because the respondents who were promised

the incentive felt as if they deserved the card

because they fulfilled their side of an

economic agreement. By making receipt of

the card contingent on filling out the

questionnaire, NASS may have made these

respondents more likely to feel that they had

earned the money. The prepaid respondents,

on the other hand, may have seen the ATM

card as a gesture of goodwill, and not felt as if

they necessarily deserved it.

The low card cashing rate directly impacts the

cost of the incentive experiment, which will

be discussed later in this report.

3.2.2. ATM Card Use: When did farm

operators cash them?

Also of interest is the relationship between the

date the card was used and the date the

questionnaire was returned. Table 5 depicts

ATM card usage with respect to the date the

questionnaire was received at the National

Processing Center (NPC). Figure 6 shows a

frequency distribution of the time difference

between when each respondent returned his

questionnaire and when he used his ATM

card. The date each respondent returned his

questionnaire is referred to as the “NPC Date”

in Figure 6.

Table 5 and Figure 6 show that about one-half

of the people in the prepaid treatment groups

(Treatment Groups 2 and 4) cashed their cards

within a period of two weeks before or after

they returned their questionnaires. These

respondents perhaps linked cashing the card to

the task of filling out the questionnaire.

Another one-third of respondents cashed their

ATM cards several weeks after completing the

questionnaire. These people may have cashed

them at this point because the expiration date

(which was June 30, 2005) was approaching.

Table 5: When ATM Cards Were Used by Respondents

(measured in days/weeks with respect to receipt of the questionnaire at NPC)

3 weeks

or more

before

receipt

2-3 weeks

before

receipt

1-2 weeks

before

receipt

4-6 days

before

receipt

Roughly

the same

time

4-6 days

after

receipt

1-2 weeks

after

receipt

2-3 weeks

after

receipt

3 weeks or

more after

receipt

Treatment

Group

Frequency

(Percent of Treatment Group)

2 33

(5.9%)

15

(2.7%)

16

(2.9%)

63

(11.2%)

125

(22.3%)

34

(6.1%)

62

(11.1%)

34

(6.1%)

179

(31.9%)

4 36

(6.3%)

19

(3.3%)

31

(5.4%)

48

(8.3%)

119

(20.7%)

28

(4.9%)

51

(8.9%)

32

(5.6%)

212

(36.8%)

5 3 1/

(0.4%)

0

(0.0%)

0

(0.0%)

0

(0.0%)

3

(0.4%)

5

(0.6%)

72

(8.8%)

180

(22.0%)

555

(67.9%)

1/ Since respondents in Treatment Group 5 did not receive their ATM cards until after they returned their ARMS III Core questionnaire, this number

should be zero. These people likely returned an incomplete questionnaire to NPC at which point NPC sent out the promised ATM incentives. Later, a

face-to-face follow-up interview was also needed to complete the interview– the later dates were used as the NPC Dates.
13

Figure 6: Timing of ATM Card Usage Relative to the NPC Date 1/

3.2.3. ATM Cards: How much do they

cost?

An important issue with offering ATM card

incentives is the various costs associated with

using them. Table 6 breaks down overall

ATM card-related costs for the treatment

groups. Card use for all operators in each

treatment group (respondents or nonrespondents) is included here. Overall costs,

including printing, mailing, and interviewing 

costs for the incentive experiment are

presented later in Section 3.3.

There are several transaction fees associated

with ATM card use besides the actual

withdrawal amount. This is why each card

was loaded with $24, instead of just the $20 

incentive amount. These fees include

withdrawal and purchase fees, balance

inquiries fees, and failure fees. In addition,

since the ATM cards could be used at pointof-sale (POS) debit card machines, users could 

withdraw money at a variety of locations (this

fact was not explicitly stated to card

recipients; hence the low numbers of such 

fees). These fees are presented separately in

Table 6.
14

Table 6: ATM Card Charges by Treatment Group

1/

ATM Withdrawal 2/ ATM Withdrawal

Fee 3/ POS Purchase 2/ POS Purchase Fee 3/ Balance InquiryFee

Transaction

Failure Fee Treatment

Group 4/

Count Cost Count Cost Count Cost Count Cost Count Cost Count Cost

Total

Cost

2 578 $12,386.85 574 $579.00 11 $201.62 11 $3.85 23 $10.35 38 $13.10 $13,194.77

4 600 $12,953.50 595 $599.00 5 $105.50 5 $1.75 24 $11.70 34 $11.90 $13,683.35

5 822 $17,695.81 814 $818.00 16 $277.60 16 $6.30 31 $16.20 44 $17.80 $18,831.71

5' 512 $10,999.41 507 $509.00 7 $118.38 7 $2.80 16 $8.10 25 $10.80 $11,648.49

1/ Numbers include distinct operators who withdrew (or tried to withdraw) money using ATM/Debit cards.

2/ Includes amount of withdrawal/purchase as well as any transaction fees imposed by the ATM owner/retailer.

3/ JPMorgan Chase transaction fee. JPMorgan Chase was the issuer of the ATM cards.

4/ Treatment Group 5 includes all respondents in the treatment group who received an ATM card (whether their questionnaires were received in the mail or by a faceto-face interview) and Treatment Group 5' includes only those respondents in the treatment group who received an ATM card because they returned their

questionnaires in the mail.

3.2.4. ATM Card Use: Who used them? 

The following tables summarize ATM card

use among ARMS Phase III Core respondents

by various demographics and other descriptive

variables. Tables 7, 13, and 15 use sampling

frame control data, while the other tables use

reported ARMS data. One particularly 

interesting thing in Tables 7-15 is that

operators who tended to leave questions

unanswered also tended to leave their ATM

cards unused.

Table 7: ATM Card Usage by Value of Sales (Control Data)

Less Than

$10,000

$10,000 -

$49,999

$50,000 -

$149,999

$150,000 -

$499,999

$500,000 Or

More Card Use Frequency

(Percent Within Sales Category)

288 297 305 950 314 No (48.6%) (51.9%) (54.0%) (53.4%) (52.3%)

305 275 260 828 287 Yes (51.4%) (48.1%) (46.0%) (46.6%) (47.8%)

Table 8: What year did the operator become involved in the operation?

Unreported Pre1950

1950-

1959

1960-

1969

1970-

1979

1980-

1989

1990-

1999

PostCard 1999

Use Frequency

(Percent Within Year Category)

375 45 120 249 421 413 437 94 No (66.4%) (56.3%) (60.6%) (52.2%) (47.4%) (49.2%) (50.3%) (49.0%)

190 35 78 228 467 427 432 98 Yes (33.6%) (43.8%) (39.4%) (47.8%) (52.6%) (50.8%) (49.7%) (51.0%)
15

Table 9: Operator Age

Unreported 30 or

Younger

30-39 40-49 50-59 60-69 70 and

Older Card Use Frequency

(Percent Within Age Category)

356 42 145 397 512 435 267 No (67.4%) (60.9%) (49.2%) (49.5%) (47.3%) (51.7%) (54.5%)

172 27 150 405 571 407 223 Yes (32.6%) (39.1%) (50.9%) (50.5%) (52.7%) (48.3%) (45.5%)

Table 10: Does the Operator’s Spouse Make Day-to-Day Decisions for this Farm/Fanch?

Unreported Yes No No Spouse

Card Use Frequency

(Percent Within Decision Category)

288 536 1,000 330 No (67.8%) (46.3%) (51.9%) (55.0%)

137 622 926 270 Yes (32.2%) (53.7%) (48.1%) (45.0%)

Table 11: Formal Education

Unreported Less Than

High School

High School

Diploma

Some College

Completed Four

Year Degree

Graduate

School Card Use Frequency

(Percent Within Education Category)

304 139 717 514 343 137 No (66.8%) (55.8%) (53.7%) (49.6%) (47.5%) (44.2%)

151 110 619 523 379 173 Yes (33.2%) (44.2%) (46.3%) (50.4%) (52.5%) (55.8%)

Table 12: Gender

Unreported Male Female

Card Use Frequency

(Percent Within Gender Category)

304 1,719 131 No (67.7%) (50.4%) (52.6%)

145 1,692 118 Yes (32.3%) (49.6%) (47.4%)

Table 13: Land Size (Control Data)

Unknown 0-99 Acres 100-499 Acres 500-999 Acres 1,000 + Acres

Card Use Frequency

(Percent Within Land Size Category)

13 745 670 280 446 No (54.2%) (52.1%) (53.8%) (50.5%) (52.3%)

11 686 576 275 407 Yes (45.8%) (47.9%) (46.2%) (49.6%) (47.7%)
16

Table 14: Race or Ethnicity

Unreported White Hispanic Asian Am. Indian Black Pacific

Islander Card Use Frequency

(Percent Within Racial/Ethnic Category)

299 1,777 28 28 8 13 1 No (67.3%) (50.4%) (65.1%) (75.7%) (22.2%) (65.0%) (50.0%)

145 1,750 15 9 28 7 1 Yes (32.7%) (49.6%) (34.9%) (24.3%) (77.8%) (35.0%) (50.0%)

Table 15: Farm Type (Control Data)

Grains/

Oilseeds/

Etc.

Tobacco Cotton

Veggies/

Melons

Fruits/Nuts/

Berries

Nursery/

Greenhouse

Christmas

Trees

Other Crops

Card Use

Frequency

(Percent Within Farm Type)

432 32 36 61 272 219 4 193 No (47.4%) (72.7%) (64.3%) (56.0%) (59.8%) (50.2%) (57.1%) (48.0%)

479 12 20 48 183 217 3 209 Yes (52.6%) (27.3%) (35.7%) (44.0%) (40.2%) (49.8%) (42.9%) (52.0%)

(Table 15 continued)

Hogs Milk/Dairy Cattle

Sheep/

Goats

Equine

Poultry/

Eggs

Aquaculture

Other

Animals

Other

Card Use Frequency

(Percent Within Farm Type)

1 172 454 19 41 191 10 10 7 No (100%) (51.7%) (54.2%) (59.4%) (49.4%) (53.7%) (55.6%) (55.6%) (63.6%)

0 161 383 13 42 165 8 8 4 Yes (0%) (48.4%) (45.8%) (40.6%) (50.6%) (46.4%) (44.4%) (44.4%) (36.4%)

3.3. Overall Costs

Cost is a major issue to be considered if NASS

is to continue using incentives for the ARMS

III Core. Data collection costs for mail and 

face-to-face interviewing need to be combined 

with incentive costs to create an overall

picture of the dollar value of using incentives.

Table 16 displays the total cost for the

approximately 2,000 records associated with

each treatment group. These dollar values are

not meant to be comparable to 2004 

Agricultural Resource Management Survey

(ARMS) Phase III Survey Administration

Analysis (Hopper, forthcoming) average per

sample or per cost values (though 2004 

Agricultural Resource Management Survey

(ARMS) Phase III Survey Administration

Analysis cost information was used in this

table for purposes of calculating total

enumeration costs). Full details on the cost

calculations for each column of the table can

be found in Appendix F.

As shown in Table 16, overall costs and the

average costs per sample and completed

record are lowest for the two prepaid incentive

groups (Treatment Groups 2 and 4). This is

due to two important factors in the data
17

collection methodology and incentive use.

First, only a little over one-third of card 

recipients cashed their ATM card. This saved

NASS several thousand dollars in ATM card

withdrawals and fees. Second, face-to-face

interviews are much more costly than selfadministered (mail) questionnaires with an

incentive. Because the prepaid incentive

treatment groups achieved such large

increases in mail response over the other

treatment groups, overall data collection costs

for those two groups were lower.

Table 17 contains cost data for a “mail-only”

data collection scenario. Note that since these

data came from a methodology that uses faceto-face follow-up, the costs may not truly 

represent a mail-only scenario.

Table 17 also shows the cost of each

additional mail return obtained from each 

treatment group over the control group 

(Treatment Group 1). Relative to the cost of

each mail-returned questionnaire in the control

group, the cost per additional mail return for

each treatment group is quite high. Indeed,

assuming all options are possible, it may be

more economical to use a different data

collection methodology instead of “mail-only

with incentive”. This suggests that the use of

indirect monetary incentives, as implemented 

in this project, may not be economical for a

hypothetical mail-only data collection 

methodology. However, for ARMS Phase III,

since even the highest mail response rate

obtained from any of the treatment groups was

only 43.9 percent, it is likely that a mail-only 

approach will never be acceptable to NASS,

ERS, or the Office of Management and

Budget (OMB).
18

Table 16: Overall Costs by Treatment Group

Costs (In Dollars)

Treatment

Group 1/ N Postage NPC

Printing

NPC

Extra

Admin

ATM Card

Charges

ATM

Card

Admin

Face-to-Face

Follow-up

Costs

Total

Average

Per

Sample

Average Per

Complete

1 (Control) 1,948 4,882 6,638 --NA-- --NA-- --NA-- 176,291 187,811 96.41 152.07

2 1,941 4,833 6,295 2,000 13,195 1,650 154,154 182,127 93.83 133.33

3 1,935 15,207 6,515 --NA-- --NA-- --NA-- 170,951 192,673 99.57 154.02

4 1,952 14,953 6,297 2,000 13,683 1,659 148,356 186,948 95.77 132.31

5 1,946 5,284 6,655 2,000 18,832 1,654 160,475 194,900 100.15 146.32

5' 1,946 5,284 6,655 2,000 11,648 1,654 160,475 187,717 96.46 140.93

1/ Treatment Group 5 includes all respondents in the treatment group who received an ATM card (whether their questionnaires were received in the mail or by a faceto-face interview) and Treatment Group 5' includes only those respondents in the treatment group who received an ATM card because they retuned their questionnaires

in the mail.

Table 17: Mail Returns Cost by Treatment Group for Hypothetical Mail-Only Methodology

Costs (In Dollars)

Treatment

Group

Mail

Returns Postage NPC

Printing

NPC Extra

Admin

ATM Card

Charges

ATM Card

Admin Total

Difference

From

Control

Average

Per

Sample

Average Per

Mail Return

Difference

in Mail

Returns

from

Control

Cost per

Additional

Mail

Return

1 (Control) 586 4,882 6,638 --NA-- --NA-- --NA-- 11,520 --NA-- 5.91 19.66 --NA-- --NA--

2 791 4,833 6,295 2,000 13,195 1,650 27,973 16,453 14.41 35.36 205 $80.29

3 635 15,207 6,515 --NA-- --NA-- --NA-- 21,722 10,202 11.23 34.21 49 $208.20

4 856 14,953 6,297 2,000 13,683 1,659 38,592 27,072 19.77 45.08 270 $100.27

5' 1/ 724 5,284 6,655 2,000 11,648 1,654 27,241 15,721 14.00 37.63 138 $113.92

1/ Includes only respondents in Treatment Group 5 who returned a questionnaire in the mail.
19

3.4. Data Quality

Incentives for the ARMS Phase III Core

increased response and lowered cost. Another

aspect of the incentive experiment was to

determine if the use of incentives affected the

quality of data provided by respondents.

It is difficult to assess data quality in a selfadministered survey. If someone reports that

he/she has 200 acres of corn, we have no way

to probe for more information or verify the

accuracy of this answer. If our control data or

other survey data in the questionnaire indicate

something looks suspicious in the selfreported response, then we may impute a

“more accurate” value. However, the only

way to really know the “true” value of, say,

corn acreage is to go out to the farm and make

the measurement ourselves. Obviously such a

quality check is not practical for NASS

surveys.

This makes it very difficult to make the claim

that farmer A filled out a survey form more

accurately than farmer B. There is no easy 

way to definitively determine whether one

group of farmers tended to fill out their

ARMS Phase III Core forms more accurately 

than another group of farmers. However,

there are some ways of looking at our data that

could possibly shed light on the issue of data

quality.

Three data quality tests were performed to 

evaluate the data obtained from respondents in

the different treatment groups. The tests

included: (1) comparing data quantity (i.e., the

number of questions answered), (2) comparing 

keyed data to edited data, and (3) comparing

reported data with NASS control data. For

tests (1) and (3), only keyed data were used.

Keyed data are those as they were recorded by

the respondent and then keyed at NPC.

Minimal editing of any sort was done to the

data before keying. For test (2), both keyed 

and edited data were used.

3.4.1. Data Quantity

Because the ARMS III Core questionnaire is

sixteen pages long, fatigue may result in a

respondent skipping questions, particularly at

the end of the questionnaire. With this in

mind, one way to assess data quality is to take

a closer look at data quantity, that is, tallying

the number of questions answered by each

respondent in each of the 11 sections of the

ARMS Phase III Core questionnaire. While

related, our notion of data quantity is not the

same as item nonresponse since not all

questions apply to all respondents; hence, the

specific sets of questions respondents are

expected to answer differ greatly. Table 18 

compares average data quantity among the

treatment groups.

As shown in Table 18, those respondents

provided with prepaid incentives (Treatment

Groups 2 and 4) answered slightly more

questions than those in the other treatment

groups. It is possible that respondents who

received the prepaid incentive felt that the

questionnaire was more important than those

who did not receive the incentive.

Consequently, they may have been more

diligent in completing the questionnaire,

which resulted in them providing answers to 

more questions. However, even those

differences that are statistically significant

(α=0.05) are small and probably not practical

when looking at the questionnaire as a whole.
20

Table 18: Mean Number of Questions Answered By Each Respondent 1/

Treatment ARMS Phase III Core Questionnaire Section

Group A B C D E F G H I J K

Total

Answers

1 (Control) 3.77 3.35 1.29 0.44 5.29 14.82 0.95 6.22 7.42 2.11 29.53 75.19

2 3.86 3.53 1.33 0.47 5.32 14.95 0.95 6.52 7.97* 2.28 29.67 76.86

3 3.84 3.21 1.21 0.44 4.93 14.39 0.94 6.04 7.25 2.19 29.37 73.83

4 3.96 3.70 1.32 0.47 5.52 15.45* 0.97* 6.64 8.21* 2.55* 30.34 79.14*

5 3.81 3.40 1.26 0.47 5.06 14.41 0.93 6.36 7.66 2.10 29.27 74.72

1/ To be considered “answered” a value greater than zero had to be keyed in. The sections are described below along with the number of questions in each:

Section A – Land in Farm/Ranch [11 questions]

Section B – Acreage and Production [54 questions]

Section C – Livestock [13 questions]

Section D – Rent Paid [3 questions]

Section E – Farm Income [80 questions]

Section F – Operating and Capital Expenditures [99 questions]

Section G – Farm Type [1 question]

Section H – Farm Labor [16 questions]

Section I – Farm Assets [25 questions]

Section J – Farm Debt [11 questions]

Section K – Farm Operator and Household [54 questions]

* Indicates that this average is significantly greater than at least one other treatment group’s average in this section of the survey. The treatment groups with

statistically significant differences (based on a two-side t-test at the α=0.05 level, computed with a Bonferroni adjustment to correct the error term for multiple

pairwise comparisons): Section F: the average number of questions answered was significantly different between treatment groups (4 and 3) and also between

(4 and 5); Section G: (4 and 5); Section I: (4 and 1), (4 and 3), and (2 and 3); Section J: (4 and 5); Total: (4 and 3), and (4 and 5)

3.4.2. Keyed Data versus Edited Data

Data quality may also be assessed by looking

at the amount of post-data collection edits that

were required to correct inconsistent or

apparent misreported data. Specifically, we

wish to determine whether the amount of

editing required for data provided by incentive

recipients was different from non-recipients.

More editing may be an indication of

satisficing (i.e., the respondent provides a

superficial response that appears reasonable or

acceptable, without going through all the

cognitive steps involved in the questionanswering process). In addition to the quality 

issues raised by data requiring significant

editing, data editing itself is labor intensive.

Therefore, if data quality were improved,

Field Offices would benefit significantly in 

terms of staff time.

Table 19 shows the mean number of answers

that were changed in each section from the

keyed data to the final edited data. Higher

numbers indicate that there were more

answers edited for that treatment group.

These numbers do not account for multiple

changes to a single question.

Although the number of changes is small

across all sections, the data provided by

respondents in the prepaid treatment groups

(Treatment Groups 2 and 4) required the most

editing overall, particularly in Section K

(Farm Operator and Household

Characteristics). It is unclear why this would 

be the case unless more editing resulted from

the slightly larger number of responses

provided in this section by those in the prepaid 

treatment groups. The difference in the

number of edits required is small (one

question at the most on average), but this is

the one element of analysis that may be

negative for the use of incentives. All other

measures showed that incentives improved or

did not impact any component of the survey

being analyzed.
21

Table 19: Mean Number of Answers Changed From Keyed Data to Final Edited Data

1/

Treatment ARMS Phase III Core Questionnaire Section

Group A B C D E F G H I J K

All

Changes

1 (Control) 0.28 0.35 0.14 0.05 1.30 1.21 0.03 0.71 1.55 0.29 5.55 11.46

2 0.30 0.40 0.17 0.07 1.27 1.48 0.04 0.70 1.59 0.31 6.48* 12.81*

3 0.29 0.35 0.15 0.05 1.15 1.36 0.04 0.59 1.65 0.28 5.47 11.38

4 0.31 0.38 0.16 0.06 1.33 1.37 0.03 0.64 1.53 0.31 6.43* 12.55

5 0.23 0.39 0.15 0.05 1.20 1.52 0.05* 0.67 1.62 0.28 6.05 12.21

1/ Figures reflect the mean number of times answers were changed from keyed data to final data for each section:

Section A – Land in Farm/Ranch [11 questions]

Section B – Acreage and Production [54 questions]

Section C – Livestock [13 questions]

Section D – Rent Paid [3 questions]

Section E – Farm Income [80 questions]

Section F – Operating and Capital Expenditures [99 questions]

Section G – Farm Type [1 question]

Section H – Farm Labor [16 questions]

Section I – Farm Assets [25 questions]

Section J – Farm Debt [11 questions]

Section K – Farm Operator and Household [54 questions]

* Indicates that this average is significantly greater than at least one other treatment group’s average in this section of the survey. The treatment groups with

statistically significant differences (based on a two-side t-test at the α=0.05 level, computed with a Bonferroni adjustment to correct the error term for multiple

pairwise comparisons): Section G: the mean number of answers that were changed from the keyed data to final edited data between treatment groups (4 and

5); Section K: (1 and 2), (1 and 4), (2 and 3), and (3 and 4); All Changes: (1 and 2) and (2 and 3)

3.4.3. Reported Data versus NASS

Control Data

Another way to assess data quality in the

ARMS Phase III Core is to look at the

difference between what operators report and 

what we expect them to report (i.e., sampling 

frame control data). Looking at this

difference provides us with some insight into

self-reporting consistency (of course, it also

may tell us something about control data

accuracy). However, ultimately it is really

just a comparison between two possible

indications of truth. We would expect that, on

average, the respondents from each treatment

group should have similar discrepancies

between what they report and their control

data. Significant differences in these

discrepancies among the different treatment

groups could indicate that certain treatment

groups may be supplying NASS with “better”

or “worse” data.

For the purpose of making this comparison

between self-reported data (as keyed at NPC)

and control data, we focus upon one variable –

total land operated (control data code 100 and 

item code 26 on the ARMS Phase III Core

questionnaire). Due to the way variables for

ARMS are calculated and the small number of

items collected on the survey that have

identical sampling frame control variables,

this variable was the only one that could be

tested. Because the distribution of total land

operated contains many outliers and near-zero

values, a nonparametric rank F-test was used.

Appendix G contains more details of the test

procedure.

Applying this test produced an F-test statistic

of 0.96 with a p-value of 0.4259; therefore, at

the α=0.05 significance level, there was not

sufficient evidence to reject the claim that the

mean ranking is the same for each treatment

group. That is, no significant difference was

evident across treatment groups in how

respondents reported their total land operated,

relative to their control data. Table 20 

contains the data associated with the test.
22

Table 20: Median Differences (Control vs. Reported) in Total Land Operated

Total Land

Treatment Group Median

Absolute Difference

Median Percent

Absolute Difference

1 (Control) 15 acres 7.14 %

2 18 acres 7.74 %

3 15 acres 7.29 %

4 20 acres 8.86 %

5 18 acres 8.38 %

F statistic = 0.96, p-value = 0.4259

3.5. Data Quality Summary

The greatest benefit NASS derives from using 

incentives is the economical increase in

response rates for the ARMS Phase III Core.

One natural concern that goes along with this

is that while these incentives may buy NASS

better response, they will not necessarily buy 

NASS better data. It is easy to argue that an 

incentive will get more data for a particular

survey (in the form of more responses), but

what if those data are also more likely to be

inaccurate?

Taking note of the limitations in our analysis,

the data quality results show that incentives

likely did not bring us “worse” data quality.

In fact, operators in Treatment Group 4 ($20 

prepaid ATM card incentive delivered in

priority mail), on average answered

significantly more questions and there was no

significant difference in “accuracy” as

measured by comparing the sampling frame

value for total land to the reported value.

However, we did see a slight increase in the

amount of editing required for the prepaid 

incentive groups over the other treatment

groups. In general, incentives helped provide

NASS with more data, as well as possibly

better data (or at least no worse).

4. CONCLUSION

Indirect monetary incentives in the form of

$20 ATM cards proved to be effective at

increasing survey response rates for the

ARMS Phase III Core. Mail response rates

for the prepaid incentive groups were

significantly higher than the control group,

saving a large amount of money on costly 

face-to-face follow-up interviews. Overall

response rates were higher as well, providing 

NASS and ERS with more completed records

for analysis.

The data provided by respondents in the

incentive groups were comparable to those

collected in the control group. There was

slightly less item nonresponse for the

incentive groups, but slightly more editing 

required. There were no differences in the

control data for total land compared to the

survey data collected for total land.

It is not known what effects the continued use

of indirect monetary incentives may have on

the response rates for ARMS or for NASS’

entire survey program. If the use of indirect

monetary incentives is implemented into

operational programs, NASS should conduct

research to determine if there are any negative

wide-ranging effects of their use.

Finally, this project was unable to investigate

whether or not the use of indirect monetary 

incentives affected nonresponse bias.
23

5. RECOMMENDATIONS

The following recommendations are offered 

from this research project.

1. Indirect monetary incentives should be

used on a large scale for the ARMS Phase

III Core mail-out/mail-back sample. In

particular, prepaid ATM cards sent by first

class or priority mail should be

implemented on a regular basis for the

Core sample.

2. Priority mail should be considered in the

mail-out/mail-back methodology that is

implemented to deliver the questionnaire

and other survey material to sampled

operations. When used in combination

with a prepaid ATM incentive, it was an

effective component of a mail-out/mailback data collection strategy. However,

when used alone, priority mail was not

effective at increasing mail response rates.

3. Any future incentives use should be

included as one piece of the larger mail

data collection strategy. The mailout/mail-back data collection methodology 

used for this research was a

comprehensive plan that incorporated 

many important components to ensure

high response, including an advance letter,

and two mail-out packets with a reminder

postcard in-between. The number and

quality of contacts is crucial to obtaining

reasonable mail response rates for mailout/mail-back surveys.

4. Any incentives considered for other

components of ARMS should be part of a

research plan designed to test incentive

effectiveness. This experiment did not

examine the use of incentives with the

face-to-face data collection methodology

or for the other versions of the ARMS

questionnaires.

5. Prior to implementing the use of indirect

monetary incentives into operational

programs, NASS should first conduct

research to determine whether there are

any negative wide ranging effects of their

use, especially in terms of response rates

to programs not using incentives. This

may include the creation and maintenance

of new data items stored in the NASS data

warehouse.

6. Prior to any future implementation of the

use of indirect monetary incentives into

operational programs, NASS should

conduct research to investigate their

effects on nonresponse bias. It is not

known what effects the use of indirect

monetary incentives may have had or may

have in the future on nonresponse bias for

ARMS Phase III.

6. REFERENCES

Church, A. H. 1993. “Estimating the Effect of

Incentives on Mail Survey Response Rates:

A Meta-Analysis.” Public Opinion 

Quarterly 57:62-79.

Hopper, R. (Forethcoming). 2004 Agricultural

Resource Management Survey (ARMS)

Phase III Survey Administration Analysis.

United States Department of Agriculture,

National Agricultural Statistics Service.

James, J. M., and R. Bolstein. 1992. “Large

Monetary Incentives and Their Effect on 

Mail Survey Response Rates.” Public

Opinion Quarterly 56:442-453.

James, J. M., and R. Bolstein. 1990. “The

Effect of Monetary Incentives and FollowUp Mailings on the Response Rate and

Response Quality in Mail Surveys.” Public

Opinion Quarterly 54:346-361.
24

Kay, W. R., S. Boggess, K. Selvavel, and M.

F.McMahon. 2001. “The Use of Targeted

Incentives to Reluctant Respondents on

Response Rate and Data Quality.”

Proceedings of the Survey Research

Methods Section, American Statistical

Association.

Moore, D. and L. An. 2001. “The Effect of

Repetitive Token Incentives and Priority

Mail On Response to Physician Surveys.”

Proceedings of the Government Section on

Survey Methods, American Statistical

Association.

Neter, J., M. H. Kutner, C. J. Nachtsheim, and 

W. Wasserman. 1996. Applied Linear

Statistical Models, Fourth Edition. Boston:

WCB/McGraw-Hill.

Singer, E. 2002. “The Use of Incentives to

Reduce Nonresponse in Household

Surveys.” In Groves, R. M., D. A. Dillman,

J. L. Eltinge, R. J. A. Little (Eds.) Survey

Nonresponse (pp. 163-177). New York:

John Wiley & Sons, Inc.

Snedecor, G. W. and W. G. Cochran. 1989.

Statistical Methods: Eighth Edition. Ames,

Iowa: Iowa State University Press.
25

APPENDIX A: Treatment Group Survey Materials

Figure A1: Survey Cover Letter for Treatment Groups 1 (Control) and 3 (Priority Mail Only)
26

Figure A2: Survey Cover Letter for Treatment Groups 2 and 4 (Pre-paid Incentive Treatment

Groups)
27

Figure A3: Survey Cover Letter for Treatment Group 5 (Promised Incentive Treatment Group)
28

Figure A4: Presentation and Instruction Sheet for ATM Cards
29

Figure A5: Images of the ATM Card Incentive

Front of Card

Back of Card
30

APPENDIX B: Response Rate Significance Calculations

Multiple comparisons of the binomial parameters of interest (i.e., the proportion,

! i , of survey 

response in each treatment group) were tested using the t-test statistic, given below, for the

overall mail and overall response rates. One-sided tests were used for those comparisons that

involved Treatment Group 1 (i.e., the control group) since only positive treatment effects were of

interest (and expected). Two-sided tests were used for all comparisons not involving Treatment

Group 1. A Bonferroni adjustment was made to control the error rate for the family of ten 

significance tests conducted. Refer to Snedecor and Cochran (1989) for more details on this test.

For one-sided tests comparing treatment groups with control:

Consider 

H0 t 1 :! " ! > 0

for treatment group

t = 2,3,4,5

t-test statistic 

zW =

( ) ( )

$ $

$ $ $ $

! !

! ! ! !

t

t t

nt n

"

"

+

" #

$

%

%

&

'

(

(

1

1 1

1

1 1

Significant at α=0.05 if 

zW

>

z !

10

= 2.58

For two-sided tests comparing treatment groups with each other (excluding control group):

Consider 

H0 t i :! " ! = 0 for treatment group 

t = 2,3,4,5

and 

i = 2,3,4,5

where

t ! i

t-test statistic 

zW =

( ) ( )

$ $

$ $ $ $

! !

! ! ! !

t i

t t

t

i i

n ni

"

"

+

" #

$

%

%

&

'

(

(

1 1

Significant at α=0.05 if  | zW| >

z !

10

= 2.81
31

Table B1: Comparisons of Overall Mail Response Rates

Comparison

Response Rate

Difference 

1/

t-test

Statistic 

2/ p-value 3/

Treatment Group 2 versus Treatment Group 1 10.67 6.9997 0.0000*

Treatment Group 3 versus Treatment Group 1 2.73 1.8356 0.3333

Treatment Group 4 versus Treatment Group 1 13.77 8.9999 0.0000*

Treatment Group 5 versus Treatment Group 1 7.12 4.7166 0.0000*

Treatment Group 3 versus Treatment Group 2 -7.94 -5.1403 0.0000*

Treatment Group 4 versus Treatment Group 2 3.10 1.9587 0.5032

Treatment Group 5 versus Treatment Group 2 -3.55 -2.2691 0.2340

Treatment Group 4 versus Treatment Group 3 11.04 7.1225 0.0000*

Treatment Group 5 versus Treatment Group 3 4.39 2.8686 0.0419*

Treatment Group 5 versus Treatment Group 4 -6.65 -4.2369 0.0002*

1/ In percentage points.

2/ Comparisons involving Treatment Group 1 (the control) are based on a one-sided t-test with a Bonferroni

adjustment for the 10 comparisons; comparisons not involving Treatment Group 1 (the control) are based on a

two-sided t-test with a Bonferroni adjustment for the 10 comparisons.

3/ P-values incorporate a Bonferroni adjustment for 10 comparisons.

* = Statistically significant at the α = 0.05 level.

Table B2: Comparisons of Overall Response Rates

Comparison

Response Rate

Difference 

1/

t-test

Statistic 

2/ p-value 3/

Treatment Group 2 versus Treatment Group 1 6.98 4.6361 0.0000*

Treatment Group 3 versus Treatment Group 1 1.25 0.8134 1.0000

Treatment Group 4 versus Treatment Group 1 8.99 6.0395 0.0000*

Treatment Group 5 versus Treatment Group 1 5.05 3.3290 0.0044*

Treatment Group 3 versus Treatment Group 2 -5.72 -3.8123 0.0014*

Treatment Group 4 versus Treatment Group 2 2.01 1.3885 1.0000

Treatment Group 5 versus Treatment Group 2 -1.93 -1.3046 1.0000

Treatment Group 4 versus Treatment Group 3 7.74 5.2097 0.0000*

Treatment Group 5 versus Treatment Group 3 3.80 2.5086 0.1218

Treatment Group 5 versus Treatment Group 4 -3.94 -2.6967 0.0705

1/ In percentage points.

2/ Comparisons involving Treatment Group 1 (the control) are based on a one-sided t-test with a

Bonferroni adjustment for the 10 comparisons; comparisons not involving Treatment Group 1 (the control) are

based on a two-sided t-test with a Bonferroni adjustment for the 10 comparisons.

3/ P-values incorporate a Bonferroni adjustment for 10 comparisons.

* = Statistically significant at the α = 0.05 level.
32

APPENDIX C: Response Rates by State and Treatment Group

Table C1: Response Rates by State for Treatment Group 1 (Control)

1st Mailing Returns Postcard 2nd Mailing Returns Overall Mail Returns

Face-to-Face

Follow-Up

Completes

Overall Returns &

Completes

State N

Count Response

Rate Count

Resp.

Rate

Increase

Count

Resp.

Rate

Increase

Count Response

Rate Count ResponseRate Count ResponseRate

Arkansas 90 6 6.7% 4 4.4% 16 17.8% 30 33.3% 39 43.3% 69 76.7%

California 202 13 6.4% 11 5.4% 19 9.4% 58 28.7% 83 41.1% 141 69.8%

Florida 144 8 5.6% 6 4.2% 19 13.2% 38 26.4% 57 39.6% 95 66.0%

Georgia 126 10 7.9% 6 4.8% 16 12.7% 36 28.6% 61 48.4% 97 77.0%

Illinois 115 14 12.2% 8 7.0% 18 15.7% 43 37.4% 28 24.3% 71 61.7%

Indiana 135 11 8.1% 10 7.4% 16 11.9% 45 33.3% 32 23.7% 77 57.0%

Iowa 123 7 5.7% 6 4.9% 17 13.8% 32 26.0% 18 14.6% 50 40.7%

Kansas 119 8 6.7% 9 7.6% 12 10.1% 39 32.8% 42 35.3% 81 68.1%

Minnesota 107 5 4.7% 8 7.5% 9 8.4% 26 24.3% 34 31.8% 60 56.1%

Missouri 138 8 5.8% 7 5.1% 15 10.9% 39 28.3% 36 26.1% 75 54.3%

Nebraska 162 13 8.0% 9 5.6% 11 6.8% 42 25.9% 31 19.1% 73 45.1%

North Carolina 125 10 8.0% 6 4.8% 15 12.0% 38 30.4% 51 40.8% 89 71.2%

Texas 119 9 7.6% 9 7.6% 18 15.1% 43 36.1% 62 52.1% 105 88.2%

Washington 136 9 6.6% 13 9.6% 10 7.4% 38 27.9% 50 36.8% 88 64.7%

Wisconsin 107 6 5.6% 5 4.7% 14 13.1% 39 36.4% 25 23.4% 64 59.8%

15 State Total 1,948 137 7.0% 117 6.0% 225 11.6% 586 30.1% 649 33.3% 1235 63.4%

Table C2: Response Rates by State for Treatment Group 2

1st Mailing Returns Postcard 2nd Mailing Returns Overall Mail Returns

Face-to-Face

Follow-Up

Completes

Overall Returns &

Completes

State N

Count Response

Rate Count

Resp.

Rate

Increase

Count

Resp.

Rate

Increase

Count Response

Rate Count ResponseRate Count ResponseRate

Arkansas 85 15 17.6% 7 8.2% 13 15.3% 39 45.9% 36 42.4% 75 88.2%

California 196 21 10.7% 17 8.7% 13 6.6% 60 30.6% 78 39.8% 138 70.4%

Florida 144 12 8.3% 8 5.6% 26 18.1% 55 38.2% 42 29.2% 97 67.4%

Georgia 121 8 6.6% 15 12.4% 17 14.0% 44 36.4% 59 48.8% 103 85.1%

Illinois 116 14 12.1% 16 13.8% 15 12.9% 52 44.8% 29 25.0% 81 69.8%

Indiana 146 22 15.1% 14 9.6% 15 10.3% 55 37.7% 37 25.3% 92 63.0%

Iowa 121 21 17.4% 18 14.9% 13 10.7% 56 46.3% 13 10.7% 69 57.0%

Kansas 119 14 11.8% 9 7.6% 12 10.1% 41 34.5% 42 35.3% 83 69.7%

Minnesota 106 8 7.5% 13 12.3% 20 18.9% 45 42.5% 28 26.4% 73 68.9%

Missouri 137 17 12.4% 15 10.9% 18 13.1% 62 45.3% 25 18.2% 87 63.5%

Nebraska 160 15 9.4% 17 10.6% 13 8.1% 64 40.0% 31 19.4% 95 59.4%

North Carolina 118 16 13.6% 8 6.8% 14 11.9% 44 37.3% 49 41.5% 93 78.8%

Texas 128 12 9.4% 18 14.1% 20 15.6% 54 42.2% 51 39.8% 105 82.0%

Washington 134 14 10.4% 25 18.7% 14 10.4% 65 48.5% 34 25.4% 99 73.9%

Wisconsin 110 14 12.7% 14 12.7% 17 15.5% 55 50.0% 21 19.1% 76 69.1%

15 State Total 1,941 223 11.5% 214 11.0% 240 12.4% 791 40.8% 575 29.6% 1366 70.4%
33

Table C3: Response Rates by State for Treatment Group 3

1st Mailing Returns Postcard 2nd Mailing Returns Overall Mail Returns

Face-to-Face

Follow-Up

Completes

Overall Returns &

Completes State N

Count Response

Rate Count Resp. RateIncrease Count Resp. RateIncrease Count ResponseRate Count ResponseRate Count ResponseRate

Arkansas 78 11 14.1% 8 10.3% 7 9.0% 30 38.5% 43 55.1% 73 93.6%

California 203 16 7.9% 16 7.9% 18 8.9% 65 32.0% 80 39.4% 145 71.4%

Florida 144 9 6.3% 8 5.6% 16 11.1% 41 28.5% 50 34.7% 91 63.2%

Georgia 121 7 5.8% 5 4.1% 16 13.2% 30 24.8% 63 52.1% 93 76.9%

Illinois 112 12 10.7% 9 8.0% 16 14.3% 43 38.4% 17 15.2% 60 53.6%

Indiana 142 12 8.5% 5 3.5% 18 12.7% 40 28.2% 35 24.6% 75 52.8%

Iowa 126 12 9.5% 9 7.1% 10 7.9% 39 31.0% 16 12.7% 55 43.7%

Kansas 116 10 8.6% 10 8.6% 21 18.1% 45 38.8% 37 31.9% 82 70.7%

Minnesota 111 4 3.6% 8 7.2% 16 14.4% 34 30.6% 45 40.5% 79 71.2%

Missouri 131 10 7.6% 8 6.1% 13 9.9% 41 31.3% 31 23.7% 72 55.0%

Nebraska 168 7 4.2% 9 5.4% 13 7.7% 44 26.2% 38 22.6% 82 48.8%

North Carolina 115 9 7.8% 12 10.4% 21 18.3% 43 37.4% 51 44.3% 94 81.7%

Texas 122 7 5.7% 9 7.4% 20 16.4% 40 32.8% 53 43.4% 93 76.2%

Washington 140 11 7.9% 9 6.4% 20 14.3% 49 35.0% 43 30.7% 92 65.7%

Wisconsin 106 9 8.5% 10 9.4% 16 15.1% 51 48.1% 14 13.2% 65 61.3%

15 State Total 1,935 146 7.5% 135 7.0% 241 12.5% 635 32.8% 616 31.8% 1251 64.7%

Table C4: Response Rates by State for Treatment Group 4

1st Mailing Returns Postcard 2nd Mailing Returns Overall Mail Returns

Face-to-Face

Follow-Up

Completes

Overall Returns &

Completes

State N

Count Response

Rate Count

Resp.

Rate

Increase

Count

Resp.

Rate

Increase

Count Response

Rate Count ResponseRate Count ResponseRate

Arkansas 89 11 12.4% 16 18.0% 17 19.1% 49 55.1% 36 40.4% 85 95.5%

California 207 14 6.8% 22 10.6% 29 14.0% 79 38.2% 70 33.8% 149 72.0%

Florida 137 5 3.6% 15 10.9% 18 13.1% 47 34.3% 56 40.9% 103 75.2%

Georgia 123 15 12.2% 13 10.6% 20 16.3% 54 43.9% 46 37.4% 100 81.3%

Illinois 115 11 9.6% 17 14.8% 17 14.8% 53 46.1% 30 26.1% 83 72.2%

Indiana 140 20 14.3% 18 12.9% 24 17.1% 71 50.7% 26 18.6% 97 69.3%

Iowa 126 16 12.7% 17 13.5% 20 15.9% 60 47.6% 9 7.1% 69 54.8%

Kansas 119 12 10.1% 11 9.2% 15 12.6% 44 37.0% 44 37.0% 88 73.9%

Minnesota 112 13 11.6% 15 13.4% 19 17.0% 54 48.2% 31 27.7% 85 75.9%

Missouri 132 14 10.6% 13 9.8% 20 15.2% 58 43.9% 24 18.2% 82 62.1%

Nebraska 165 16 9.7% 18 10.9% 24 14.5% 74 44.8% 27 16.4% 101 61.2%

North Carolina 115 11 9.6% 11 9.6% 21 18.3% 48 41.7% 44 38.3% 92 80.0%

Texas 122 6 4.9% 13 10.7% 22 18.0% 47 38.5% 55 45.1% 102 83.6%

Washington 147 21 14.3% 17 11.6% 24 16.3% 68 46.3% 37 25.2% 105 71.4%

Wisconsin 103 12 11.7% 10 9.7% 17 16.5% 50 48.5% 22 21.4% 72 69.9%

15 State Total 1,952 197 10.1% 226 11.6% 307 15.7% 856 43.9% 557 28.5% 1413 72.4%
34

Table C5: Response Rates by State for Treatment Group 5

1st Mailing Returns Postcard 2nd Mailing Returns Overall Mail Returns

Face-to-Face

Follow-Up

Completes

Overall Returns &

Completes

State N

Count Response

Rate Count

Resp.

Rate

Increase

Count

Resp.

Rate

Increase

Count Response

Rate Count ResponseRate Count ResponseRate

Arkansas 91 9 9.9% 9 9.9% 13 14.3% 34 37.4% 44 48.4% 78 85.7%

California 207 11 5.3% 18 8.7% 27 13.0% 73 35.3% 75 36.2% 148 71.5%

Florida 143 17 11.9% 9 6.3% 24 16.8% 54 37.8% 59 41.3% 113 79.0%

Georgia 117 9 7.7% 6 5.1% 17 14.5% 35 29.9% 54 46.2% 89 76.1%

Illinois 116 19 16.4% 9 7.8% 19 16.4% 49 42.2% 24 20.7% 73 62.9%

Indiana 134 14 10.4% 7 5.2% 26 19.4% 54 40.3% 39 29.1% 93 69.4%

Iowa 128 15 11.7% 15 11.7% 14 10.9% 49 38.3% 17 13.3% 66 51.6%

Kansas 120 18 15.0% 5 4.2% 16 13.3% 41 34.2% 46 38.3% 87 72.5%

Minnesota 104 7 6.7% 8 7.7% 15 14.4% 39 37.5% 27 26.0% 66 63.5%

Missouri 142 21 14.8% 11 7.7% 24 16.9% 60 42.3% 32 22.5% 92 64.8%

Nebraska 161 9 5.6% 12 7.5% 17 10.6% 50 31.1% 30 18.6% 80 49.7%

North Carolina 121 15 12.4% 13 10.7% 13 10.7% 45 37.2% 43 35.5% 88 72.7%

Texas 114 12 10.5% 8 7.0% 18 15.8% 42 36.8% 49 43.0% 91 79.8%

Washington 143 11 7.7% 12 8.4% 17 11.9% 54 37.8% 49 34.3% 103 72.0%

Wisconsin 105 6 5.7% 14 13.3% 16 15.2% 45 42.9% 20 19.0% 65 61.9%

15 State Total 1,946 193 9.9% 156 8.0% 276 14.2% 724 37.2% 608 31.2% 1332 68.4%

Table C6: Overall Response Rates by State and Treatment Group

Overall Returns & Completes

Treatment Group 1

(Control) Treatment Group 2 Treatment Group 3 Treatment Group 4 Treatment Group 5 State

Count Response

Rate Count ResponseRate Count ResponseRate Count Response Rate Count ResponseRate

Arkansas 69 76.7% 75 88.2% 73 93.6% 85 95.5% 78 85.7%

California 141 69.8% 138 70.4% 145 71.4% 149 72.0% 148 71.5%

Florida 95 66.0% 97 67.4% 91 63.2% 103 75.2% 113 79.0%

Georgia 97 77.0% 103 85.1% 93 76.9% 100 81.3% 89 76.1%

Illinois 71 61.7% 81 69.8% 60 53.6% 83 72.2% 73 62.9%

Indiana 77 57.0% 92 63.0% 75 52.8% 97 69.3% 93 69.4%

Iowa 50 40.7% 69 57.0% 55 43.7% 69 54.8% 66 51.6%

Kansas 81 68.1% 83 69.7% 82 70.7% 88 73.9% 87 72.5%

Minnesota 60 56.1% 73 68.9% 79 71.2% 85 75.9% 66 63.5%

Missouri 75 54.3% 87 63.5% 72 55.0% 82 62.1% 92 64.8%

Nebraska 73 45.1% 95 59.4% 82 48.8% 101 61.2% 80 49.7%

North Carolina 89 71.2% 93 78.8% 94 81.7% 92 80.0% 88 72.7%

Texas 105 88.2% 105 82.0% 93 76.2% 102 83.6% 91 79.8%

Washington 88 64.7% 99 73.9% 92 65.7% 105 71.4% 103 72.0%

Wisconsin 64 59.8% 76 69.1% 65 61.3% 72 69.9% 65 61.9%

15 State Total 1,235 63.4% 1,366 70.4% 1,251 64.7% 1,413 72.4% 1,332 68.4%
35

Appendix D: Response Rates by Farm Type and Treatment Group

Table D1: Overall Response Rates by Farm Type and Treatment Group

Overall Returns & Completes

Treatment Group 1

(Control) Treatment Group 2 Treatment Group 3 Treatment Group 4 Treatment Group 5 Farm Type

Count Response

Rate Count ResponseRate Count ResponseRate Count Response Rate Count ResponseRate

Grains, Oilseeds, Etc. 250 50.3 306 61.6 264 53.1 315 63.8 291 58.2

Tobacco 13 54.2 18 72.0 15 68.2 12 60.0 14 60.9

Cotton 15 65.2 16 72.7 16 72.7 24 88.9 16 69.6

Veggies, Melons 30 48.4 34 57.6 39 63.9 40 65.6 35 55.6

Fruits, Nuts, Berries 145 71.4 150 74.3 145 69.7 157 74.4 148 71.2

Nursery, Greenhouse 126 67.0 138 73.8 130 71.4 149 79.7 149 80.1

Christmas Trees 1 100.0 2 66.7 2 100.0 2 100.0 3 100.0

Other Crops 127 72.2 134 77.9 123 71.5 135 77.6 133 76.4

Hogs 1 100.0 0 0.0 2 100.0 1 100.0 0 0.0

Milk, Dairy 106 62.4 118 68.6 100 58.8 114 66.7 101 60.8

Cattle 263 67.3 275 71.2 260 67.7 291 74.4 272 70.8

Sheep, Goats 9 75.0 10 71.4 10 76.9 9 81.8 13 92.9

Equine 23 67.6 31 88.6 22 62.9 25 69.4 27 79.4

Poultry, Eggs 115 77.7 120 81.6 109 75.2 123 84.2 113 76.4

Aquaculture 6 85.7 5 71.4 4 50.0 7 87.5 6 85.7

Other Animals 1 16.7 5 71.4 6 85.7 6 85.7 7 87.5

Other 4 80.0 4 80.0 4 80.0 3 60.0 4 80.0
36

Appendix E: Response Rates by Value of Sales and Treatment Group

Table E1 : Overall Response Rates By Value Of Sales

Overall Returns & Completes

Treatment Group 1

(Control) Treatment Group 2 Treatment Group 3 Treatment Group 4 Treatment Group 5

Value Of Sales

Count Response

Rate Count ResponseRate Count ResponseRate Count ResponseRate Count ResponseRate

Less Than $20,000 266 76.4 284 81.8 255 75.2 294 81.7 271 79.5

$20,000 - $39,999 70 68.6 79 76.7 79 78.2 82 79.6 87 75.0

$40,000 - $99,999 67 60.9 75 69.4 73 59.8 67 67.7 83 77.6

$100,000 - $249,999 318 58.5 381 70.3 335 62.4 380 69.6 361 67.0

$250,000 - $499,999 329 61.5 344 64.5 328 62.4 373 70.0 349 65.6

$500,000 Or More 185 59.9 203 65.9 181 58.4 217 69.8 181 58.2
37

Appendix F: Detailed Data Collection Costs by Treatment Group

This appendix contains detailed information on how the costs in Table 16 were calculated. Table

F1 is a reproduction of Table 16.

Table F1: Detailed Data Collection Overall Costs by Treatment Group

Costs (In Dollars)

Treatment

Group 1/ N Postage NPC

Printing

NPC Extra

Admin

ATM Card

Charges

ATM Card

Admin

Face-toFace

Follow-up

Costs

Total Average Per

Sample

Average Per

Complete

1 (control) 1,948 4,882 6,638 --NA-- --NA-- --NA-- 176,291 187,811 96.41 152.07

2 1,941 4,833 6,295 2,000 13,195 1,650 154,154 182,127 93.83 133.33

3 1,935 15,207 6,515 --NA-- --NA-- --NA-- 170,951 192,673 99.57 154.02

4 1,952 14,953 6,297 2,000 13,683 1,659 148,356 186,948 95.77 132.31

5 1,946 5,284 6,655 2,000 18,832 1,654 160,475 194,900 100.15 146.32

5' 1,946 5,284 6,655 2,000 11,648 1,654 160,475 187,717 96.46 140.93

1/ Treatment Group 5 includes all respondents in the treatment group who received an ATM card (whether their questionnaires were received in

the mail or by a face-to-face interview) and Treatment Group 5' includes only those respondents in the treatment group who received an ATM

card because they returned their questionnaires in the mail.

The calculations for each of the columns are below.

Postage

Postage includes postage for all mailings to the sampled operations and the return postage for

completed questionnaires.

Specifically, Postage was calculated as follows:

Postage = ni·(Ai + Bi + Ci) + mci·Ei + Di· (ni – r(t1)i) + I5·r5·F5

where:

i = the treatment group number

ni = the sample size of treatment group i

r(t1)i = the number of surveys returned/completed in treatment group i by date t1 (received

before the date NASS began following up with a second survey mailing)

Ai = postage cost associated with mailing the pre-survey letter ($0.301 for i=1,2,3,4,5)

Bi = postage cost associated with the initial form mail-out ($0.967 for i=1,2,5, $3.850 for 

i=3,4)

Ci = postage cost associated with the post-card follow-up/thank you ($0.187 for i=1,2,3,4,5)

Di = postage cost associated with mailing the second, follow-up survey form ($0.967 for

i=1,2,5, $3.850 for i=3,4)

mci = the number of questionnaires returned in the mail for treatment group i

Ei = postage cost associated with a return questionnaire mailing ($0.70 for all treatment

groups)

I5= 1 if i=5 ; 0 otherwise

r5 = the number of overall respondents in treatment group 5

F5 = postage costs associated with mailing ATM cards to respondents in treatment group 5

($0.301)
38

NPC Printing

The NPC Printing includes the costs to print all survey-related materials (i.e., cover letters,

questionnaires, etc.). Specifically, NPC Printing was calculated as follows:

NPC Printing = ( ) * ( ) ( ) *( ) ( ) *( ) ( ) *( ) *( ) n A B C n ( ) ( ) i i + i + i + i

! rt1 i Di + ni! rt 2 i Ei + I5 r5 F5

where:

i = the incentive (treatment) group number

ni = the sample size of incentive i

r(t1)i = the number of surveys returned/completed in treatment group i by date t1 (received before the date NASS

began following up with a second survey mailing)

r(t 2)i = the number of surveys returned/completed in treatment group i by date t2 (received before the date NASS

began following up with enumerator visits)

Ai = printing costs associated with mailing the pre-survey letter (Cover Letter: $0.045 for i=1,2,3,4,5

+ ARMS Brochure: $0.26 for i=1,2,3,4,5 )

Bi = printing costs associated with the initial form mail-out (Cover Letter: $0.045 for i=1,2,3,4,5 +

Farm Foundation Flyer: $0.26 for i=1,2,3,4,5 + Report Form: $1.040 for i=1,2,3,4,5 + Debit

Card Instructions: $0.045 for i=2,4,5)

Ci = printing costs associated with the post-card follow-up/thank you ($0.030 for i=1,2,3,4,5)

Di = printing costs associated with mailing the second, follow-up survey form (Cover Letter: $0.045

for i=1,2,3,4,5 + Report Form: $1.040 for i=1,2,3,4,5 + Debit Card Instructions: $0.045 for i=5)

Ei = printing costs associated with the enumerator follow-up (Report Form: $1.040 for i=1,2,3,4,5)

I5 = 1 if i=5 ; 0 otherwise

r5 = the number of overall respondents in treatment group 5

F5 = printing costs associated with mailing ATM cards to respondents in treatment group 5 (Cover

Letter: $0.045 for i=5 + Debit Card Instructions: $0.045 for i=5)

NPC Extra Admin

This amount is the National Processing Center’s (NPC) best estimate for the labor cost

associated with administering the incentive program for all treatment groups. It includes 20 staff

days at the GS 5/6 level for statistical assistants. The work performed consisted of preparing the

cards in the initial mail packages, preparing 15 waves of mailings for the Treatment Group 5 

"promise to pay" respondents (i.e., this was to send out their ATM cards after they returned their

questionnaires), and processing replacement ATM cards as requested by the respondents. The

charge for 20 staff days was approximately $6,000. Since individual charges were unavailable

for each treatment group, for simplicity, the total was evenly divided between each of the

treatment groups that provided an incentive.

ATM Card Charges

ATM Card Charges includes all charges associated with ATM card usage among the recipients.

They include: ATM withdrawals, ATM withdrawal fees, POS purchases, POS purchase fees,

ATM balance inquiry, ATM failure fees, POS purchase failure fees. Refer to Table 6 for details

on each of these individual charges.
39

ATM Card Admin

ATM Card Admin includes the amount JPMorgan Chase charged for the physical cards. This

was computed as follows:

ATM Card Admin = $0.85

ni

for i=2,4,5

where:

ni = the sample size of incentive i

NASS was charged a “per card” cost of $0.85 (appearing in the above equation).

Not included is the trivial expense for the few replacement cards that were requested.

Note: Not included is an additional $1,000 “training” charge from JPMorgan Chase. This was

excluded as it is a one-time charge that would not necessarily be incurred in the future.

NASDA Costs

NASDA Costs includes enumeration costs for face-to-face follow-up interviews. Face-to-face

follow-up interviews were attempted for all sampled operations that did not return a

questionnaire in the mail by February 21. In addition, a small number of follow-up interviews

were conducted for operators who mailed back “completed” questionnaires with inadequate data.

The following steps detail how the NASDA Costs were calculated:

1. An average enumeration cost was computed by state for every sampling unit in the

sample that needed to be enumerated. Those who mailed in their surveys by February 21

were given an average enumeration cost of zero and taken out of the denominator since

no enumerator visit was necessary for them. The total 2004 ARMS Phase III NASDA

expenditures in each state were then divided among everyone who did not mail in their

survey by this date. (These average costs were all computed at the state level since

enumeration costs were so variable across the states for ARMS Phase III.) Specifically:

A

E

n m j

j

j j

=

!

where:

Aj = Average enumeration cost in state j

E j = NASDA Expenditures in state j (provided in 2004 Agricultural Resource Management

Survey (ARMS) Phase III Survey Administration Analysis)

nj = Overall sample size in state j for ARMS Phase III (long and core form)

mj = All questionnaires mailed-in by February 21

2. Sum the average enumeration costs for all the sampling units in each treatment group to

get the estimated total NASDA expenditure among each treatment group. These results
40

can be found in the NASDA Costs column of Table F1.

Total

Total = NPC Postage + NPC Printing + NPC Extra Admin + ATM Card Charges +

ATM Card Admin + NASDA Costs

Averages

Average per Sample = Total /

ni

Average per Complete = Total /

ri

where:

ri = total survey responses in treatment group i (Item Code 910 in (1,3,5) where 1=mail complete,

3=non-mail complete, 5=out of scope)

ni = total survey samples
41

APPENDIX G: Nonparametric F-Test

The following explains how the nonparametric F-test was used to test if there was evidence of a

difference in reporting total land operated (relative to control data) across treatment groups:

• For each operation, calculate the absolute value of the difference between the NASS

control data value (controlTypeID 100) for total land operated and self-reported total

land from ARMS Phase III Core (Item Code 26).

• Take this absolute difference and “standardize” it by dividing by the control value.

This provides the percentage difference between what is reported and NASS’ control

value.

• Take all of these percentage differences and rank them from smallest to largest. (Any

“ties” in the ranking process were given the mean of the ties’ ranks. So if the three

smallest absolute differences were the same, they would each get a ranking of

(1+2+3)/3, or 2).

• Test the hypothesis that the mean rank is the same for each treatment group – this is

done with the nonparametric rank F-test (Neter, et al., 1996):

Let: controlTypeId

| controlTypeId100 - IC26 |

Y =

ij

ij ij

ij

And: Rij be the rank of Yij

Calculate: 

!

MSTR =

ni(Ri• - R••)

2

i

"

r -1

and 

 

!

MSE =

ni(Rij- Ri• )

2

j

"

i

"

nT-1

MSTR = Mean Squared Error for Treatment Groups

MSE = Mean Squared Error

Where:

!

Ri• =

Rij

j

"

ni

and 

 

!

R•• =

Rij

j

"

i

"

nT

=

(nT + 1)

2

nT is the total sample size across all treatment groups

ni is the sample size in treatment group i

The F-test statistic is: MSE

MSTR

F =

*

R

with (r-1, nT – r) degrees of freedom (r is the number of treatment

groups)

This test resulted in an F-test statistic of 0.96 with a p-value of 0.4259. The test was re-run using 

a non-standardized difference (i.e., Yij = |controlTypeId100ij – IC26ij| ); this resulted in an F-test

statistic of 0.71 with a p-value of 0.5862.