Targeted Data 

Collection Efforts for the 

2012 ARMS III

Melissa Mitchell

Kathy Ott 

Jaki McCarthy

United States 

Department of

Agriculture

National

Agricultural

Statistics

Service

Research and

Development Division

Washington DC 20250

RDD Research Report

Number RDD-15-01

May 2015

This report has been prepared for limited distribution to the research community outside the U.S. Department of 

Agriculture (USDA). The views expressed herein are not necessarily those of NASS or USDA.

i

EXECUTIVE SUMMARY

USDA’s National Agricultural Statistics Service (NASS) conducts the annual Agricultural 

Resource Management Survey (ARMS). ARMS is conducted in three phases and ARMS III, the 

economic portion that takes an average of 90 minutes to complete with an enumerator, suffers 

from much lower response rates than other phases of ARMS, as well as other NASS surveys. In 

order to focus data collection efforts and resources on those operations that are unlikely to 

respond, NASS began to use data mining approaches to target those operations. 

In 2011, likely nonrespondents to ARMS III were flagged using nonresponse propensities. The 

flagging procedure in 2011 was helpful in targeting these nonrespondents, but improvements 

were made for the 2012 data collection. This report covers the 2012 data collection period but 

also highlights the changes from 2011 to 2012. The differences in the flagging procedures 

between the two years are summarized, along with results of the 2012 study. In addition to the 

new flagging procedure, in 2012, NASS introduced the addition of impact groups to identify 

those likely nonrespondents that have the most influence on survey estimates. In 2012, we found 

that providing a logo token item to highly likely nonrespondents was correlated with increased 

response rates. However, procedures in 2012 were not implemented consistently by all state 

offices, so we cannot determine if other experimental procedures may have been effective.

ii

RECOMMENDATIONS

1. Continue using the response propensity scores as a tool to identify the operations that are

least likely to respond to ARMS III.

2. Conduct studies that can be monitored more rigorously to determine if targeted procedures 

are effective at increasing response rates.

3. Consider using response propensity scores that identify operators who are less likely to 

respond, but not the LEAST likely to respond. Maybe this group of respondents can be 

persuaded to respond using targeted methods.

4. Consider using token items as a routine component for the data collection strategy used for 

the ARMS III. Although NASS has not done a controlled experiment using token items, there 

are several studies in survey methods literature that show increases in response rates when 

they are used.
iii

TABLE OF CONTENTS

1. INTRODUCTION ...................................................................................................................1

1.1 Background..................................................................................................................... 1

1.2 Classification Tree Models............................................................................................. 3

2. METHODS..............................................................................................................................4

3. RESULTS................................................................................................................................6

3.1 Field Office Adherence with Interview Instructions ...................................................... 7

3.2 Enumerator Incentives.................................................................................................... 9

4. CONCLUSIONS AND RECOMMENDATIONS ................................................................10

4.1 Recommendations......................................................................................................... 11

5. REFERENCES ......................................................................................................................12

APPENDIX A: Impact Group Memo and Procedures............................................................... A-1

APPENDIX B: Census of Agriculture Splitting Variables .........................................................B-1

APPENDIX C: Scoring Supplement Sheet .................................................................................C-1

APPENDIX D: Conclusions from the ARMS Callbacks........................................................... D-1
1

Targeted Data Collection Efforts for the 2012 ARMS III

Melissa Mitchell 1, Kathy Ott2and Jaki McCarthy3

Abstract

Low response rates for the Agricultural Resource Management Survey Phase III 

(ARMS III) are a growing concern. Beginning in 2011 and enhanced in 2012, 

data mining techniques were used to proactively identify operations that are likely 

to be nonrespondents to ARMS III. Once identified, a measure of the impact on 

the calibration targets of each likely nonrespondent was calculated. Likely 

nonrespondents were then divided into a treatment and control group to evaluate 

data collection methods. Likely nonrespondents in the treatment group were 

targeted with more intense data collection strategies in an effort to increase 

response rates. These strategies were evaluated to determine whether they could 

increase response to the survey but because these strategies were not uniformly 

applied, definitive conclusions about their effectiveness cannot be drawn. 

Key Words: Nonresponse propensity scores, impact scores, data mining, targeted data 

collection, ARMS, calibration target

1. INTRODUCTION 

1.1 Background 

The Agricultural Resource Management Survey (ARMS) is an annual survey conducted by the 

National Agricultural Statistics Service (NASS) in partnership with the Economic Research 

Service (ERS). It is conducted in three phases over the course of the year. The first phase is the 

screening phase. The second phase focuses on production expenses, chemical use, and the 

targeted commodity (targeted commodities change annually). The third phase of ARMS (ARMS 

III) focuses on financial data such as expenses and income. The focus of this research is on low 

response rates to ARMS III and strategies to overcome them. 

There are multiple versions of ARMS III– the Cost and Returns Report (CRR), Core, and 

commodity specific version(s) (i.e., Soybean in 2012). The CRR is a comprehensive survey of 

operational practices and economic well being conducted in the contiguous 48 states. The CRR 

and commodity specific versions have traditionally been personally enumerated, however, 

 

1 Melissa Mitchell was a Mathematical Statistician with the National Agricultural Statistics Service, Research and 

Development Division, 3251 Old Lee Highway, Room 305, Fairfax, VA 22030. 

2 Kathy Ott is a Mathematical Statistician with the National Agricultural Statistics Service, Methodology Division, 

1400 Independence Ave, SW, Washington, DC, 20250.

3

Jaki McCarthy is a Senior Cognitive Research Psychologist with the National Agricultural Statistics Service, 

Research and Development Division, 3251 Old Lee Highway, Room 305, Fairfax, VA 22030. 
2

beginning in 2012, they were mailed out, with personal visit followup for nonrespondents. The 

Core is not as comprehensive as the CRR and is only administered in the 15 highest cash receipt 

states in order to produce estimates for those states. The Core has always been a mail-out/mailback survey, with personal visit followup of nonrespondents. This research included all versions 

of the questionnaire. Calibration is used in ARMS III to create sample weights. By re-weighting 

records to known benchmark calibration targets, adjustments are made for nonresponse and 

coverage.

ARMS III suffers from low response rates (with higher response rates in the years ending in two 

or seven when the ARMS is combined with the mandatory Census of Agriculture). Response 

rates typically fall well below the Office of Management and Budget (OMB) target of 80% (see 

Table 1). There have been many attempts to increase response rates and reduce bias such as 

offering incentives and calibrating to known targets (McCarthy, Beckler, & Ott, 2006; Earp, 

McCarthy, Schauer, & Kott, 2008; Earp, McCarthy, Schauer, & Kott, 2009; Earp, McCarthy, 

Porter, & Kott, 2010). Recently, work has focused on proactively identifying nonrespondents 

(Earp & McCarthy, 2011; McCarthy, Jacob, & McCracken, 2010; McCarthy & Jacob, 2009). By 

identifying the highly likely nonrespondents, NASS can use targeted data collection methods for 

those cases. Targeted data collection methods for this project included having State Statisticians 

make the initial contact with operations, using in-person visits as the first contact with potential 

respondents, using token items, and offering enumerator incentives for completed cases.

Table 1: ARMS III Response Rates 2000-2011

ARMS III 

Sample Year

ARMS III 

Sample Size (n)

ARMS III 

Respondents (nr)

ARMS III 

Nonrespondents 

(nn)

ARMS III 

response rates

2000 17,903 11,295 6,608 63.1%

2001 13,313 8,500 4,813 63.8%

2002 18,219 13,484 4,735 74.0%

2003 33,861 21,278 12,583 62.8%

2004 33,908 22,966 10,942 67.7%

2005 34,937 24,704 10,233 70.7%

2006 34,203 23,237 10,966 67.9%

2007 31,924 22,304 9,620 69.9%

2008 36,388 24,066 12,322 66.1%

2009 33,348 22,753 10,595 68.2%

2010 35,431 23,285 12,146 65.7%

2011 34,070 22,130 11,940 65.0%

Total 357,505 240,002 117,503 67.1%

Since response rates fall below eighty percent, nonresponse bias analyses were conducted in the 

past to assess how nonresponse impacts ARMS III survey estimates. This analysis showed that 

although calibration does a good job of correcting for bias, some key estimates had significant 

bias after calibration (Earp et al, 2008; Earp et al, 2009; Earp et al, 2010). Calibration is a 
3

retroactive approach to correct for bias after survey data collection is complete. Ideally, we 

would like to prevent nonresponse bias as much as possible by obtaining better response during 

data collection. Proactively identifying likely nonrespondents prior to data collection and

tailoring our data collection efforts for those operations is one way to possibly increase response, 

thereby likely decreasing nonresponse bias.

1.2 Classification Tree Models

In 2010, NASS developed a procedure to flag likely nonrespondents for ARMS III using a data 

mining technique called classification trees. Classification (or decision) trees are used to predict 

the outcome of a binary variable, such as survey response/nonresponse, from auxiliary data. The 

primary objective of classification trees is classification of groups (in our case 

respondent/nonrespondent operations). For the classification trees developed for predicting 

ARMS nonresponse, the auxiliary data used (which is available for both ARMS respondents and 

nonrespondents) were variables from the 2002 and 2007 Census of Agriculture. 

The classification trees are created by segmenting a dataset by a set of simple rules. The rules 

assign an observation to a segment based on the input variable that maximizes the difference 

between two groups based on the target (in this case survey nonresponse). The dataset is 

sequentially split into subsets by these rules until no more splits can be created. No more splits 

can be created when the sample size per segment is too small, no more significant splits can be 

created, or the maximum depth of the tree is too large.

Classification trees create a hierarchy (tree) where the segments are called nodes. The first node, 

known as the root node, contains the entire dataset. From the root node, there are branches or

paths to and from nodes within a tree. Terminal nodes are nodes that have no branches coming 

from them and are known as leaves. Each record will appear in only one of the leaves and the 

leaves will collectively contain all the records in the dataset. The leaves of interest are those that 

have the highest proportion of records with the target (in this case, refusal/non-contact).

When using classification trees, usually a single tree is created with the best initial split. 

However, an ensemble of trees was created for ARMS III, for a variety of reasons. In the single 

tree framework, the tree is created with the best initial split based on a portion of the data 

(training data), but there is no guarantee that is the best split based on the entire dataset. Also, the 

initial split directly affects the subsequent splits, so although it may be a good initial split, it may 

not identify the greatest number of operations in the target. It is possible that a split that is not 

initially the most optimal will provide better subsequent splits. By growing multiple trees (an 

ensemble), we have a richer understanding of likely nonrespondents that could possibly bias our 

key estimates, and more stable nonresponse propensity scores. 

Therefore, every variable was forced to be used as an initial split in one tree for this study. There 

were 71 variables in the dataset that could serve as the initial split and 70 of those splits were 

statistically significant at p<0.20 level (see Appendix B for a list of the variables). These 70 
4

initial splits were used separately in the model to grow 70 different trees. Therefore, each 

variable was considered when assessing characteristics of nonrespondents. All variables were 

considered for subsequent splits in the tree and those splits were determined by the splitting 

algorithm used by SAS Enterprise Miner (SAS, 2009). A total of 140 trees were created using 

the 70 variables from the Census of Agriculture (COA), 70 trees from the 2002 COA and 70 

trees from the 2007 COA. 

2. METHODS

In 2011, a threshold of 0.70 was chosen to act as a cut-off to distinguish between likely 

nonrespondents and all other operators. Therefore, all leaves (also called terminal nodes) with 

seventy percent nonresponse rate or higher were selected from each tree. This identified 543 

likely nonrespondent subgroups with over 6,878 likely nonrespondent records. If an operation 

had a score of 0.70 or higher in any of the trees, the operation was flagged as a likely 

nonrespondent. 

This 2012 study builds off the 2011 ARMS III study (Earp, Mitchell, McCarthy, & Kreuter, 

2012; Earp, Mitchell, McCarthy, & Kreuter, 2014). The procedure used for identifying 

nonrespondents in 2011 worked well, but there were some improvements made to the method for 

2012. While flagged records all had a nonresponse propensity of 70% or higher in at least one 

tree, the average nonresponse propensity across all trees ranged from 12% to 90%. Therefore, 

some of the records flagged based on a high nonresponse propensity in only some trees may, in 

fact, have a low nonresponse propensity when their scores are considered across all trees. As 

shown in Figure one, there are two distinct subgroups of flagged records with nonresponse 

propensities, those above .7 and those below. Based on this, in 2012 we only flagged operations 

whose average nonresponse propensity across all trees was greater than or equal to .70. This 

allowed us to target the operations that are the most highly likely nonrespondents. 

 

Figure One: Average Non-Response Propensity Scores for Flagged Operations in 2011
5

The flagging procedure in 2011 flagged a total of 6,878 records as likely nonrespondents across 

all questionnaire versions. This large number of operations was too high for our targeted data 

collection techniques because of the high cost of some of the data collection techniques and

limited staff resources. Because of this large number, only the records flagged from the CRR 

version were targeted for changes in data collection in 2011. Using the new procedure in 2012, 

887 operations were flagged from all questionnaire versions. These 887 flagged operations were 

randomly split into a treatment group (n=441) and comparison group (n=446) after stratifying by 

state and farm type. 

In 2011, NASS began to use these classification trees to identify likely ARMS III nonrespondent 

operations (Earp, Mitchell, McCarthy, & Kreuter, 2012; Earp, Mitchell, McCarthy, & Kreuter, 

2014). In addition to using the classification trees, in 2012 we incorporated impact groups, a 

measure combining the likelihood of response with a measure of how important an operation is 

to meeting the calibration target for particular data items. We assigned highly likely 

nonrespondent operations to one of three impact groups. Impact group 3 included highly likely 

nonrespondents that are the most important to the calibration targets. Impact group 2 included 

highly likely nonrespondents that are of mid-importance to calibration targets. Impact group 1 

included highly likely nonrespondents that are least important to calibration targets. The 

assignment of impact groups was based on a point system using list frame data. High impact 

records were defined using the following criteria: whether the record is in the top 10% of 

production or inventory for a calibration target within their state, if a record is in the top 10% of 

sales within their state, and if a record has a positive value for each calibration target. See 

Appendix A for the official 2012 data collection memorandum and procedures for details 

regarding the creation of impact groups. 

Once likely respondents and nonrespondents were identified and cost was considered, the data 

collection methodology for each group was targeted using different methods for different groups. 

Initial data collection techniques were the same for flagged operations in the treatment and 

control groups. All operations in both groups were mailed a questionnaire as the first survey 

contact. All mail nonrespondent records from both groups were sent to the state offices to be 

enumerated using a follow-up, in-person contact. As part of the in-person contact for all 

operations, the staff member explained the importance of the individual operation and provided a 

data product, NASS logo token item, or report, if available. 

Enumerator incentives were offered to enumerators who completed records in the treatment 

sample in any of the three impact groups. Enumerators were allocated a bonus of twenty dollars 

for every flagged operation in the treatment group for which they obtained a response.

In addition to the enumerator incentives which were offered for all treatment sample records, 

specific follow-up contact techniques were used for records in the treatment group based on the 

impact group assignment. For records in impact group 3 (those highly likely nonrespondent

records most important to calibration targets), records were not allowed to be held out of data 

collection (unless there was a dangerous situation), and the first non-mail follow-up contact was 
6

an in-person visit done by the director, deputy, or office staff (a supervisory enumerator could be 

assigned if no office staff were available) to set the stage for a supervisory enumerator to collect 

the survey data. For records in impact group 2, the first non-mail follow-up contact was made in 

person by a supervisory enumerator to collect the data or set the stage for an experienced 

enumerator to collect the data. For records in impact group 1, usual ARMS procedures were 

followed (i.e., an enumerator was assigned to collect the data from the operations). See Appendix 

A for the official memorandum and procedures on the follow up data collection procedures.

Field office personnel completed a scoring supplement sheet for each case in the treatment 

group. The scoring supplement contained questions about the data collection techniques that 

were used for each operation and was used to assess the effectiveness of the treatment options. 

See Appendix C for the Scoring Supplement Sheet that was used in 2012. 

3. RESULTS

The Fischer Exact Test was used to determine if the proportion of nonresponse was different for 

the records flagged as likely nonrespondents compared to those that were not flagged. Response 

rates were significantly different for these two groups (χ

2

(3, N =32,096) = 54.10, p < 0.01) (see 

Table 2). This shows that the flagging operation does indeed identify those records that are 

likely nonrespondents. Even with the extra emphasis put on these records, the response rates 

were still significantly lower. 

Table 2: Response Rates for the ARMS flagged records and other (non-flagged) records

Likely nonrespondents 

(flagged records)

Other records (non-flagged 

records)

Completed 55.6% 73.1%

Refusal 36.9% 21.8%

Inaccessible 4.9% 4.2%

Office Hold 2.9% 0.9%

N* 861 32,096

*The number of sample operations is not equal to the number of flagged operations (861 versus 887) due to 

processing inconsistencies for a small number of cases. Also, note that the sample size of likely nonrespondents 

(887) is approximately 20 times greater than the sample size of likely nonrespondents 32,096). 

The Fischer Exact Test was also used to determine if the proportion of nonresponse was different 

for the treatment group compared to the comparison group. We found no difference in response 

rates between the treatment and comparison group (χ

2

(1, N =861) = 0.0031, p = 0.96). In fact, 

the response rate was 0.1% lower in the treatment group (see Table 3). 
7

Table 3: Response Rates for the comparison group and treatment group

Comparison Group Treatment Group

Completed 55.3% (n=242) 55.2% (n=234)

Refusal 36.2% (n=158) 37.5% (n=159)

Inaccessible 4.8% (n=21) 5.2% (n=22)

Office Hold 3.7% (n=16) 2.1% (n=9)

N* 437 424

*The number of sample operations is not equal to the number of flagged operations (861 versus 887) due to 

processing inconsistencies for a small number of cases. 

The major change in data collection was for the operations that were identified as highly likely 

nonrespondents who would have the most impact on calibration targets. Instructions for these

cases involved having the state director or other office staff or supervisory enumerator contact 

the operation in person and providing an enumerator incentive if the supervisory enumerator was 

able to get a response. The response rates for this subgroup of operations are shown in Table 4.

The Fischer Exact Test was also used to determine if the proportion of nonresponse was different 

for the treatment group compared to the comparison group for the highly likely nonrespondent 

impact operations. We found no statistical difference in response rates between the treatment 

and comparison group (χ

2

(1, N =172) = 0.878647, p = 0.35), but we do see that for this 

subgroup, response rates are slightly higher (2.2% higher) in the treatment group. This will be 

discussed more in the next section. 

Table 4: Response Rates for the highly likely nonrespondent impact operations - comparison 

group and treatment group

Comparison Group Treatment Group

Completed 48.4% (n=44) 50.6% (n=41)

Refusal 37.4% (n=34) 43.2% (n=35)

Inaccessible 7.7% (n=7) 3.7% (n=3)

Office Hold 6.6% (n=6) 2.5% (n=2)

N 91 81

Within the treatment group, we examined the specialized data collection techniques recorded on 

the Supplemental Scoring Sheet to see if any of the specific techniques improved response rates.

Similar to 2011, within the treatment group, the use of a logo token item was the only strategy 

that was significantly related to response rates (χ

2

(1, N =424) = 5.07, p < 0.05). Approximately 

74 percent of operations that received a logo token item responded to ARMS. No other strategy 

is associated with a significant difference in response rates. 

3.1 Field office adherence with interviewing instructions

After data collection, we analyzed the data from the Supplemental Scoring Sheet and followed 

up with each field office to discuss their experience with the procedures. We found out that the 

procedures were not followed for many cases in the treatment group. For operations in Impact 
8

Group 3, procedures called for the follow up contact to be made by a state director/deputy 

director or supervisory enumerator, then assigned to an experienced or supervisory enumerator. 

From responses on the Scoring Supplement Sheet, 12% of these cases were sent to

directors/deputies/state statisticians for the first contact and 15% used supervisory enumerators

for the first contact, so 27% of these cases were initially handled as the procedures indicated. 

Enumerators assigned to these operations were often standard enumerators (not supervisory or 

refusal conversion). As noted above, the treatment group response rate for this subgroup of 

operations was 2.2% higher than the control group. Had more of the operations been handled by 

the directors and state statisticians, the response rate may have been even better for the treatment 

group, but given the results, there is no way to evaluate that in this study.

For those operations in Impact Group 2, procedures called for the follow up contact to be made 

in person by a supervisory enumerator. These cases could be handled completely by that 

enumerator or assigned to a different enumerator with the best refusal conversion techniques. We 

found that 11% of these cases were contacted by a supervisory enumerator and only one refusal 

conversion enumerator was used (out of 160 cases). Although procedures called for these 

operations to be targeted by a supervisory or refusal conversion enumerator, most of these 

operations were contacted by standard enumerators. 

For those operations in Impact Group 1, we asked enumerators to make their first contact in 

person and either collect the data or set up an appointment. Enumerators could be assigned to 

these cases as they normally would, and they were to do the follow-up in person. In 96% of these 

cases, enumerators visited in person (mostly standard enumerators but 11 supervisory 

enumerators and 3 refusal conversion enumerators). This group had the most buy-in for our 

instructions, most likely because it reflects the normal way data collection is handled for ARMS. 

Overall, the proposed experimental procedures were likely not fully implemented for a variety of 

reasons. During the time of the study, NASS was undergoing a field office reorganization with

many employees transitioning from state offices to regional offices. Therefore, staff were not 

available to work on ARMS III in general, and to make additional survey contacts specifically. 

Also, field offices were reluctant to change their data collection strategies during the production 

survey, particularly for operations they have worked with in the past. In addition, although RDD 

staff worked extensively with the headquarters survey team, they had much less communication 

with the field staff who ultimately was tasked with carrying out the procedures. 

For operations in Impact Groups 2 and 3, the operations that are more important to calibration 

targets, procedures were implemented as planned only part of the time. This makes our results 

difficult to analyze because we cannot compare the treatment and control groups if they 

essentially used the same procedures for both groups. It also demonstrates the difficulty of 

implementing an experiment in a production setting. As researchers, we need to recognize the 

importance of working more closely with the FO staff and field enumerators who are ultimately 

responsible for carrying out the procedures in any study.
9

3.2 Enumerator Incentives

During the ARMS 2012 data collection period, field offices were asked to offer incentives to 

enumerators for each record they completed from any of the impact groups in the treatment 

sample. Enumerators were to be told of this incentive before the data collection period. Only 

those states that had records in the treatment group were given the money to provide the 

incentives.

Only about half of the states used the incentives as planned; twelve offices used the incentives as 

planned while fourteen did not. In most of the states that did not use the incentive, office staff

and supervisory enumerators said that they did not want to recognize individuals, but wanted to 

recognize the group effort of the survey data collection.

Since only half of all field offices used the enumerator incentives, we cannot statistically test the 

comparison of overall response rates for the treatment and control group across the whole 

experimental sample. However, we can provide some descriptive information that may inform 

NASS moving forward with the enumerator incentives. We can provide summary information 

about the response rates for the treatment and control groups, but we have to subdivide the 

sample by those offices that used the incentives and those that did not. In addition, because 

response rates differ by state, we cannot directly compare the actual response rates. Instead, we 

compare the difference between the response rates for the control and treatment groups for those 

states that used the incentives and those states that did not use them. Table 5 shows the 

differences in response rates for the treatment and control group for offices that used the 

incentives and those that did not. Since the sample was not selected in a way that differentiates 

field offices that use/do not use incentives, we cannot provide any statistical test. However, we 

can see that the difference in the response rates between the treatment and control groups for the 

states that used the incentive is higher than the difference between the treatment and control 

groups in the states that did not use the incentives. It could be that the use of the enumerator 

incentives did help achieve slightly higher response rates.

Table 5: Response rates for the states that used incentives and those that did not

States that did use incentives States that did not use incentives

Treatment 

group rate

Control 

group rate

Difference Treatment 

group rate

Control 

group rate

difference

Complete 56.4% 55.8% 0.6% 53.4% 55.2% -1.8%

Refusal 37.7% 38.3% -0.6% 36.8% 31.8% 5.0%

Inaccessible 5.6% 4.8% 0.8% 4.8% 5.2% -0.4%

Office hold 1.1% 1.1% 0% 4.1% 7.8% -3.7%

The use of enumerator incentives may have been inconsistent because NASS has not typically 

used enumerator incentives in this way in the past. As with the data collection procedures, RDD 

staff worked with the headquarters survey team to develop the procedures for the enumerator 

incentives, but had less direct contact with the field office staff who administered the incentives. 
10

Since we offered enumerators additional money to complete certain cases, we conducted a small 

quality control callback operation to confirm that these enumerators actually collected the survey 

data from respondents. We found no reason to expect falsification. The procedures used and 

results from the quality control callback operation are in Appendix D.

4. CONCLUSION AND RECOMMENDATIONS

This study illustrates that our nonresponse propensity models work well to identify likely 

nonrespondents for ARMS III. However, using the propensity scores to target data collection 

efforts to increase response has been challenging. 

During the study, field offices and enumerators did not consistently implement the procedures 

that were being tested. Therefore, statistically valid analysis could not always be done to 

determine how effective the data collection strategies were in increasing response. Without 

standard implementation, we cannot conduct experiments that result in a clean comparison of the 

procedures. Moreover, we cannot use statistical evidence to improve or enhance our data 

collection process if we cannot test our strategies using clean comparisons. 

This study reiterates that it is difficult to entice the likely nonrespondent operations to reply to 

our surveys. Without standard implementation of the procedures, results are difficult to interpret. 

We did see a 2.2% non-significant increase in response rates for the treatment group of highly 

likely nonrespondent impact operations, but no overall increase in response rates for the whole 

treatment group. We do not know if focusing on these highly likely nonrespondent operations 

would be beneficial, even if the procedures were always implemented. Instead of targeting these 

highly likely nonrespondent operations, perhaps RDD should focus on operations that are likely 

nonrespondents but have more of a 50/50 chance of responding to the survey. 

The only data collection strategy associated with a significant difference in response rates in both 

2011 and 2012 was providing a logo token item, although the relationship between the use of 

logo items and response may or may not be causal.

Implementing this experiment had unforeseen challenges in 2012, so the experiment will be 

conducted again in 2013 with a more limited number of field offices. This should allow us to 

work more closely with the offices, ensure they understand the objectives of the experiment, and 

have agreed to follow predefined procedures on using the scores and implementing the strategies 

to the best of their ability and providing explanations when they have deviated from the 

instructions. When we have this clear buy in, we can hopefully make the clean comparisons we 

need to assess the effectiveness of our strategies to get reluctant operations to respond to ARMS.
11

4.1 Recommendations

Based on the research done, the following are recommended: 

1. Continue using the response propensity scores as a tool to identify the operations that are 

least likely to respond to ARMS III.

2. Conduct studies that can be monitored more rigorously to determine if targeted 

procedures are effective at increasing response rates.

3. Consider using response propensity scores that identify operators who are less likely to 

respond, but not the LEAST likely to response. Maybe this group of respondents can be 

persuaded to respond using targeted methods.

4. Consider using token items as a routine component for the data collection strategy used 

for the ARMS III. Although NASS has not done a controlled experiment using token 

items, there are several studies in survey methods literature that show increases in 

response rates when they are used.
12

5. REFERENCES

Earp, M.S., & McCarthy, J.S. (2009). Using Respondent Prediction Models to Improve Efficiency 

of Incentive Allocation. In JSM Proceedings, American Association for Public Opinion 

Research. Hollywood, Florida: American Statistical Association.

Earp, M.S., & McCarthy, J.S. (2010). Who is Responsible for the Bias? Using Classification 

Trees to Identify Subgroups of Likely Nonrespondents and Assessing their Relationship 

to Key Survey Estimates Using Structural Equation Modeling. In JSM Proceedings, 

American Association for Public Opinion Research. Chicago, Illinois: American 

Statistical Association. 

Earp, M.S, McCarthy, J.S., Shauer, N.D., & Kott, P.S. (2008). Assessing the Effect of 

Calibration on Nonresponse Bias in the 2005 ARMS Phase III Sample Using Census

2002 Data, Appendix A. Research and Development Division Staff Report RDD-08-01. 

United States Department of Agriculture, National Agricultural Statistics Service.

Earp, M.S, McCarthy, J.S., Shauer, N.D., & Kott, P.S. (2009). Assessing the Effect of 

Calibration on Nonresponse Bias in the 2006 ARMS Phase III Sample Using Census 

2002 Data. In JSM Proceedings, American Statistical Association. Washington, D.C: 

American Statistical Association.

Earp,M.S., McCarthy, J.S., Porter, E., & Kott, P.S. (2010). Assessing the Effect of Calibration

on Nonresponse Bias in the 2008 ARMS Phase III Sample Using Census 2007 Data. In 

JSM Preceedings, American Statistical Association. Vancouver, British Columbia: 

American Statistical Association.

Earp, M. and McCarthy, J. (2011). Using Nonresponse Propensity Scores to Improve Data 

Collection Methods and Reduce Nonresponse Bias. In JSM Proceedings, American 

Association of Public Opinion Research. Phoenix, AZ: American Statistical Association.

Earp, M., Mitchell, M., McCarthy, J., and Kreuter, F. (2012). Who is Responsible for the Bias? 

Using Proxy Data and Tree Modeling to Identify Likely Nonrespondents and Reduce 

Bias. In ICES IV Proceedings, International Conference on Establishment Surveys. 

Montreal, Quebec, Canada: American Statistical Association.

Earp, M., Mitchell, M., McCarthy, J. and Kreuter, F. (2014). Modeling Nonresponse in 

Establishment Surveys: Using an Ensemble Tree Model to Create Nonresponse 

Propensity Scores and Detect Potential Bias in an Agricultural Survey. Journal of Official 

Statistics, Vol 30 (4).
13

McCarthy, J.S., Beckler, D.G., and Ott, K. (2006). The Effect of Incentives on Response in 2005 

ARMS Phase III Interviews. Research and Development Division Staff Report RDD-06-

07, United States Department of Agriculture, National Agricultural Statistics Service.

McCarthy, J.S., & Jacob, T. (2009). Who Are You?: A Data Mining Approach to Predicting

Survey Non-respondents. In JSM Proceedings, American Association for Public 

Opinion Research. Hollywood, Florida: American Statistical Association. 

McCarthy, J.S., Jacob, T, & McCracken, A. (2010). Modeling Non-response in National 

Agricultural Statistics Service Surveys Using Classification Trees, Research and 

Development Division Research Report Number RDD-10-05, US Department of 

Agriculture, National Agricultural Statistics Service.

SAS Institute Inc., Enterprise Miner 6.2 Help and Documentation, Cary, NC: SAS Institute, 

2009.
A-1

APPENDIX A: IMPACT GROUP MEMO AND PROCEDURES

****************************************** OFFICIAL MAIL************************************

January 2, 2013

TO: State Directors

Deputy Directors

ARMS Phase III Statisticians

NASS Field Offices, Except Alaska & Hawaii

THROUGH: Barbara Rater

Chief

Survey Administration Branch

FROM: Curt Stock

Survey Administrator – ARMS Phase III

Environmental and Economic Surveys Section

PROJECT CODE: 904

38009904 = Field Office Accounting Code

DUE DATE: N / A

The labels for the RDD variables “Scoring = 21” and “Scoring = 23” are reversed on the Sample

Master compared to the documentation that has been previously released. Instead of updating

the Marked Sample Table in ELMO and putting out new Sample Masters, all variable labels

were updated in the documentation materials and records with “Scoring = 23” are now the “Most

Important Group” instead of “Scoring = 21”. The information following this paragraph is from the

memo on December 19, 2012. Updates to this memo and to the attached documentation

materials are in red.

The Data Collection Plan for the RDD Non-Response Propensity Scoring Project can be found

under “Impact Group Data Collection Procedures and Documentation” at the following link:

http://nassnet/csd_sab/docs/ARMS3/2012/2012_ARMS_III_RDD_index.html

The 441 records in this project were selected using an average non-response propensity score

of 70% (to be a non-respondent on ARMS Phase III) across all 140 “non-response trees” in

RDD’s model.

In short, there are 3 groups:

 Most important group for calibration targets (Scoring = 23, Records in group = 87)

 Mid importance group for calibration targets (Scoring = 22, Records in group = 185)

 Low importance group for calibration targets (Scoring = 21, Records in group = 169)

1) Scoring = 23:

 Mail a questionnaire on December 27, 2012.

 Initial in-person follow-up contact should be made by State Director, Deputy Director,

experienced office staff, or supervisory enumerator.

SUBJECT: 2012 ARMS Phase III – RDD Scoring

Variable ACTION: 907 = NASDA Incentive Project Code

Plans by Group:
A-2

o Field Office (FO) personnel can fill out a questionnaire or set stage for

an enumerator to contact operation to collect data at a later day.

o FO personnel are sent to explain how important the respondent’s operation is

to the survey.

 Supervisor or experienced enumerator is sent out if need be.

 Fill out Scoring Supplement.

2) Scoring = 22:

 Mail a questionnaire on December 27, 2012.

 Initial in-person follow-up contact should be made by a Supervisory Enumerator

or Experienced Enumerator.

 Fill out Scoring Supplement.

3) Scoring = 21:

 Mail a questionnaire on December 27, 2012.

 Initial in-person contact should be made by an enumerator like what is typically

done.

 Fill out Scoring Supplement.

 FO Personnel – The funds in the attached spreadsheet under tab “FO -

RDD Funds” should be used to focus on the 87 records in the high group (Scoring =

23). If FOs have money left over they should then focus on records in Scoring = 22 

and then on to Scoring = 21.

Use of Appropriate Accounting Code: Travel should be charged to the CSD

account under project code 9904. Travel arrangers should do a search for 9904 and

select Accounting Code 38009904.

 NASDA – The funds in the attached spreadsheet under tab “NASDA - RDD

Funds” should be used for enumerators only. Each enumerator would get $20

per usable report, if the record is in Scoring Group 21, 22, or 23.

If a State does not use all their funds, the money will roll over to the general ARMS

NASDA enumerator incentive fund. For example, Arizona has $160 for 8 records with

Scoring=21, 22, or 23. If enumerators complete 5 of these as usable reports and the

remaining 3 records came in the mail or were non-response, AZ would use $100 of

their allocated $160 for the RDD incentive. The remaining $60 will be added to the

general incentive fund. The attached spreadsheet under tab “Total NASDA Incentive

Funds” shows the breakout by Field Office for both general and RDD funds. Please

use project code 907 for both funds.

If you have any questions or concerns, please contact Curt Stock at (202) 720-3598.

********************************************************************************************************

Incentives:
A-3

2012 ARMS Phase III Data Collection Procedures

Kathy Ott and Melissa Mitchell, October 10, 2012

Assignment of Impact Group

During the ARMS III data collection, different procedures will be used for different operations 

based on their propensity to refuse and their importance to the calibration target for specific 

commodities. For this purpose, a sample of operations was selected for a treatment sample and a 

control sample, with an “Impact Group Code” assigned for each operation in those samples. 

Details on how the Impact Group Codes were assigned are in the Appendix.

Operations in the treatment sample in each of the impact groups will follow a specific follow up 

data collection methodology as outlined below. Field Offices can identify operations in the 

treatment group using the second digit of the Extract variable “x5” on the label. Operations in 

the control sample will not be marked for the field offices. Therefore, impact group assignment 

will not be used to determine data collection procedures for the control sample. Details on how 

the Impact Group Codes will be labeled for the field offices are in the Appendix.

For operations in ALL impact groups:

- All cases selected for all ARMS versions will be initially mailed a questionnaire to fill 

out and mail back or fill out through EDR. The procedures below describe the followup 

contacts. 

o Provide a data product, NASS logo item, ARMS report, or other item to the 

operator at the followup contact, if available in the office. 

o Explain the importance of their particular farm on the estimates for their 

commodity/size.

o Fill out the “ARMS Phase III Scoring Supplement” documenting what strategies 

you used.

Operations in Impact Group 3 – Most important for calibration targets (the variable 

“Scoring” =23; for information on the “Scoring” variable, see the Appendix)

In addition to items listed above,

- Office Hold: Do not assign these to Office Hold unless it is a dangerous situation.

- Initial In-Person Followup contact: The initial in-person followup contact should be 

made by the state director, deputy director, experienced office staff, or supervisory 
A-4

enumerator. This contact should be made in person if possible, with a telephone contact 

only if absolutely necessary. The director/deputy/office staff should bring a 

questionnaire with them, but the intent of this contact is not to collect data, but to set the 

stage for an enumerator to contact the operation to collect data. The questionnaire can be 

filled out; however, if the respondent wishes to at this time. If the supervisory 

enumerator makes this contact, they may collect data at this time or set up an 

appointment for data collection. During this followup contact, the director/deputy/office 

staff/supervisory enumerator should provide a data product, NASS logo item, ARMS 

report, or other item to the operator, provide information about the survey and relevant 

ARMS data, describe that this operation in particular is important to the survey and that is 

why they have been visited in person, and give the operator the name of the enumerator 

who will contact them if possible. (Talking points for the initial contact are given at the 

end of this document).

- Enumerator assignment: Assign these cases to a supervisory or experienced 

enumerator. The enumerator should contact the operation after the followup contact by 

the director, deputy, experienced office staff, or supervisory enumerator. No enumerator 

assignment is necessary if the director/deputy/office staff/supervisory enumerator 

collected all data items at the initial in-person followup contact.

- Data collection:

o Followup contact made as specified above.

o Collect data (even partial data) if they will provide it.

o Provide detailed notes about the data collection experience.

o Review all known information about an operation (size, type, special instructions, 

ELMO comments, etc.) to tailor data collection if needed.

o Fill out the Scoring Supplement.

Operations in Impact Group 2 – Mid-importance for calibration targets (“Scoring” =22)

In addition to items listed above in the “For Operations in ALL Impact Groups” section,

- Office Hold: Assign Office Holds as you normally would, using guidance from SAB.

- Respondent contact: Followup contact should be made in person by a supervisory 

enumerator, with a telephone contact used only if absolutely necessary. The supervisory 

enumerator may collect data at this time, set up an appointment for data collection at a 

later time, or set the stage for a non-supervisory enumerator to come at a later time.

(Talking points for the initial contact are given at the end of this document).

- Enumerator assignment: Assign these cases to a supervisory enumerator or a nonsupervisory enumerator with the best refusal conversion techniques.
A-5

- Data collection: 

o Follow other procedures as usual.

o Fill out the Scoring Supplement.

Operations in Impact Group 1 – Not as important for calibration targets (“Scoring” =21)

In addition to items listed above in the “For Operations in ALL Impact Groups” section,

- Office Hold: Assign Office Holds as you normally would, using guidance from SAB.

- Respondent contact: Enumerator should make their first contact in person to either 

collect the data or to set up an appointment at a later time. (Talking points for the initial 

contact are given at the end of this document).

- Enumerator assignment: Assign these cases to enumerators as you normally would, 

but have the enumerator make the followup contact in person.

- Data collection: 

o Follow other procedures as usual.

o Fill out the Scoring Supplement.

Talking points for in-person followup contact:

- Remind the operator that they received a questionnaire in the mail in late December or 

early January that combined this data collection with the mandatory 2012 Census of 

Agriculture. 

- Give a brief description of the ARMS survey and tell them that their operation was 

selected.

- Give information on why this particular operation is important and that you are visiting 

them specifically because of their importance.

- Give examples of how the data are used.

- Tell the operator that they can fill out the questionnaire and mail it back, fill it out 

through EDR, or you can have an enumerator set up an appt.

- Give the operator the name of the enumerator who will call, if possible.

- Ask if they have any questions.
A-6

Appendix 

Assignment and Labeling of Impact Group Codes

Assignment of Impact Group Codes

There were two steps in the assignment of the Impact Group for the 2012 ARMS III. 

1. Nonresponse Propensity Score Flags

RDD (Melissa Mitchell) assigned Nonresponse Propensity Score flags to the entire 

ARMS Phase III sample using the classification tree models. For ARMS Phase III, 140 

classification trees were grown using Census of Agriculture variables. For each tree, a 

propensity was calculated for each operation which indicates their likelihood of being a 

nonrespondent. Operations that had an average propensity of .70 across all 140 trees 

were flagged to signify that they were highly likely to be a nonrespondent.

The operations that were identified by the models as the most likely to be nonrespondents 

were then divided into either the control sample or the treatment sample. Using all 

ARMS sample from all versions, 887 operations with an average nonresponse propensity 

score of .70 across all trees were flagged, with 441 in the treatment sample and 446 in the 

control sample.

2. Impact Group Score

The treatment and control sample cases were then sent on to SMB (Eric Porter/Suzette 

Qualey). SMB divided those operations in the control and treatment samples into 3 

“impact groups”, based on their importance to the calibration targets, by state. The 

control sample was divided into impact groups for analysis purposes only. 

To make the assignments, SMB downloaded the control data that matched each 

calibration target for the records in the ARMS sample. Then, point assignments were 

made using the following criteria:

a. A record received 1 point for having any positive value for each calibration target or 

for value of sales. For example, if an operation had 5 acres of corn and 10 cattle, it 

automatically received 2 points, one for each item of interest.

b. If a record was in the top 10% of production or inventory within their state (from the 

approximately 887 records in the control and treatment groups) for any particular 

item, that record received 3 points.

c. If a record was in the top 10% of sales within their state, they would also receive 3 

points.
A-7

The points across all states ranged from 1 – 18.

Finally, using the point totals, the records were split into 3 groups: 1 = Impact Group 1, 2 

= Impact Group 2, and 3 = Impact Group 3. Since Group 3 records require the most 

intensive follow-up work, our target for Group 3 was to have less than 20 operations 

selected for each state to make this task manageable. The splits resulted in the following 

breakdowns:

Group 1: Points >= 7 

Group 2: 4 – 6 points

Group 3: 0 – 3 points

Labeling of Impact Group Codes

Records identified for this study are identified using the Sample Master variable “Scoring”. 

Values for “Scoring” are “blank”, 11, 12, 13, 21, 22, or 23 based on which group they were 

placed in. The first digit indicates whether the operation is in the control (1x) or treatment group 

(2x) and the second digit indicates which impact group they were assigned to. 

Sample Design Section included the “Scoring” variable on the Sample Master for all values, but 

only values 21, 22, and 23 will show up on the markedSample table (all others will be blank) 

under the x5 variable. When labels are produced, the x5 variable will be pulled from the 

markedSample table for the labels. Therefore, operations in the control group will not be marked 

on the labels.
B-1

APPENDIX B: COA SPLITTING VARIABLES

Rank Variable Name

1 Total Sales Not Under Production Contract (NUPC) 

2 Total Value of Products Sold + Government Payments 

3 Total Production Expenses 

4 The Number of Hired Workers Employed More than 150 Days 

5 Machinery and Equipment Value in Dollars 

6 Acres of Cropland Harvested 

7 Cropland Acres 

8 Total Reported Acres of Crops Harvested 

9 Acres of Land Owned 

10 State 

11 Total Acres Operated 

12 The Number of Hired Workers Employed Less Than 150 Days 

13 Any Migrant Workers Y/N 

14 Total Cattle and Calf Inventory 

15 Total Expenditures 

16 Farm Type Code 

17 Type of Organization

18 Percent of Principle Operator's Income from the Farm Operation 

19 Computer Used for the Farm Business Y/N 3 

20 Acres of All Other Land 

21 Principal Occupation of Principle Operator is Farming Y/N 

22 Total Government Payments 

23 ARMS III Production Region (Atlantic, South, Midwest, Plains, or West) 

24 Acres of Land Rented from Others 

25 Any Hired Manager Y/N 

26 Operation had Internet Access Y/N 

27 Number of Households Sharing in Net Farm Income 

28 Acres of all Irrigated Hay and Forage Harvested 

29 Number of Days Principle Operator Worked off Farm 

30 Total Fruit Acres 

31 Total Acres of Vegetables 

32 Acres of Woodland Pasture 

33 Principal Operator's Age 

34 Acres of Woodland Not in Pasture 

35 Number of Operators 

36 Acres on Which Manure Was Applied 

37 Acres of Permanent Pasture & Rangeland 

38 Acres of all Hay and Forage Harvested 

39 Total Poultry Inventory 

 

3 Only asked on the 2002 Census of Agriculture.
B-2

40 Partnership Registered Under State Law Y/N 

41 Acres of Cropland Used for Pasture 

42 Total Hog and Pig Inventory 

43 Principal Operator Lives on Operation Y/N 

44 Percent of Operators that are Women

45 Acres of Cropland for Which All Crops Failed 

46 Acres of Cropland in Summer Fallow 

47 ARMS III Questionnaire Version

48 Total Sales Under Production Contract (UPC) 

49 Total Citrus Acres 

50 Nursery Indicator Y/N 

51 Principal Operator's Sex 

52 Principal Operator – Race, Black 

53 Acres of Land Rented to Others 

54 Operation Farm Tenure (1=full owner, 2=part owner, or 3=tenant) 

55 Number of Persons Living in Principle Operator's Household 

56 Acres of Cropland Idle or Used for Cover Crops 

57 Have other farm Y/N 

58 Principal Operator – Race, White 

59 Sheep and Lamb Indicator Y/N 

60 Year Principal Operator Began this Operation 

61 Number of Women Operators 

62 Other Livestock Animals 

63 Agriculture on Indian Reservations Y/N 

64 Principal Operator – Race, American Indian 

65 Acres of Christmas Trees and Short Rotation Woody Crops 

66 Acres of Certified Organic Farming 

67 Possible duplicate Y/N 

68 Principal Operator is of Spanish Origin Y/N 

69 Principal Operator – Race, Asian 

70 Aquaculture Indicator Y/N 

71 Principal Operator – Race, Native Hawaiian or Pacific Islander 4

 

4 Not significant at the 0.20 level.
C-1

APPENDIX C: SCORING SUPPLEMENT SHEET
D-1

APPENDIX D: CONCLUSIONS FROM THE ARMS CALLBACKS

Conducted and written by Miriam Thorne, July 2013

Background

As part of 2012 ARMS data collection, enumerator incentives were offered to enumerators who 

completed records in the treatment sample in any of the three impact groups. Enumerators were 

given a bonus of twenty dollars for every flagged operation in the treatment group for which they 

obtained a response. To confirm that these enumerators actually collected the survey data from 

respondents—and did not fill in the data themselves—we called back a sample of ARMS 

respondents to verify that an enumerator showed up and conducted the survey with them. We 

chose a simple random sample of 40 individuals who had responded to ARMS and whose 

interviewers were offered the monetary incentive. Staff at headquarters made all of the callbacks 

in May and June 2013, 2-4 months after the initial survey, and tried to call each respondent 5 

times. Of the 40 respondents in our sample, we were only able to reach and speak with 21 

individuals. We used the standard Quality Control Worksheet that asks respondents about the 

enumerator’s visit and a few questions from ARMS data to cross-check the data recorded during 

the interview. The Worksheet is 2 pages long and is shown in Figure 1.

Fraud

Almost all respondents (95%) clearly remembered an enumerator coming to conduct the survey.

There was one respondent who did not remember an enumerator coming for the ARMS survey 

but he said that he responded to many surveys and could not pinpoint this one survey and 

enumerator. Most (86%) confirmed that the enumerator had verified their name and address and 

those who did not say that the enumerator had done so simply did not remember; none of the 21 

respondents said that the enumerator did not show up or did not ask those questions. Given this, 

there is no reason to believe that enumerators filled in the survey themselves or that the data was 

falsified.

Data Quality

After reviewing the callback results, there may be reason to be concerned about errors in the 

ARMS data. Close to half of the respondents (43%) gave at least one answer in the callback

interview that did not match the answers recorded during ARMS. In some cases, respondents 

seemed unsure of the answer they gave in the callback, asking us to clarify terminology or 

simply saying that they were not certain of their answer. But in other cases, respondents seemed 

very sure of their answer, even when it was different from the answers recorded in ARMS.

It seems most likely that respondents gave different answers during the callbacks and ARMS 

because the questions were not clear, leading respondents to interpret the question differently at 

different times. During the ARMS interview, the questions are in context and the enumerator 

can explain the questions, but in the very short callback, there is not the same context or 
D-2

explanation given. 38% of respondents expressed confusion with at least one question during the 

callbacks. Respondents seemed most confused by the questions about marketing contracts and 

production contracts (4b and 4c). Numerous respondents asked for definitions of these terms 

and, even with the definitions, were still unsure about what the terms meant and whether they 

had the contracts. Some respondents were also confused about what land we were asking them 

to measure in question 4a. It is possible that some answers from the callbacks do not match the 

answers from ARMS because respondents included different types of land in their calculation

during the callback and during ARMS. It may be worth clarifying these questions to reduce 

confusion and errors in the data.
D-3

Figure 1, page 1
D-4

Figure 1, page 2