Targeted Data 

Collection Efforts for the 

2013 ARMS III 

Melissa Mitchell

Kathy Ott

Heather Ridolfo

Jaki McCarthy

United States 

Department of

Agriculture

National

Agricultural

Statistics

Service

Research and

Development Division

Washington DC 20250

RDD Research Report

Number RDD-15-02

May 2015

This report has been prepared for limited distribution to the research community outside the U.S. Department 

of Agriculture (USDA). The views expressed herein are not necessarily those of NASS or USDA.

i

EXECUTIVE SUMMARY

In 2012, the Research and Development Division (RDD) of National Agricultural Statistics 

Service (NASS) conducted a study examining targeted data collection techniques for the 

Agricultural Resource Management Survey Phase III (ARMS III) for likely nonrespondent 

operations identified by classification trees. Due to a variety of reasons, the study was not 

conducted as anticipated in 2012. RDD conducted the study again in 2013 using a subset of 

states to assess whether specific data collection techniques can be implemented and whether they 

improve response rates. 

Likely nonrespondent operations were identified using classification trees and a measurement of 

the impact the operation has on the calibration targets for ARMS III, so that targeted data 

collection could be used to increase response rates for those operations. The targeted data 

collection techniques included providing enumerator incentives and using supervisory 

enumerators and/or regional office directors and State Statisticians to make in-person contacts 

with highly likely nonrespondents. 

Because this was largely an exploratory study, our sample sizes are too small to demonstrate 

statistical differences in response rates. Overall, we found a meaningful difference in response 

rates (7%) for the treatment group above the control group, although it was not statistically 

significant. Working with the smaller number of states, we were able to obtain much needed 

buy-in for the study, something that was missing from the 2012 study. 

ii

RECOMMENDATIONS

1. Continue to use the nonresponse propensity scores to identify and target likely 

nonrespondent operations in ARMS III.

2. Continue to use the impact scores to identify records that impact calibration targets for 

ARMS III so that targeted data collection strategies can be used. 

3. Continue the use of enumerator incentives for likely nonrespondents to ARMS III, but 

also consider using varying amounts for the incentives based on the level of nonresponse.

4. Conduct a controlled, large scale study, with consistent use of the same targeted package 

of data collection procedures and enumerator incentives to determine if they increase 

response rates. This was attempted in 2012, but results were confounded by inconsistent 

implementation. 
iii

TABLE OF CONTENTS

1. INTRODUCTION ...................................................................................................................1

1.1 Background...................................................................................1

1.2 Classification Tree Models..................................................................2

2. METHODS..............................................................................................................................3

3. RESULTS................................................................................................................................5

3.1 Field office Adherence with Interview Instructions.....................................5

3.2 Enumerator Incentives.......................................................................6

4. CONCLUSIONS AND RECOMMENDATIONS..................................................................7

4.1 Recommendations..............................................................................8

5. REFERENCES ........................................................................................................................9

APPENDIX A: Targeted procedures for 2013 ARMS............................................................... A-1

APPENDIX B: Census of Agriculture Splitting Variables .........................................................B-1

APPENDIX C: Scoring Supplement Sheet .................................................................................C-1
1

Targeted Data Collection Efforts for the 2013 ARMS III

Melissa Mitchell1, Kathy Ott2, Heather Ridolfo, & Jaki McCarthy3

Abstract

In 2013, the Research and Development Division of the National Agricultural 

Statistics Service conducted a study to determine if targeted data collection for 

likely nonrespondents could positively impact response rates for the Agricultural 

Resource Management Survey (ARMS) Phase III. The methodology for the study 

was the same as a study done in 2012, but was implemented with only four 

participating regions, encompassing 12 states. This smaller sample was used in 

2013 due to the unforeseen complications with carrying out the 2012 study across 

the United States. Likely nonrespondents were flagged using classification trees 

and their relative importance to the calibration target was taken into consideration 

with the use of impact scores. Flagged records were then assigned to targeted data 

collection methods including the use of enumerator incentives and having state or 

regional office staff make in person contacts to attempt to increase response rates 

for those records. Given our small sample size, this is an exploratory study. 

Key Words: Nonresponse propensity scores, impact scores, data mining, targeted data 

collection, ARMS, calibration target

1. INTRODUCTION

1.1 Background

Nonresponse is a major issue for the Agricultural Management Survey (ARMS) conducted by 

the National Agricultural Statistics Service (NASS) in collaboration with the USDA’s Economic 

Research Service (ERS). ARMS is conducted in three phases, but this research focuses only on 

the third phase. The third phase of ARMS (otherwise known as ARMS III) focuses on farm 

production and financial information. The ARMS III survey has traditionally been personally 

enumerated, however, beginning in 2012 and continued in 2013, it was mailed out, with personal

visit followup for nonrespondents.

Response rates for ARMS III have been low over time (except during years ending in 2 and 7 

when ARMS is collected with the mandatory Census of Agriculture), typically falling below the 

Office of Management and Budget (OMB) standard response rate of 80% (see Table 1). There 

have been several research endeavors to increase response rates and reduce bias that include 

 

1 Melissa Mitchell was a Mathematical Statistician with the National Agricultural Statistics Service, Research and 

Development Division, 3251 Old Lee Highway, Room 305, Fairfax, VA 22030. 

2 Kathy Ott is a Mathematical Statistician with the National Agricultural Statistics Service, Methodology Division, 

1400 Independence Ave SW, Washington DC. 

3 Heather Ridolfo is a Survey Statistician and Jaki McCarthy is the Senior Cognitive Research Methodologist with 

the National Agricultural Statistics Service, Research and Development Division, 3251 Old Lee Highway, Room 

305, Fairfax, VA 22030. 
2

offering incentives and calibrating to known targets (McCarthy, Beckler, & Ott, 2006; Earp, 

McCarthy, Schauer, & Kott, 2008; Earp, McCarthy, Schauer, & Kott, 2009; Earp, McCarthy, 

Porter, & Kott, 2010). Recently, work has focused on proactively identifying nonrespondents 

(Earp, Mitchell, McCarthy, & Kreuter, 2014; Earp, Mitchell, McCarthy, & Kreuter, 2012; Earp 

& McCarthy, 2011; McCarthy, Jacob, & McCracken, 2010; McCarthy & Jacob, 2009) in order 

to target data collection methods for those cases.

Table 1: ARMS III Response Rates 2000-2012

ARMS III 

Sample Year

ARMS III 

Sample Size (n)

ARMS III 

Respondents (nr)

ARMS III 

Nonrespondents 

(nn)

ARMS III 

response rates

2000 17,903 11,295 6,608 63.1%

2001 13,313 8,500 4,813 63.8%

2002 18,219 13,484 4,735 74.0%

2003 33,861 21,278 12,583 62.8%

2004 33,908 22,966 10,942 67.7%

2005 34,937 24,704 10,233 70.7%

2006 34,203 23,237 10,966 67.9%

2007 31,924 22,304 9,620 69.9%

2008 36,388 24,066 12,322 66.1%

2009 33,348 22,753 10,595 68.2%

2010 35,431 23,285 12,146 65.7%

2011 34,070 22,130 11,940 65.0%

2012 32,096 23,492 8,604 73.2%

Total 389,601 263,494 126,107 67.6%

This study was done in response to the results of a previous ARMS study (Mitchell, Ott, and 

McCarthy, 2015). In that study, the treatment condition procedures were not followed as RDD 

anticipated. Therefore, in 2013, RDD attempted the 2012 study again with a smaller group of 

states. Procedures are outlined below, but are identical to the 2012 study, with an added 

emphasis on making the first follow-up contact in person.

1.2 Classification Tree Models

The method NASS used to target likely nonrespondent operations is an ensemble, or set, of 

classification trees. Classification (or decision) trees are used to predict the outcome of a binary 

variable, such as survey response/nonresponse, from auxiliary data. The primary objective of 

classification trees is classification of groups (in our case respondent/nonrespondent operations). 

For the classification trees developed for predicting ARMS nonresponse, the auxiliary data used 

(which is available for both ARMS respondents and nonrespondents) were variables from the 

2002 and 2007 Census of Agriculture. 

These classification trees are created by segmenting a dataset by a set of simple rules. The rules 

assign an observation to a segment based on the input variable that maximizes the difference 

between two groups based on the target (in this case survey nonresponse). The dataset is 

sequentially split into subsets by these rules until no more splits can be created. No more splits 
3

can be created when the sample size per segment is too small, no more significant splits can be 

created, or the maximum depth of the tree is too large.

Classification trees create a hierarchy (tree) where the segments are called nodes. The first node, 

known as the root node, contains the entire dataset. From the root node, there are branches which 

are paths to and from nodes within a tree. Terminal nodes are nodes that have no branches 

coming from them and are known as leaves. Each record will appear in only one of the leaves 

and the leaves will collectively contain all the records in the dataset. The leaves of interest are 

those that have the highest proportion of records with the target, in this case, nonresponse.

In addition to using classification trees to identify likely nonrespondent operations, NASS also 

used impact scores to identify influential operations within our likely nonrespondent group. 

Impact groups are related to calibration targets and are created by determining how important an 

operation is to meeting the calibration targets for particular data items. Operations are given a 

rank score from 1 to 3 depending on how important they are to the calibration target. Operations 

in Impact Group 3 are the most important to the calibration targets, those in Impact Group 2 are 

of mid-importance to calibration targets, and those in Impact Group 1 are not as important to 

calibration targets. Only likely nonrespondent operations were given an impact group ranking. 

Assignment to impact groups was based on a point system where the number of points are based 

on the following criteria: one point for each commodity for which the operation is in the top 10% 

of production or inventory within their state, one point for each commodity for which the 

operation is in the top 10% of sales within their state, and one point for each calibration target for 

which an operation has any positive value. See Appendix A for the official procedures regarding 

the creation of impact groups.

2. METHODS

When using classification trees, usually a single tree is created with the best initial split. 

However, an ensemble of trees was created for ARMS III, for a variety of reasons. In the single 

tree framework, the tree is created with the best initial split based on a portion of the data 

(training data), but there is no guarantee that is the best split based on the entire dataset. Also, the 

initial split directly affects the subsequent splits, so although it may be a good initial split, it may 

not identify the greatest number of operations in the target. It is possible that a split that is not 

initially the most optimal will provide better subsequent splits. By growing multiple trees (an 

ensemble) we have a richer understanding of likely nonrespondents that will possibly bias our 

key estimates. 

Therefore, every variable was forced to be used as an initial split in one tree for this study. There 

were 71 variables in the dataset that could serve as the initial split and 70 of those splits were 

statistically significant (p<0.20) (see Appendix B). These 70 variables were the same 70 that 

were significant in the 2012 study. The 70 statistically significant initial splits were used 

separately in the model to grow 70 different trees. Therefore, each variable was considered when 

assessing characteristics of nonrespondents. All variables were considered for subsequent splits 

in the tree and those splits were determined by the splitting algorithm used by SAS Enterprise 

Miner (SAS, 2009). A total of 140 trees were created using the 70 variables from the Census of 

Agriculture (COA), 70 from the 2002 COA and 70 from the 2007 COA. 
4

In 2013, a threshold of 0.70 was chosen to act as a cut-off to distinguish between likely 

nonrespondents and all other operators. Therefore, all leaves (also called terminal nodes) with 

seventy percent nonresponse rate or higher were selected from each tree. We flagged operations 

whose average nonresponse propensity was greater than or equal to 0.70 across all trees. These 

most likely nonrespondent operations were flagged for targeted data collection methods. 

Four regional offices, covering 12 states, were selected to participate in the study. The regional 

offices that participated were the Northwest Region (Washington, Oregon, Idaho), Great Lakes 

Region (Michigan, Indiana, Ohio), Northern Plains Region (Nebraska, Kansas, North Dakota, 

South Dakota), and Pacific Region (California/Nevada). When planning this experiment, RDD 

recognized that the sample would be too small to make any statistically significant claims about 

our treatment conditions. However, given the difficulty of the 2012 experiment, RDD wanted to 

conduct a small scale experiment with a few regions to determine if the procedures could be 

implemented in a production environment. 

In this study, 464 operations within these 12 states were flagged as likely nonrespondent 

operations across all questionnaire versions. These 464 flagged operations were randomly split 

into a treatment group (n=230) and comparison group (n=234) after stratifying by state and farm 

type. 

For 2013, as in 2012, all operations in the treatment and control group were mailed a 

questionnaire as the first survey contact. All mail nonrespondent records from both groups were 

sent to the state offices to be enumerated by field enumerators. Field enumerators were asked to 

make each followup visit in person (not on the phone) to gain response. Field office personnel 

completed a scoring supplement sheet for each case in the treatment group. The scoring 

supplement contained questions about the data collection techniques that were used for each 

operation and was used to assess the effectiveness of the treatment options. State offices could 

indicate multiple techniques on the sheet. See Appendix C for the Scoring Supplement Sheet 

that was used in 2013. 

For records in the treatment group, specific follow-up contact techniques were used based on the 

impact group assignment. For records in impact group 3 (those highly likely nonrespondent 

records most important to calibration targets), no records were held out of data collection (unless 

there was a dangerous situation), and the follow-up contact was done in person by the director, 

deputy, office staff, or supervisory enumerator to set the stage for a supervisory enumerator to 

collect the survey data. For records in impact group 2, the follow-up contact was made by a 

supervisory enumerator in person to collect the data or set the stage for an experienced 

enumerator to collect the data. For records in impact group 1, an enumerator was sent to the 

operation to either collect the data or set up an appointment to collect the data. See Appendix A

for the official procedures for the 2013 data collection.

Also similar to 2012, in 2013, the package of targeted follow-up contact techniques also included 

enumerator incentives for enumerators who completed records in the treatment sample in any of 

the three impact groups. Enumerators were given a bonus of twenty dollars for every flagged 

operation in the treatment group for which they obtained a response.
5

3. RESULTS

Comparing the response rates from the treatment and control group, there was not a significant 

difference in response rate (χ

2

(1,N=464)=2.76, p=.10). However, the difference in response rates 

is in the expected direction with over 7 percent more completions in the treatment group (see 

Table 2).

Other numbers to note are the office hold and inaccessible percentages. The instructions 

specifically asked for states to only hold the treatment records in the office if there were safety 

concerns (often, records are held in the office for other reasons). The results show that they 

followed the procedures as there are no office holds in the treatment group. Also, for the 

treatment group, we asked that states go in person to make the first contact with a potential 

respondent, instead of calling to set up an appointment. In a short data collection period, this 

may lead to increased inaccessible rates, but the data collection window for ARMS is several 

months and enumerators could call if they could not make contact in person, so any effect would 

be expected to be small. The inaccessible rate is higher for the treatment group, but only

slightly.

Table 2: Response Rates for the Control and Treatment Groups

Control Group Treatment Group

Complete 40.2% 47.8%

Refusal 52.6% 45.2%

Inaccessible 5.5% 7.0%

Office Hold 1.7% 0.0%

N 234 230

Although it is important to assess specific treatments in order to best use limited resources, the 

small sample size did not allow us to make any statistically valid conclusions. Enumerators and 

state offices were not limited to a certain number or combination of treatment options to try, and 

we did not have big enough sample size to test all possible combinations. There may be a 

combination that works best, but we were unable to identify that combination in this study.

3.1 Field Office Adherence with Interview Instructions

Overall, for the treatment sample, offices followed the instructions about who should contact a 

targeted operation for 60% of the cases. Compared to 2012, this study had much more 

compliance. 

Impact Group 3 contained the operations that were highly likely nonrespondents with the most 

impact on calibration targets. There were 25 operations in this group and offices were able to 

comply with the instructions in 68% of the cases. For 40% of the cases, a director/deputy/state 

statistician contacted the operation in addition to a supervisory enumerator. For 12% of the cases, 

only a director/deputy/state statistician contacted the operation. In another 16% of these cases, 

only a supervisory enumerator contacted the operation. The first follow-up was completed in 

person 64% of the time. 
6

Operations in Impact Group 2 were highly likely to be nonrespondents and of mid-importance to 

calibration targets. There were 58 operations in this group and offices were able to follow the 

instructions regarding who should make contact with an operation in 31% of cases. The 

instructions specified the use of a supervisory or experienced (or refusal conversion) enumerator, 

but for most cases, RFOs used a standard field enumerator. Fifty percent of the first contacts 

made for these operations were done in person. It may be too difficult logistically for RFOs and 

State Statisticians to assign supervisory enumerators to this many individual cases.

Operations in Impact Group 1 were those operations that were highly likely to be nonrespondents 

and were the least likely to impact calibration targets. This group had the largest subset of the 

treatment group (N=147). The instructions given for this group closely mirrored current 

procedures carried out by the regions. In this group, offices were able to follow the instructions

regarding who should contact the operations 100% of the time. As stated above, the instructions 

provided for this group mirror the current data collection strategy employed in the RFOs, likely 

the main reason for the high compliance for this group. However, the first follow-up was only 

completed in person 42% of the time. 

3.2 Enumerator Incentives

For all of the operations in the treatment group, the instructions called for enumerator incentives 

($20) to be provided to any enumerator that completed a report for these likely nonrespondent

operations. The intention of the enumerator incentives was to emphasize the importance of these 

operations and increase enumerator emphasis on encouraging cooperation from likely 

nonrespondents. The treatment and control group in this experiment were a small part of the 

overall sample. Interviewers knew which cases were in the treatment group, but could not 

distinguish between cases in the control group and the cases that were not included in study. In 

2013, all states in all of the regions (Northwest Region, Great Lakes Region, Pacific Region, and 

Northern Plains Region) that participated in the study used the enumerator incentives except for 

North Dakota (in the Northern Plains region). North Dakota only had two operations in the 

treatment group. 

All of the enumerators were informed of the incentive prior to data collection. Ten incentives 

were given in the Northwest Region (6 in Washington, 1 in Oregon and 3 in Idaho). In the Great 

Lakes Region, 11 were given (0 in Ohio (only operation eligible for incentive was mail refusal), 

4 in Indiana (out of 9), and 7 were given in Michigan (out of 14)). In the Pacific Region, 56 

incentives were given. Finally, in the Northern Plains region, there were 10 incentives given (1 in 

Kansas, 6 in Nebraska, and 3 South Dakota). 

HQ recommended that any leftover funds be put towards the general incentive fund. There were 

also some additional modifications of how the incentives were used in the regions. In the 

Northwest Region, they used the incentive funds to give separate awards or incentives to 

enumerators that were able to turn an incomplete or refusal mail response into a completed 

interview. In the Pacific Region, they allocated the incentive money relative to the impact score. 

For operations with an impact score of 1 (denoted 21) – a $10 incentive was given, for operations 

with an impact score of 2 (denoted 22) – a $25 incentive was given and for operations with an 

impact score of 3 (denoted 23) – a $50 incentive was given. 
7

Unlike 2012 when only half of the offices used the enumerator incentives, in 2013 the incentive 

money was allocated as recommended and enumerator incentives were given. These incentives 

could have motivated enumerators to work harder for those cases deemed very difficult to obtain 

a response, helping lead to the 7% increase in response we saw in the treatment group. Moving 

forward, enumerator incentives could be used for these hard to get cases. It may make more 

sense to use a tiered system where operations with an impact score of 3 would have a higher 

incentive amount linked to it than impact group 2 or 1 as was done in the Pacific Region as 

opposed to a flat rate of $20, but that will have to be evaluated and established by Survey 

Administration Branch (SAB).

4. CONCLUSION AND RECOMMENDATIONS

This study was an exploratory study on data collection techniques for ARMS III because sample 

sizes were too small to evaluate the treatment. Compared to the study in 2012, this study in 2013 

had better compliance by the participating regional and field offices. This experience should 

allow us to move forward with a larger national study, similar to 2012, but with better 

compliance across all offices. However, not all regions may be able to devote the time to 

implement in-person interviews by office management and staff for an experiment. If the study 

is conducted with a larger and more representative sample i.e., the entire sample of likely 

nonrespondents, RDD and SAB may be able to outline the best data collection techniques that 

should be used in ARMS III. 

Although the sample size was too small to show statistical significance, we saw a 7% increase in 

the response rate for the treatment group in the four participating regions. These regions used 

targeted data collection methods including using supervisory enumerators and office staff for 

first follow-up contacts and enumerator incentives. This is promising and could indicate that 

when used consistently, in-person visits by high level staff to highly likely nonrespondent impact 

operations and enumerator incentives could increase response rates across all states for the 

ARMS Phase III. 
8

4.1 Recommendations

Based on the research done, the following are recommended:

1. Continue to use the nonresponse propensity scores to identify and target likely 

nonrespondent operations in ARMS III.

2. Continue to use the impact scores to identify records that impact calibration targets for 

ARMS III so that targeted data collection strategies can be used. 

3. Continue the use of enumerator incentives for likely nonrespondents to ARMS III, but 

also consider using varying amounts for the incentives based on the level of nonresponse.

4. Conduct a controlled, large scale study, with consistent use of the same targeted package 

of data collection procedures and enumerator incentives to determine if they increase 

response rates. This was attempted in 2012, but results were confounded by inconsistent 

implementation. 
9

5. REFERENCES

Earp, M.S, McCarthy, J.S., Shauer, N.D., & Kott, P.S. (2008). Assessing the Effect of 

Calibration on Nonresponse Bias in the 2005 ARMS Phase III Sample Using Census

2002 Data, Appendix A. Research and Development Division Staff Report RDD-08-01. 

United States Department of Agriculture, National Agricultural Statistics Service.

Earp, M.S, McCarthy, J.S., Shauer, N.D., & Kott, P.S. (2009). Assessing the Effect of 

Calibration on Nonresponse Bias in the 2006 ARMS Phase III Sample Using Census 

2002 Data. In JSM Proceedings, American Statistical Association. Washington, D.C: 

American Statistical Association.

Earp,M.S., McCarthy, J.S., Porter, E., & Kott, P.S. (2010). Assessing the Effect of Calibration

on Nonresponse Bias in the 2008 ARMS Phase III Sample Using Census 2007 Data. In 

JSM Preceedings, American Statistical Association. Vancouver, British Columbia: 

American Statistical Association.

Earp, M. and McCarthy, J. (2011). Using Nonresponse Propensity Scores to Improve Data 

Collection Methods and Reduce Nonresponse Bias. In JSM Proceedings, American 

Association of Public Opinion Research. Phoenix, AZ: American Statistical Association.

Earp, M, Mitchell, M., McCarthy, J., & Kreuter, F. (2012) Who is Responsible for the Bias? 

Using Proxy Data and Tree Modeling to Identify Likely Nonrespondents. In ICES IV 

Proceedings, American Statistical Association. Montreal, Quebec, Canada: American 

Statistical Association.

Earp, M., Mitchell, M., McCarthy, J. and Kreuter, F. (2014). Modeling Nonresponse in 

Establishment Surveys: Using an Ensemble Tree Model to Create Nonresponse 

Propensity Scores and Detect Potential Bias in an Agricultural Survey. Journal of Official 

Statistics, Vol 30 (4).

McCarthy, J.S., Beckler, D.G., and Ott, K. (2006). The Effect of Incentives on Response in 2005 

ARMS Phase III Interviews. Research and Development Division Staff Report RDD-06-

07, United States Department of Agriculture, National Agricultural Statistics Service.

McCarthy, J.S., & Jacob, T. (2009). Who Are You?: A Data Mining Approach to Predicting

Survey Non-respondents. In JSM Proceedings, American Association for Public 

Opinion Research. Hollywood, Florida: American Statistical Association. 

McCarthy, J.S., Jacob, T, & McCracken, A. (2010). Modeling Non-response in National 

Agricultural Statistics Service Surveys Using Classification Trees, Research and 

Development Division Research Report Number RDD-10-05, US Department of 

Agriculture, National Agricultural Statistics Service.
10

Mitchell, M., Ott, K., & McCarthy, J. (2015). Targeted Data Collection Efforts for the 2012 

ARMS III. Research and Development Division Research Report Number RDD-15-01, 

US Department of Agriculture, National Agricultural Statistics Service. 

SAS Institute Inc., Enterprise Miner 6.2 Help and Documentation, Cary, NC: SAS Institute, 

2009.
A-1

APPENDIX A: TARGETED PROCEDURES FOR 2013 ARMS

2013 ARMS Phase III Data Collection Procedures Experiment

Kathy Ott and Melissa Mitchell, January 2014

A. Background/Introduction 

Similar to the 2012 ARMS, during the 2013 ARMS III data collection, slightly different data 

collection procedures will be used for operations based on their propensity to be a nonrespondent on the ARMS survey and their importance to the calibration target for specific 

commodities. The main differences in the data collection methodology are the mode of the first 

data collection follow-up contact, the person making that contact, and the use of enumerator 

incentives for the operations that are the highly likely to be non-respondents. To analyze the 

impact of the procedures, a sample of operations that are identified as highly likely to be nonrespondents was selected for a treatment sample and a control sample, with an “Impact Group 

Code” assigned for each operation in those samples. Response rates, non-response rates, and 

cost for the two groups will be analyzed to determine if the procedures improve response and at 

what cost.

Unlike 2012 ARMS, only selected ROs will participate in the study. Four regional offices, 

covering 12 states will be asked to participate in this study – Northwest Region 

(Washington/Oregon/Idaho), Great Lakes Region (Michigan/Indiana/Ohio), Northern Plains 

Region (Nebraska/Kansas/North Dakota/South Dakota), and Pacific Region (California/Nevada). 

The participating states agree to use the data collection methods described below and to 

document what they do for each case in the treatment sample.

Operations in the treatment sample in each of the impact groups will follow a specific follow up 

data collection methodology as outlined below. Regional and Field Offices can identify 

operations in the treatment group using the “Scoring” variable from the sample master. A 2013 

ARMS—Phase III Scoring Supplement is required for all records with a scoring indicator of 21, 

22 or 23. The scoring variable and POID from the sample master should be recorded on the 

scoring supplement and matched with the appropriate ARMS III record to determine the required 

follow-up strategy and to document the actual follow-up methods used. Treatment group records 

submitted to the SPS edit without the Scoring Supplement will trigger a critical error. Details on 

how the Impact Group Codes were assigned are in the Appendix.

B. Data Collection Procedures

All cases selected for all ARMS versions will be initially mailed a questionnaire in late 

December to fill out and mail back or fill out through EDR. Non-respondents will be 

mailed a second questionnaire 3-4 weeks later. The procedures below describe the 

follow-up contacts to those mailings for the treatment sample. 
A-2

Procedures are identified for different groups of operations based on their impact group 

and previous notes about the operation. The main differences in the procedures are the 

mode of the first follow-up contact and the person making the contact. 

1. For operations in the treatment sample:

An enumerator incentive will be provided to the enumerator for each operation that is 

identified as highly likely to be a non-respondent, that is, all operations in the treatment 

sample (Scoring = 21, 22, or 23). The incentive will be given just for those cases where 

the initial follow-up contact is made by an enumerator. If the case is mailed in, or if the 

initial follow-up contact is made by the RO director, deputy director, state statistician or 

other office person, then the incentive money for that case can be used in the general 

enumerator incentive pool.

Operations in Impact Group 3 – Operations in the treatment group that are highly likely to be 

nonrespondents and have the most impact on calibration targets (the variable “Scoring” =23; for 

information on the “Scoring” variable, see the Appendix).

- Office Hold: Do not assign these to Office Hold unless it is a dangerous situation.

- Initial In-Person Followup contact: 

o This contact should be made IN PERSON by the regional director, deputy 

director, state statistician, survey coordinator, or supervisory enumerator. Do 

NOT call ahead to make an appointment. Try to reach the operation at least two 

times in person before calling.

 If the supervisory enumerator makes this contact, they may collect data at 

this time or set up an appointment for data collection. 

 If a non-enumerator makes the contact, they should bring a questionnaire 

with them, but the intent of this contact is not to collect data, but to set the 

stage for a supervisory enumerator to contact the operation to collect data. 

The questionnaire can be filled out, however, if the respondent wishes to 

at this time. 

o Whoever collects the data should:

 Provide a data product, ARMS report, or other token item to the operator, 

 Provide information about the survey and relevant ARMS data, 

 Describe that this operation in particular is important to the survey and 

that is why they have been visited in person, and 

 Give the operator the name of the enumerator who will contact them if 

applicable. 

- Enumerator assignment: 

o Assign these cases to a supervisory or very experienced enumerator to contact 

after the in-person follow-up contact by the regional director, deputy director, 

state statistician, survey coordinator, or supervisory enumerator. 

- Data collection:
A-3

o Provide detailed notes about the data collection experience.

o Before making contact, review all known information about an operation (size, 

type, special instructions, ELMO comments, etc.) to tailor data collection if 

needed.

o Fill out the Scoring Supplement.

Operations in Impact Group 2 – Operations in the treatment group that are highly likely to be 

nonrespondents and have medium impact on calibration targets (“Scoring” =22).

- Office Hold: Assign Office Holds as you normally would, using guidance from SAB.

- Initial In-Person Followup contact: 

o This contact should be made in person by a supervisory enumerator, with a 

telephone contact used only if necessary. 

o The supervisory enumerator may collect data at this time, set up an appointment 

for data collection at a later time, or set the stage for a non-supervisory 

enumerator to come at a later time.

o The supervisory enumerator should:

 Provide a data product, ARMS report, or other token item to the operator, 

 Provide information about the survey and relevant ARMS data, 

 Describe that this operation in particular is important to the survey and 

that is why they have been visited in person. 

- Enumerator assignment: Assign these cases to a supervisory enumerator or a nonsupervisory enumerator with the best refusal conversion techniques.

- Data collection: 

o Follow other procedures as usual.

o Fill out the Scoring Supplement.

Operations in Impact Group 1 – Operations in the treatment group that are highly likely to be 

nonrespondents and have a low impact on calibration targets (“Scoring” =21).

- Office Hold: Assign Office Holds as you normally would, using guidance from SAB.

- Initial In-Person Followup contact: 

o This contact should be made in person by an enumerator as normally assigned, 

with a telephone contact used if necessary. 

o The enumerator should provide information about the survey and relevant ARMS 

data. 

- Enumerator assignment: Assign these cases to enumerators as you normally would.

-

- Data collection: 
A-4

o Follow other procedures as usual.

o Fill out the Scoring Supplement.

Talking points for in-person follow-up contact:

- Remind the operator that they received a questionnaire in the mail in late December 2013 

or early January 2014 and another one in late January 2014. 

- Tell the operator that they can fill out the questionnaire on paper to mail back, through 

EDR, or with an enumerator.

- If data are not collected at the initial visit, give the operator the name of the enumerator 

who will conduct the actual interview, if possible.

- Give a brief description of the ARMS survey and selection of their operation.

- Give information on why this particular operation is important (it has a particular 

commodity, is a certain size, etc.) and that you are visiting them in person because of 

their importance.

- Give examples of how the data are used.

- Ask if they have any questions.

C. For operations that you have previous data collection agreements

Conduct data collection according to your previous agreement. For operations in the 

treatment sample, note this on the Scoring Supplement.

D. For all other operations in the ARMS sample:

- Office Hold: Assign Office Holds as you normally would, using guidance from SAB.

- Initial In-Person Followup contact: 

o This contact should be made by an enumerator as normally assigned. A telephone 

contact can be used to set up an appointment. 

o The enumerator should provide information about the survey and relevant ARMS 

data. 

- Enumerator assignment: Assign these cases to enumerators as you normally would.

-

- Data collection: 

o Follow other procedures as usual.
A-5

Assignment and Labeling of Impact Group Codes

Assignment of Impact Group Codes

There were two steps in the assignment of the Impact Group for the 2013 ARMS III. 

1. Nonresponse Propensity Score Flags

RDD assigned Nonresponse Propensity Score flags to the entire ARMS Phase III sample 

using the classification tree models. For ARMS Phase III, 140 classification trees were 

grown using Census of Agriculture variables. For each tree, a propensity was calculated 

for each operation which indicates their likelihood of being a nonrespondent. Operations 

that had an average propensity of .70 across all 140 trees were flagged to signify that they 

were highly likely to be a nonrespondent.

The operations that were identified by the models as the most likely to be nonrespondents 

were then divided into either the control sample or the treatment sample. Using all 

ARMS sample from all versions, 803 operations with an average nonresponse propensity 

score of .70 across all trees were flagged, with 395 in the treatment sample and 408 in the 

control sample. For the 12 states participating in the study, there are 450 flagged 

operations with 223 in the treatment sample and 227 in the control sample.

2. Impact Group Score

The treatment and control sample cases were then sent on to SMB. SMB divided those 

operations in the control and treatment samples into 3 “impact groups”, based on their 

importance to the calibration targets, by state. The control sample was divided into 

impact groups for analysis purposes only. 

To make the assignments, SMB downloaded the control data that matched each 

calibration target for the records in the ARMS sample. Then, point assignments were 

made using the following criteria:

a. A record received 1 point for having any positive value for each calibration target or 

for value of sales. For example, if an operation had 5 acres of corn and 10 cattle, it 

automatically received 2 points, one for each item of interest.

b. If a record was in the top 10% of production or inventory within their state (from the 

approximately 803 records in the control and treatment groups) for any particular 

item, that record received 3 points.

c. If a record was in the top 10% of sales within their state, they would also receive 3 

points.

The points across all states ranged from 1 – 30.
A-6

Finally, using the point totals, the records were split into 3 groups: 1 = Impact Group 1, 2 

= Impact Group 2, and 3 = Impact Group 3. Since Group 3 records require the most 

intensive follow-up work, our target for Group 3 was to have less than 20 operations 

selected for each state to make this task manageable. The splits resulted in the following 

breakdowns:

Group 3: Points >= 7 

Group 2: 4 – 6 points

Group 1: 0 – 3 points
B-1

APPENDIX B: COA SPLITTING VARIABLES

Rank Variable Name

1 Total Sales Not Under Production Contract (NUPC) 

2 Total Value of Products Sold + Government Payments 

3 Total Production Expenses 

4 The Number of Hired Workers Employed More than 150 Days 

5 Machinery and Equipment Value in Dollars 

6 Acres of Cropland Harvested 

7 Cropland Acres 

8 Total Reported Acres of Crops Harvested 

9 Acres of Land Owned 

10 State 

11 Total Acres Operated 

12 The Number of Hired Workers Employed Less Than 150 Days 

13 Any Migrant Workers Y/N 

14 Total Cattle and Calf Inventory 

15 Total Expenditures 

16 Farm Type Code 

17 Type of Organization

18 Percent of Principle Operator's Income from the Farm Operation 

19 Computer Used for the Farm Business Y/N 4 

20 Acres of All Other Land 

21 Principal Occupation of Principle Operator is Farming Y/N 

22 Total Government Payments 

23 ARMS III Production Region (Atlantic, South, Midwest, Plains, or West) 

24 Acres of Land Rented from Others 

25 Any Hired Manager Y/N 

26 Operation had Internet Access Y/N 

27 Number of Households Sharing in Net Farm Income 

28 Acres of all Irrigated Hay and Forage Harvested 

29 Number of Days Principle Operator Worked off Farm 

30 Total Fruit Acres 

31 Total Acres of Vegetables 

32 Acres of Woodland Pasture 

33 Principal Operator's Age 

34 Acres of Woodland Not in Pasture 

35 Number of Operators 

36 Acres on Which Manure Was Applied 

37 Acres of Permanent Pasture & Rangeland 

38 Acres of all Hay and Forage Harvested 

39 Total Poultry Inventory 

40 Partnership Registered Under State Law Y/N 

 

4 Only asked on the 2002 Census of Agriculture.
B-2

41 Acres of Cropland Used for Pasture 

42 Total Hog and Pig Inventory 

43 Principal Operator Lives on Operation Y/N 

44 Percent of Operators that are Women

45 Acres of Cropland for Which All Crops Failed 

46 Acres of Cropland in Summer Fallow 

47 ARMS III Questionnaire Version

48 Total Sales Under Production Contract (UPC) 

49 Total Citrus Acres 

50 Nursery Indicator Y/N 

51 Principal Operator's Sex 

52 Principal Operator – Race, Black 

53 Acres of Land Rented to Others 

54 Operation Farm Tenure (1=full owner, 2=part owner, or 3=tenant) 

55 Number of Persons Living in Principle Operator's Household 

56 Acres of Cropland Idle or Used for Cover Crops 

57 Have other farm Y/N 

58 Principal Operator – Race, White 

59 Sheep and Lamb Indicator Y/N 

60 Year Principal Operator Began this Operation 

61 Number of Women Operators 

62 Other Livestock Animals 

63 Agriculture on Indian Reservations Y/N 

64 Principal Operator – Race, American Indian 

65 Acres of Christmas Trees and Short Rotation Woody Crops 

66 Acres of Certified Organic Farming 

67 Possible duplicate Y/N 

68 Principal Operator is of Spanish Origin Y/N 

69 Principal Operator – Race, Asian 

70 Aquaculture Indicator Y/N 

71 Principal Operator – Race, Native Hawaiian or Pacific Islander 5

 

5 Not significant at the 0.20 level.
C-1

APPENDIX C: Scoring Supplement Sheet