United States 

 Department of 

 Agriculture

 

 National

 Agricultural 

 Statistics 

 Service

 Research and 

 Development Division

 Washington DC 20250

RDD Research Report 

Number RDD-08-08 

September 2008

Categorizing 

Nonresponse in Phase 

III of the 2006 

Agricultural Resource 

Management Survey in 

Washington State 

Michael W. Gerling 

HoaiNam N. Tran 

Morgan S. Earp 

This paper was prepared for limited distribution to the research community outside the United States 

Department of Agriculture. The views expressed herein are not necessarily those of the National 

Agricultural Statistics Service or of the United States Department of Agriculture. 

i 

EXECUTIVE SUMMARY 

The National Agricultural Statistics Service (NASS) surveys farmers and ranchers across the 

United States and Puerto Rico to estimate crops and livestock, assess production practices, and 

identify economic trends. The Agricultural Resource Management Survey (ARMS) collects data 

covering chemical use and production practices with a specific focus on the financial well being 

of agricultural operations. ARMS is composed of three phases. Phase I is conducted May 

through July and screens for potential inclusion for Phases II and III. Phase II is conducted 

October through December and collects data on cropping practices and chemical usage. Phase 

III occurs February through April of the following year and collects detailed economic 

information about the agricultural operation and the operator’s household. ARMS data are used 

by farm organizations, commodity groups, agribusiness, Congress, state departments of 

agriculture, and the United States Department of Agriculture (USDA). The USDA uses ARMS 

data to evaluate the financial well being of farms and ranches and to objectively evaluate critical 

issues related to agriculture and the rural economies. 

ARMS Phase III (ARMS III) is the only phase with response rates generally lower than 80 

percent, and thus the potential for nonresponse bias for this phase is higher. To improve ARMS 

III response rates, NASS’ Research and Development Division and NASS’ Washington Field 

Office decided to conduct a trial study examining the reasons behind nonresponse in 

Washington. 

The data collection methods used in ARMS III were a combination of personal interview and 

mail. Some operations received only a field visit, while other operations received a field visit 

only if the operator failed to return his / her questionnaire in the mail. Field enumerators asked 

sample operators who had declined to cooperate on the survey to explain why they refused to 

complete the ARMS III questionnaire. The reasons provided were recorded and classified using 

an updated listing of refusal reasons originally created in a previous NASS study, Identifying and 

Classifying Reasons for Nonresponse on the 1991 Farm Costs and Returns Survey (O’Connor, 

1992). Field enumerators also recorded the reasons for identifying sample operations as 

inaccessible. 
ii 

Table 1 below displays the primary reasons for nonresponse. The authors recommend that 

nonresponse research be expanded to additional surveys and states to determine if reasons for 

nonresponse are survey specific and / or state specific. This will also assist in targeting those 

areas of the data collection process that need improvement. 

Table 1: Top Three Reasons for Nonresponse (Refusal & Inaccessible) 

 Reasons for Refusal Reasons for Inaccessible 

1. Would not take time / too busy. 1. 

Tried several times; could not reach 

anyone for an appointment. Just an 

extremely busy person. 

2. Will do other surveys, but not 

financial surveys. 2. No respondent, as listed on the label, could be found. 

3. Information too personal / none of 

your business. 3. Respondent postponed the interview beyond the end of the survey period. 
iii 

RECOMMENDATIONS 

1. Continue researching the reasons for nonresponse in Washington to 

identify and study any trends that are occurring. 

2. Gradually expand nonresponse research to all surveys and states. This 

will improve NASS’ understanding of survey specific, state, regional, and 

national trends. 

3. Apply the lessons learned to future nonresponse studies. Specifically, 

 implement the following: 

a. Designate an office use box on the back page of the questionnaire 

for recording the nonresponse code. 

b. Add an edit check that triggers a warning if the questionnaire is 

coded refusal, inaccessible or incomplete and is missing a 

nonresponse reason code. 

4. As NASS’ knowledge of nonresponse grows, improve current training 

scenarios to address the major reasons for refusals and inaccessibles. 
iv 
1 

 

Addressing Nonresponse in Phase III of the 

Agricultural Resource Management Survey 

in Washington State 

Michael W. Gerling, HoaiNam N. Tran, Morgan S. Earp 1,2

Abstract 

The National Agricultural Statistics Service (NASS) surveys farmers and ranchers

in the fifty states and Puerto Rico in order to estimate crops and livestock, assess 

production practices, and identify economic trends. One of NASS’ annual 

surveys is the Agricultural Resource Management Survey (ARMS). ARMS 

collects data on chemical usage, production practices, and the financials / 

economics of agricultural operations. 

ARMS is composed of three phases. Of the three, Phase III is the only one with

response rates generally lower than 80 percent, the Office of Management and 

Budget’s (OMB) threshold for assessing the impact of nonresponse bias. The low 

response rate for this phase increases the probability of biased results. 

To understand and thereby improve their ARMS III response rate, NASS’ 

Washington Field Office and NASS’ Research and Development Division, 

conducted a study assessing the reasons for nonresponse. The results of this study 

are summarized in this report. 

Key Words: Nonresponse, Response Rate, Data Collection, Agriculture 

1 Michael W. Gerling, HoaiNam N. Tran, and Morgan S. Earp (Mathematical Statisticians) are 

 with the National Agricultural Statistics Service-Research & Development Division, located at 

 Room 305, 3251 Old Lee Highway, Fairfax, VA 22030. 

2 Special thanks to NASS’ Washington Field Office (especially Joe Parsons, Dave Losh and 

 Matthew Pautler), Washington Field Enumerators, Rich Hopper of NASS’ Survey 

 Administration Branch, and Terry O’Connor of NASS’ Research and Development Division

for making this research possible. 
2 

1. INTRODUCTION

The mission of the United States 

Department of Agriculture’s (USDA) 

National Agricultural Statistics Service 

(NASS) is to provide timely, accurate, and 

useful statistics on United States and Puerto 

Rico agriculture. NASS annually conducts 

hundreds of surveys to estimate crops and 

livestock, measure production practices, and 

identify economic trends. 

The Agricultural Resource Management 

Survey (ARMS) is a voluntary, multipurpose national survey conducted annually 

on targeted commodities that vary from 

year-to-year. ARMS collects economic 

information used for assessing the financial 

well-being of the United States’ agricultural 

sector. ARMS is also the primary vehicle 

by which NASS obtains data on production 

practices and chemical use for these targeted 

commodities. 

 

ARMS data are used by farm organizations, 

commodity groups, agribusiness, Congress, 

state departments of agriculture, and the 

USDA. The USDA uses ARMS data to 

objectively evaluate government policies 

and other critical issues related to agriculture 

and the rural economy. 

The ARMS is composed of three phases. 

Phase I, conducted May through July, serves 

as a screener for the later two phases, 

ensuring that operations are in business and 

determining whether they are currently 

producing the targeted commodities. Phase 

II, conducted October through December, 

focuses on chemical usage and production 

practices for target commodities. Phase III, 

conducted February through April of the 

following year, assesses the financial 

condition of the farm sector. This phase 

collects data on income, expenses, assets, 

debt, and operator characteristics. 

1.1 Nonresponse Problem 

Over time, the response rate for ARMS has 

been decreasing. Monetary incentives are 

now being employed to boost response rates. 

However, monetary incentives add 

significantly to the cost of data collection 

and are ultimately limited in terms of their 

effectiveness. Therefore, additional research 

into the reasons for nonresponse is needed. 

In general, survey nonresponse negatively 

impacts the quality of the survey data and 

estimates, and increases survey costs. 

Nonresponse in ARMS III affects NASS’ 

income / expense estimates, increases data 

collection time, and significantly 

complicates the data editing process. 

Nonresponse also introduces bias that can 

not easily be explained / identified. 

Therefore, this study focuses on 

understanding nonresponse to pinpoint those 

area(s) of the data collection process 

needing improvement. 

In ARMS III, there are three types of 

nonresponse: refusals, inaccessibles, and 

incompletes. 

1.) Refusals are operators who were 

actually contacted for data 

collection but refused to respond. 

2.) Inaccessibles are operators who 

were not available and thus were 

not contacted for data collection. 

3.) Incompletes occur when the 

respondent is unable or unwilling 
3 

to answer all of the questions on 

the questionnaire. 

1.2 Purpose 

The goal of the study was to gain insight 

into the most common reasons for 

nonresponse occurring in Washington State 

for the 2006 ARMS III. Data collection 

occurred February through April of 2007. 

1.3 Definitions 

Additional terms used throughout the report 

are defined below: 

Usable: Reports with complete data. 

Out of Operation was not operating

Business: during the survey reference 

period. 

Office Hold: Questionnaire was held in the 

 office and not enumerated. 

Non-Farm: Operation failed to meet 

 USDA’s definition of a farm. 

2. METHOD 

In 2006, Washington State had 1,506 

agricultural operations sampled for ARMS 

III. The operations were selected based on 

income, commodities, and acreage. 

Washington utilized two different 

questionnaires for ARMS III. The Cost and 

Returns Report (CRR) is the field 

enumerated version and was used to collect 

data on 484 operations through personal 

interviews. The CRR sample received presurvey letters informing the operators that a 

field representative would be contacting 

them to schedule an interview. Nine CRR 

sampled operations were excluded from the 

study since they did not meet the definition 

of a farm or were held in the office for other 

reasons. 

The Core Version is a mailed version of the 

field questionnaire, which was mailed to 

1,022 operations. Each operator receiving 

this version also received a pre-survey letter 

stating the importance of their participation. 

Of the 1,022 operations, 914 received a 

monetary incentive [$20 Automated Teller 

Machine (ATM) card]. Ninety-nine 

operations did not receive any incentive. 

These 99 operations were used as a control 

group in a separate study focusing on the 

effectiveness of incentives on response rate. 

Also, 9 Core sampled operations did not 

meet the definition of a farm or were held in 

the office for other reasons. These nine 

were excluded from this study.

2.1 Enumerator Training 

Survey workshops are conducted annually to 

train field enumerators on ARMS III data 

collection procedures. The Washington 

field enumerators were provided with the 

following instructions on how to handle 

nonresponse: 

For refusals, the field enumerators were 

instructed to ask and record the operator’s 

reasoning for not participating in the survey. 

In the case of an inaccessible, the field 

enumerators were asked to document why 

the operator could not be contacted. 

For incompletes, the field enumerators were 
4 

to record why the operator did not answer 

specific questions. 

In all three cases, field enumerators were 

instructed to review a supplemental handout 

listing various nonresponse reasons. Each 

reason had its own unique corresponding 

numerical code. The field enumerator would 

then record the appropriate code in the 

designated Office Use Box on the 

questionnaire. Appendix A contains a copy 

of the supplemental handout. If a reason did 

not have a corresponding nonresponse code, 

the staff in the WA FO would allocate a new 

code number for that nonresponse reason. 

An additional 30 minutes was added to the 

overall field enumerator workshop on 

recording reasons for nonresponse. All 30 

enumerators participated in this additional 

training. 

2.2 Project Costs 

Additional field enumerator training costs 

were not incurred since the additional 

nonresponse training was absorbed into the 

general survey workshop. No additional 

burden fell on the field enumerators since 

documenting the reasons for the 

nonresponse is a requirement for all surveys. 

3. RESULTS 

The research study classified the reasons for 

nonresponse in 2006 ARMS III for 

Washington State. 

3.1 Overall Response Rates 

As shown in Table 2, Washington’s 63.6 

percent response rate falls below the Office 

of Management and Budget’s (OMB) 

threshold level of 80 percent. 

Twenty-seven percent of respondents 

refused to complete ARMS III and 4.7 

percent were deemed inaccessible. 

Table 2: Overall Response Counts and Rates

1,506 in 

Sample Usable Out of Business Refusal Inaccessible Incomplete Office Hold 

NonFarm 

Frequency 957 42 414 71 4 11 7 

Percent1/ 63.6 2.8 27.5 4.7 <0.3 0.7 <0.5 

1/ Totals may be over / under 100% due to rounding. 
5 

Table 3 summarizes the number of 

questionnaires that were usable, out of 

business, refusal, inaccessible, office hold, 

and non-farm by questionnaire version. 

A Chi-Square Test of Independence was 

conducted to compare the response rates of 

the Core and CRR samples. The test 

showed that the response rates are 

significantly different across the two groups 

(X2 =9.93, df = 3, and p-value < .05). This is 

primarily due to a large percentage of the 

Core version recipients receiving a monetary 

incentive with the questionnaire. 

Table 4, summarizes response rates by 

percentages across questionnaire versions. 

Table 3: Response Counts by Questionnaire Version (Number) 

Questionnaire 

Version Usable Out of 

Business Refusal Inaccessible Incomplete Office Hold 

NonFarm 

& 

Out 

of 

Scope 

Total 

CRR 286 11 157 19 2 6 3 484 

Core 671 31 257 52 2 5 4 1,022 

 X2 =9.93, df =3, p=0.02 

 

1/ The Chi-square test excluded "Out of Business", "Office Hold", and "Non-Farm - Out of Scope categories since 

these were not relevant to the study. Therefore, examining only "Usable", "Refusal", "Inaccessible", and 

"Incomplete" by questionnaire version (CRR and Core), the degree of freedom is calculated as (Number of response 

categories - 1) * (Questionnaire version - 1) = (4 - 1) * (2 - 1) = 3. 

Table 4: Response Rates by Questionnaire Version (Percent) 1/

Questionnaire 

Version Usable Out of 

Business Refusal Inaccessible Incomplete Office Hold 

Non-Farm 

& Out of 

Scope 

CRR 59.1 2.3 32.4 3.9 0.4 1.2 0.6 

Core 65.7 3.0 25.1 5.1 <0.2 <0.5 <0.4 

1/ Totals may be over / under 100% due to rounding. 
6 

3.2 Reasons for Refusal 

An operator can refuse the survey either by 

writing “refusal” on the mail questionnaire 

or by refusing to complete the questionnaire 

when contacted by the field enumerator. For 

all mailed refusals, Washington’s staff 

reviewed the questionnaire and assigned the 

most appropriate reason using the 

nonresponse coding handout. For field 

interview refusals, the field enumerator 

determined which nonresponse reason best 

described the situation and coded it on the 

questionnaire. 

The results are displayed in Table 5, located 

on the following page. The top three 

reasons for refusing were “Would not take 

time / too busy.”, “Will do other surveys, 

but not financial surveys.”, and 

“Information too personal / none of your 

business.” 

 
7 

Table 5: Reasons for Refusing to Participate in ARMS III (by descending frequency order) 

Frequency Percent Reason for Refusal 

85 22 Would not take the time / too busy. 

45 11 Will do other surveys, but not financial surveys. 

44 11 Information too personal / none of your business. 

24 6 Refused, but no reason given. 

23 6 Out of business now, will not answer for the previous year. 

18 5 Family illness / death.

17 4 Contact attempted, but respondent refuses on all surveys, and 

refused on this one.

14 4 Respondent only does compulsory surveys. 

14 4 “I do not like surveys / I do not do surveys.” 

14 4 “This is not a farm.” 

13 3 “My farm is too small to count / too small to be representative.”

12 3 “You contact me too often.” 

11 3 The respondent feels that surveys and reports hurt the farmer 

more than help. 

10 3 I did this survey before, but not again. 

7 2 Feels the survey items are too complex – too much recollection 

is involved. 

6 2 “I will have nothing to do with the Government.” 

5 1 Quitting farming. 

5 1 The respondent feels the operation’s records are inadequate to 

complete the interview. 

4 1 Would not keep appointments. 

4 1 Farm records are at the tax advisors / lawyers. 

3 <1 The operator called the office after receiving the pre-survey 

letter, and asked not to be contacted further. 

3 <1 Does not think the information is kept confidential. 

3 <1 Does not want to report due to legal / financial problems. 

2 <1 Does not want to talk about farming. 

2 <1 Spouse / secretary / etc. will not let the enumerator see the 

operator. 

1 <1 “I just did a different survey for your office.” 

1 <1 Known refusal, no contact attempted. 

1 <1 Violent / threatening refusal. 

1 <1 I just did a survey for someone else. 

1 <1 Would not answer the door even though they were home. 

21 

Not included 

in the percent 

breakdown. 

Missing Data (No reason provided on the questionnaire.) 

414 100 Total 
8 

In 1990 and 1991, reasons for refusals and 

inaccessibles were studied across multiple 

states in ARMS III, (O’Connor, 1991 and 

1992). In those years, ARMS was called the 

Farm Costs and Returns Survey. Table 6 

compares Washington’s top five reasons and 

where these ranked in previous studies. 

The top three reasons are ranked similarly 

across studies. However, there is variation 

in the ranking of the fourth and fifth reasons 

across the three studies. This may be 

attributed to examining only one state while 

past studies focused on multiple states. 

Additional research would need to be 

conducted to determine whether or not 

Washington’s refusal reasons are state 

specific. 

Table 6: Comparison Ranking of the Reasons for Refusal in Washington’s 2006 ARMS III 

 Study Versus Nonresponse Studies from 1991 and 1992 

2006 

Washington 

Ranking 

1991 Study 

Ranking 

1990 Study 

Ranking Reason for Refusal 

1 1 1 Would not take the time / too busy. 

2 6 27 Will do other surveys, but not financial 

surveys. 

3 3 3 Information too personal / none of your 

business. 

4 2 2 Refused, but no reason given. 

5 22 17 Out of business now, will not answer for 

the previous year. 
9 

3.3 Reasons for Inaccessibles 

If the field enumerator failed to contact / 

reach the operator to conduct a CRR 

interview, then the questionnaire would be 

coded as inaccessible. If a Core 

questionnaire receives no response via mail 

and the enumerator was unable to contact 

the operator, then the questionnaire was 

coded as inaccessible. 

According to Table 7, the three main reasons 

for inaccessible were: 1.) “Tried several 

times; could not reach anyone for an 

appointment.”, 2.) “No respondent, as 

listed on the label, could be found.”, and 3.) 

“Respondent postponed the interview 

beyond the end of the survey period.”

Table 7: Reasons for Inaccessibles in ARMS III (by descending frequency order) 

Number Percent Reasons for Inaccessible 

28 40 Tried several times; could not reach anyone for an 

appointment. Just an extremely busy person. 

17 24 No respondent, as listed on the label, could be found. 

7 10 Respondent postponed the interview beyond the end of the 

survey period. 

4 6 Illness / death in the family prevents the operator from 

responding.

3 4 Farm records are not available until after the survey period 

closes. 

3 4 No operation, as listed on the label could be found. 

3 4 The operator is away on an extended vacation. 

2 3 Non-English speaking respondent, interpreter not available. 

1 1 The operator is away on business. 

1 1 Access to the address on the label was denied by a gate / guard / 

etc. 

1 1 Inaccessible, but no reason given. 

1 

Not 

included in 

the percent. 

Missing Data (No reason provided on the questionnaire.) 

71 100 Total 
10 

As stated earlier, nonresponse studies for 

ARMS III have been conducted in the past. 

Table 8 shows Washington’s top three 

inaccessible reasons and their ranking 

compared to past studies. 

The results of the Washington study differ 

somewhat from the 1991 study and 

substantially from the 1990 study. In 1990, 

the top two reasons were “The operator is 

away on extended vacation” and “Illness / 

death prevents the operator from 

responding.” However, the earlier studies 

covered more states, which may account for 

these differences. 

Table 8: Comparison Ranking of the Reasons for Inaccessibles in Washington’s 2006 

ARMS III Study Versus Nonresponse Studies from 1991 and 1992 

2006 

Washington 

Ranking 

1991 Study 

Ranking 

1990 Study 

Ranking Reasons for Inaccessible or Incomplete 

1 3 1 

Tried several times; could not reach anyone 

for an appointment. Just an extremely busy 

person. 

2 5 9 No respondent, as listed on the label, could 

be found. 

3 6 5 Respondent postponed the interview 

beyond the end of the survey period. 

4. INCOMPLETES 

Incompletes occur when the respondent is 

unable or unwilling to answer all of the 

questions on the questionnaire. Field 

enumerators recorded four reports as 

incomplete. This is less than 0.3 percent of 

the total nonresponse. 

Table 9: Incompletes 

Number of Incompletes Percent of Total Nonresponse 

4 0.3 
11 

5. LIMITATIONS 

Analyzing the data uncovered 33 operations 

incorrectly coded as refusal instead of 

inaccessible. Also, two operations were 

incorrectly coded as inaccessible instead of 

refusal. There were also 22 operations 

where the enumerators failed to record a 

nonresponse reason code. 

6. RECOMMENDATIONS 

The authors recommend expanding the 

research of nonresponse to additional states 

and surveys. This will assist in determining 

if particular reasons for nonresponse are 

survey specific, regional, or national. 

Additionally, the authors recommend 

designating a cell on the questionnaire for 

recording the reason for nonresponse. This 

will standardize the process for future 

research. Also, the authors recommend 

adding an edit to the questionnaire editing 

process to check if there is a nonresponse 

reason code for those questionnaires coded 

as refusal or inaccessible. 

7. CONCLUSION 

Studying the reasons for nonresponse has 

provided the Washington Field Office with a 

starting point for addressing future data 

collection activities. As understanding of 

the underlying reasons for nonresponse 

grows, enumerator training scenarios on 

handling particular refusals and inacessibles 

can be improved. 

Finally, the Research and Development 

Division and the Washington Field Office, 

will continue working together to study 

nonresponse and how best to increase 

response rates. 

8. REFERENCES 

Beckler, D., J. McCarthy, K. Ott. (2006) The 

Effect of Incentives on Response in 2005 

ARMS Phase 3 Interviews, Research and 

Development Division Report RDD-06-06, 

United States Department of Agriculture, 

National Agricultural Statistics Service. 

Beckler, D., P. Horvath, K. Ott. (2005) 

Indirect Monetary Incentives for the 2004 

ARMS Phase 3 Core, Research and 

Development Division Report RDD-05-05, 

United States Department of Agriculture, 

National Agricultural Statistics Service. 

O’Connor, T., (1991) Identifying And 

Classifying Reasons For Nonresponse On 

The 1990 Farm Costs And Returns Survey, 

Research Applications and Development 

Division Report SRB-91-11, United States 

Department of Agriculture, National 

Agricultural Statistics Service. 

O’Connor, T., (1992) Identifying And 

Classifying Reasons For Nonresponse On 

The 1991 Farm Costs And Returns Survey, 

Research Applications and Development 

Division Report SRB-92-10, United States 

Department of Agriculture, National 

Agricultural Statistics Service. 

Ott, L., (1988) An Introduction To Statistical 

Methods And Data Analysis Third Edition, 

Boston, Massachusetts, PWS-Kent 

Publishing Company. 

Gerling, M., (2005) Using Handheld Global 

Positioning System Receivers for Phase II of 

the Agricultural Resource Management 
12 

Survey, Research and Development Division 

Report RDD-05-04, United States 

Department of Agriculture, National 

Agricultural Statistics Service.
13 

Appendix A 

(Reasons for Nonresponse Coding Handout)
14