ARMS Progress Report 

USDA’s National Agricultural Statistics Service 

and Economic Research Service 

Respond to Recommendations by the 

Agricultural Resource Management Survey Review Panel 

July 2015 

United States 

Department 

of 

Agriculture

National Agricultural 

Statistics Service

Economic 

Research

 Service

 

NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 3 

In 2008, the Committee on National Statistics (CNSTAT) of the National Research 

Council released the findings and recommendations of an independent review of 

USDA’s Agricultural Resource Management Survey in Understanding American 

Agriculture.

1

 The review was requested by the National Agricultural Statistics 

Service and the Economic Research Service as part of a program of continuous 

improvement for ARMS. 

Senior executives at the National Agricultural Statistics Service (NASS) and the 

Economic Research Service (ERS), two USDA agencies that jointly manage the 

Agricultural Resource Management Survey (ARMS), reviewed the recommendations 

and developed an implementation strategy based on a cost/benefit analysis. NASS 

and ERS continually reevaluate resources and priorities and will continue to respond 

to the recommendations as resources allow. 

The ERS/NASS ARMS Steering Committee was formed based on the numerous 

recommendations from this review. The committee, whose members are ARMS 

managers and specialists from both agencies, meets monthly to discuss survey 

issues and solutions. 

RECOMMENDATIONS and RESPONSES 

In the pages that follow, the review panel’s recommendations are presented as 

they appeared in the report along with actions NASS and ERS have taken in 

response as of February 2014. Updates will be issued at least annually, and more 

frequently as progress warrants, and will be posted to the “Independent Reviews” 

box on the NASS Surveys Web page. 

Data Integration and Relevance 

CNSTAT Recommendation 2.1: The Natural Resources Conservation Service 

[NRCS], NASS, and ERS should engage in a focused research and testing program 

and use experience with integrating the Conservation Effects Assessment Project 

and ARMS to assess the feasibility of integrating ARMS with other surveys and data 

sources. 

NASS/ERS Response: The team formed to examine ARMS/census of 

agriculture integration succeeded in improving the joint ARMS/census data 

set for 2012. After an extensive effort that looked at all data items common 

to ARMS and the census, the team proposed ways to align the concepts and 

questions asked on the two data collection instruments. The 

recommendations were accepted by NASS and ERS senior management, and 

 

1

 National Research Council (2008). Understanding American Agriculture: Challenges for the Agricultural Resources 

Management Survey. Panel to Review USDA’s Agricultural Resource Management Survey. Committee on National 

Statistics, Division of Behavioral Social Sciences and Education. Washington, DC: The National Academies Press. 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 4 

resulted in a shortened form for 2012. The link with the Conservation Effects 

Assessment Project was facilitated at the time by new funding from the 

Natural Resource Conservation Service (NRCS). The current work with NRCS 

on watersheds does not easily fit into the context of the ARMS program. 

Status: Completed. Integration with census was of high importance and 

has been completed for the 2012 Census of Agriculture. No other integration 

is relevant in the current NASS program. If additional funding is received for 

other survey programs that address ARMS-related content, NASS will pursue 

opportunities for survey integration.

CNSTAT Recommendation 2.2: In preparation for funds becoming available for a

longitudinal design of ARMS, ERS and NASS should systematically conduct research 

and explore the need for and feasibility of obtaining panel data from ARMS. 

Furthermore, as a test of the power of such information, more use should be made 

of the existing longitudinal microdata from the census of agriculture. One possible 

approach would be to create a pseudopanel of such data. Another would be to 

make a retrospective link between the census of agriculture and a year of ARMS. 

NASS/ERS Response: ERS has done some research on linking ARMS and 

census in the context of measuring structural change in lieu of pursuing a 

panel data collection. Additional work will be primarily done by ERS with 

access to NASS ARMS and census data. NASS recently obtained access to 

census of agriculture data prior to 1974; once NASS finishes converting them 

into a compatible file format, they will be available for ERS research in 

structural change. 

Examples of ERS research projects using linked census records and research 

projects that link ARMS and census of agriculture records include: 

Linked Census Records: 

T. Kirk White, B, Kirwan, and Y. Uchida. "Aggregate and Farm-level 

Productivity Growth in Tobacco: Before and After the Quota Buyout". Amer. 

J. Agricultural Economics, forthcoming (2012). 

Weber, Jeremy, and Nigel Key. “How Much Do Decoupled Payments Affect 

Production?” Amer. J. Agricultural Economics 94 (2012). 

Hoppe, Robert, J. MacDonald, and P. Korb. “Small Farms in the United 

States: Persistence Under Pressure.” USDA Economic Research Service 

Economic Information Bulletin No. EIB-63. 2010. 

O’Donoghue, Erik, Michael Roberts, and Nigel Key. “Did the Federal Crop 

Insurance Act Alter Farm Enterprise Diversification?” J. Agricultural 

Economics 60 (2009). 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 5 

Key, Nigel, and Michael Roberts. “Government Payments and Farm Business 

Survival.” Amer. J. Agricultural Economics 88 (2006). 

Linked ARMS and Earlier Census Records: 

Kirwan, Barrett. “The Incidence of Agricultural Subsidies on Farmland Rental 

Rates.” J. Political Economy 117 (2009). 

Status: Completed for post-1974 Census of Agriculture data. Linkage 

has been completed between the current ARMS sample and previous 

censuses of agriculture back through 1974. Linked data sets are available, 

with approved written agreements, for ongoing analyses by ERS and 

academic researchers using the NORC Data Enclave at the University of 

Chicago. Once census data prior to 1974 are available in a compatible file 

format, additional linkages will be made available to researchers. 

Survey Management 

CNSTAT Recommendation 3.1: The ARMS program should have structured 

mechanisms in place for stakeholder feedback and discussion on ARMS, beyond 

what is currently done, such as organized stakeholder forums, with some obligation 

to respond. Specifically, USDA should solicit input in developing the survey from 

stakeholders from within USDA and from other government agencies, universities, 

professional associations, and the private sector. 

NASS/ERS Response: An ARMS data users’ conference was held in 

conjunction with the February 2009 Ag Outlook Forum. A webinar was 

conducted in spring 2009 and a data users forum was held at the Agricultural 

and Applied Economics Association (AAEA) in August 2009. The NASS LongRange Planning Team requested input from data users in the agricultural 

community during 2009. Stakeholders provided significant input both before 

and after the chemical use component of ARMS was reinstated in summer 

2009. NASS continually seeks input from data users at various trade 

association meetings, often setting up forums at those meetings to discuss 

surveys relevant to the stakeholder group. Comments on the three phases of 

ARMS are also accepted at NASS annual Data Users’ Meetings.

The ARMS briefing room on the ERS Web site provides an opportunity for 

stakeholder feedback regarding data characteristics, use of the information 

for statistical purposes, and questionnaire content. ERS receives 40 to 60 

inquires annually from this facility. Feedback is systematically reviewed by 

the ERS/NASS ARMS Steering Committee. 

In 2011, an external panel of experts in farm financial analysis was 

assembled to conduct a comprehensive review of the ARMS process for 

constructing financial statements and to provide recommendations regarding 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 6 

possible changes to questionnaire content, variable summarization methods, 

and data collection procedures. The external panels received briefings on the 

ARMS process, asked questions of the process, and then met to discuss 

possible recommendations. An AAEA symposium was held to further vet 

recommendations. A special issue of the Agricultural Finance Review is in 

progress to publish the outcome of the review and panel recommendations 

and the ERS/NASS ARMS Steering Committee will develop a response to the 

panel. 

ERS staff contacts academic animal scientists and economists, extension 

staff, other government agencies, and commodity groups during the 

development of community of practice livestock versions, and solicits their 

advice on pressing issues and specific question formulations. The efforts have 

been expanded and systematized since the ARMS review. 

Status: Completed. NASS will conduct an ongoing annual program to 

solicit stakeholder input on the ARMS three-phase program and report to the 

Office of Management and Budget (OMB) in its Paperwork Reduction Act 

(PRA) submissions. The special issue of the Agricultural Finance Review

was published in July of 2012 and is available via Emerald Subscription 

http://www.emeraldinsight.com/journals.htm?issn=0002-1466. 

CNSTAT Recommendation 3.2: The NASS Advisory Committee on Agriculture 

Statistics should expand its scope to include an annual review of ARMS. 

NASS/ERS Response: The NASS Advisory Committee is organized to review 

ARMS content and methodology. In addition, a subgroup was formed in 

February 2008 to strengthen outreach efforts. The 2010 Advisory Committee 

meeting was not held because it had not yet been re-chartered by USDA. The 

ARMS was reviewed at the 2012 Advisory Committee meeting in March. 

Status: Completed. NASS will continue to conduct a review of the ARMS 

program at annual meetings of the NASS Advisory Committee on Agriculture 

Statistics and report to OMB in its PRA submissions. 

CNSTAT Recommendation 3.3: ERS and NASS should establish an ongoing, 

jointly sponsored, and appropriately funded methodology research and 

development program. Such a program should provide adequate resources to 

support current and future research, development, and statistical analysis needs 

throughout the implementation of ARMS and to assess and manage the quality of 

the data. If new funds cannot be obtained, funds from existing programs must be 

reallocated. 

NASS/ERS Response: In FY 2009 – FY 2011, NASS redirected funds to invest 

$1.2 million in a cooperative agreement with the National Institute of 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 7 

Statistical Sciences to address three high-priority projects. Two had 

particular impact on ARMS – one looked at a multivariate approach to 

imputation for the ARMS Phase III data (to which ERS staff contributed); the 

other investigated the large difference between farm numbers in the 2007 

June Area Survey (JAS) and the 2007 Census of Agriculture (JAS farm 

numbers are the sample control for ARMS estimates). Both projects involved 

NASS staff working with academics, doctoral candidates, and recent postdocs in an effort to bring in technical expertise that NASS did not have. Both 

projects were successfully completed in June 2011. NASS is in the process of 

developing implementation plans for the multivariate imputation; changes 

have already been made in operational procedures relevant to the JAS. 

NASS has additionally developed cooperative agreements with the University 

of Florida (Malay Ghosh and Linda Young), Iowa State University’s Center for 

Survey Statistics and Methodology (Sarah Nusser, Jae, Kwan Kim, Cindy Yu, 

and Zhengyuan Zhu), the Joint Program in Survey Methodology (Frauke 

Kreuter), Washington State University (Don Dillman and Dana Moore), 

University of Nebraska (Jolene Smyth and Kristen Olson), American 

Statistical Association Research Fellowship (Partha Larhiri), and others to 

continue to bring in academic experts to enhance research contributions and 

to develop NASS staff. In 2010-11, NASS successfully recruited six doctorallevel mathematical statisticians or survey methodologists to build a stronger 

research base in the organization; the agency expects to hire three more in 

2012. These researchers are working on a number of projects to strengthen 

NASS’s general foundation for statistical and survey research. Some are 

specifically assigned to ARMS-related research. 

In April 2012, NASS and ERS finalized a joint multi-year (2012-2016) ARMS 

Research Plan. 

Status: Completed. See the ARMS Research Plan. Update as of May 

2014: NASS processed both 2011 and 2012 ARMS Phase III data through 

both Iterative Sequential Regression (ISR) Imputation Methodology and 

operational imputation methodologies. Summarized results under these 

methodologies were compared, with favorable results. A similar comparison 

is currently being conducted using 2013 ARMS Phase III data. Assuming 

favorable results are again obtained, ISR imputation methodology will be 

operational for the 2014 ARMS III data. 

Update as of March 2015: ERS has conducted research on use of 

multivariate imputation for missing items that are not imputed by NASS. 

Initial results, for farm debt, were presented at the 2014 AAEA meetings 

(See: http://ageconsearch.umn.edu/handle/169401). A team consisting of 

two recently hired ERS economists and two North Carolina State faculty are 

currently working under a cooperative agreement to develop procedures for 

household items that can be effectively implemented in the survey process. 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 8 

ERS is also embarked upon a project to assess non-response bias in the 

household section of ARMS. Farms that do not respond to the household 

section of ARMS do provide demographic and other household information on 

the Census of Agriculture, and the project will use that information to 

evaluate systematic differences in household attributes among respondents 

and non-respondent to the ARMS household section. 

CNSTAT Recommendation 3.4: NASS and ERS should commit resources to 

developing a five-year plan tied to the census of agriculture for ARMS content, 

coverage, and methodology. The agencies should develop measures to control 

changes during the five-year period to minimize disruptions to the time series of 

the core content in ARMS. 

NASS/ERS Response: The content of Version 5 of the questionnaire for 

ARMS Phase III (commonly referred to as the CORE version), which is 

designed for mail, has been stable for almost ten years. Solicitation for input 

to the content for the 2012 Census of Agriculture was done in 2009/2010 and 

ERS took the opportunity to respond. A large ERS-NASS effort examined the 

content of both data collections to align similar data items. These changes 

have been made in the 2012 Census of Agriculture and will be made in the 

2012 ARMS. 

The time series disruptions after the 2007 Census of Agriculture were 

primarily due to a shift in the number of farms from previously published 

estimates. The June Area Survey is the primary survey source for 

establishing the annual number of farms estimates. To address this issue, 

NASS now: 1) puts increased emphasis during enumerator training on 

screening the JAS frame tracts for agricultural production, 2) provides 

additional administrative information to the enumerators that may be useful 

during screening, and 3) provides additional time for screening data 

collection. NASS has also implemented list frame maintenance procedures 

that will facilitate better tracking of changes over time to the list frame 

records. This will enhance the coverage and quality of the NASS frame for 

all NASS surveys including, in particular, ARMS and the census of agriculture. 

See responses to recommendations 5.1, 5.2, 5.3, 5.4, 5.5, 6.3, 6.4, 6.5, 6.6, 

6.7, 6.9, 7.3 and the ARMS Research Plan for information on methodology

research. 

Status: Completed. See the ARMS Research Plan. Update as of May 

2014: After the 2007 Census, a Farm Number Research Project was 

conducted that focused on identifying why the farm number indications from 

the June Area Survey were low. A key findings from the project indicated 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 9 

that our field enumerators were not following the instructions consistently 

across all states in the screening of tract operators. As a result Agricultural 

tracts were being coded up as Non-Ag Tracts. Several national training 

schools were held to re-iterate the June Area Frame methodology and the 

proper procedures needed to be followed for screening area segments to 

identify tract operators. As a result of the training the number of agricultural 

tracts has increased over the last several years which in turn led to an 

increase in the number of farms direct expansions. 

The largest undercount is for the number of farms. As measured in the 2012 

Census 12.3 percent of the total adjustment is from undercoverage, while 

only 3.4 percent of the Land in Farms adjustment is from 

undercoverage. The JAS is critical in the measurement of the coverage for 

the Census of Agriculture and for several other agriculture surveys. Major 

research efforts have been conducted by the Research and Development 

Division during the past four years to understand and model the undercount, 

misclassification and non-response. As a result of the research, new 

methodologies to adjust farm counts based on a Capture-Recapture 

methodology have been implemented for the Census of Agriculture. The 

methodology encompasses four sources of error, non-response, imputation, 

misclassification and coverage. The work is now being extended into the 

estimation process for the annual number of farms publications. 

Sample and Questionnaire Design 

CNSTAT Recommendation 4.1: The methodology research and development 

program the panel recommends should systematically (1) evaluate current 

instruments and practices, (2) collect data that inform both the revision of existing 

items as well as the creation of new items, (3) test revised instruments before they 

are put into production, (4) use experimental control groups to evaluate the 

differences between the old and new questionnaires, (5) improve understanding of 

respondent record-keeping practices and their effect on survey quality, and (6) 

designate a subsample of the existing ARMS sample for research and testing 

purposes. Key parts of this work would best be conducted in a cognitive or usability 

laboratory facility. It would be enabled by obtaining a generic clearance from the 

Office of Management and Budget for testing of all phases of the survey to allow for 

broader cognitive testing, evaluate the quality of data reported in response to each 

question, and evaluate the impact of mode of data collection across the three 

phases. 

NASS/ERS Response: NASS now has an OMB-approved generic clearance 

docket (OMB Control # 0535-0248), which is used to do testing and 

evaluation of NASS questionnaires. A variety of testing methods, including 

cognitive testing, focus groups, split sample field tests, etc., are used to test 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 10 

ARMS and other NASS surveys. NASS does not plan to create a cognitive 

laboratory facility due to the geographic dispersion of farm operators needed 

for testing. As is typical in establishment surveys, most testing is conducted 

with onsite visits. NASS is using the OMB-approved generic clearance docket 

to evaluate current instruments and practices (item 1) and to test revised 

instruments before they are put into production (item 3). 

NASS conducted an extensive analysis of imputation for the 2007 Census of 

Agriculture and then used the analysis to inform questionnaire design for the 

2012 Census of Agriculture and the 2012 ARMS, which have many of the 

same questions (item 2). See also the response to Recommendation 3.4. 

NASS currently uses an experimental control group to evaluate differences 

between data reported on ARMS mail and field versions to determine whether 

less detailed information obtained on mail can substitute for the 

disaggregated detail on the field version (for the 2011 data). NASS will 

continue to use the experimental design approach to assess questionnaire 

differences (item 4). 

NASS has hired an individual with prior experience with agricultural data to 

lead a project on designing data collection methods for large and complex 

operations across its surveys, including ARMS. This could involve 

examination of respondent record-keeping practices, but past research in this 

area was not productive (item 5). 

A subsample of ARMS for research and testing purposes will be considered 

when there are sufficient new ARMS initiatives to justify this mode of testing 

(item 6). 

Update as of February 2014: ERS and NASS have research underway to 

study the challenges posed for USDA statistical survey programs by changing 

technology and farm structure, particularly the shift of production to larger 

and more complex farms. A case study was developed and presented to a 

workshop of large commercial farmers, held at UC/Davis in March of 2013. 

It identified the key issues and barriers in eliciting response from large and 

complex producers and alternative treatment paradigms. The presentation 

allowed ERS and NASS staff to engage with large and complex producers, to 

market our surveys to them, and to engage them in discussions on ways to 

elicit data more efficiently. Staff will continue these discussions with 

producers from the workshop, and we intend to explore further workshop 

opportunities in other programs. 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 11 

Status: Completed. The ARMS Research Plan identifies the components of 

this recommendation that are of most importance in the next 5 years. 

CNSTAT Recommendation 4.2: ERS and NASS should improve the consistency of 

variables across ARMS versions and over time. 

NASS/ERS Response: NASS developed and expanded the Questionnaire 

Repository System that allows for improved standardization of variables 

across ARMS questionnaire versions and over time. Consistent master 

variable names are shared across questionnaire versions. The same master 

variable names that are used to generate questionnaires are used to 

populate the Data Warehouse. This also facilitates re-use of these master 

variable names from year to year, enabling researchers to consistently query 

using the same names. 

ERS provides metadata and other documentation that informs ARMS data 

users on constructing variables across questionnaire versions. See, for 

example, this list of variables and the Phase III summary listing and 

description of classification variables in Attachment B. 

Status: Completed. Current ARMS metadata are recorded in the NASS 

Questionnaire Repository System as part of the agency's operational 

efficiency measures. ERS provided historical information. 

CNSTAT Recommendation 4.3: NASS and ERS should explore the collection of 

auxiliary information on a formal basis, as well as feasibility of enriching the ARMS 

data files with information from administrative data sources, geospatial data, and 

the like. 

NASS/ERS Response: ERS and NASS are participating in an OMB-led 

initiative to incorporate selected administrative data into surveys, and will 

evaluate opportunities with regard to current ARMS questions. NASS is a key 

participant in a USDA effort to synchronize reporting of administrative 

(program) data for the Farm Service Agency (FSA), the Risk Management 

Agency (RMA), and the Natural Resources Conservation Service (NRCS) that 

is seeking common definitions and reporting. The NASS role has been to 

inform the data development process. Ultimately the administrative data will 

be of more value for developing agricultural production and conservation 

statistics – several components addressed by ARMS. NASS has also made 

progress in developing the Cropland Data Layer (CDL) using geospatial data 

that provide end-of-season crop acreage estimates, with staff researching 

the development of yield estimates for major commodities. These data could 

feed into the ARMS database. 

Status: In progress. As administrative data from the USDA project become

available, NASS will assess their use in ARMS. NASS continues to assess 

applications of its geospatial data for its survey programs, including ARMS 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 12 

and will conduct relevant research as opportunities become available. See 

the ARMS Research Plan.

Data Collection 

CNSTAT Recommendation 5.1: ARMS should use automated means to collect 

paradata on interviewer assignments to cases, the relationship between the 

interviewer and the sample farm respondent (i.e., whether they know each other), 

demographic characteristics of the interviewer and the characteristics of the sample 

farms for nonrespondents that are coordinated with information obtained for 

respondents, either through the interview or interviewer observation. These 

paradata could be used to determine the need for additional research on the impact 

of the relationship between the interviewer and the respondent on the quality of 

answers. This data collection can best be facilitated using computer-assisted 

technologies. 

NASS/ERS Response: The use of paradata in managing the respondentinterviewer interaction is best accomplished using computer-assisted 

technologies. NASS initiated an operational efficiency in FY 2010 to pilot the 

use of computer-assisted reporting in the field using personal enumeration 

devices. The Apple iPad was selected for this purpose, using wireless 

broadband transmission. Prototypes were developed; as of mid-2012field 

offices in 18 states are equipped with iPads. Initially the iPads are being used 

for questionnaires available with Web instruments. Once iPads are 

implemented in all states, it will be feasible to use paradata for managing 

field interviewing. During this implementation time-period, ARMS instruments 

will be designed for access on the iPad. NASS is also developing a system to 

facilitate the use of paradata on the iPADs. This can include scores from 

recently developed ARMS nonresponse propensity models. See 

recommendations 5.5 and 6.3. 

Status: In progress. NASS has begun to develop the systems that will 

facilitate the use of paradata, designing the systems specifically for this use. 

As systems are implemented, paradata will be used in managing the 

interview process. See the ARMS Research Plan. . Update as of May 2014: 

All Field Offices are equipped with CAPI instruments but updates are needed 

to the system to allow for a more interview friendly application to complete 

the complex ARMS surveys. Less than 2% of the 2014 ARMS survey was 

completed using the CAPI instrument. 

CNSTAT Recommendation 5.2: NASS should systematically explore the 

consequences of interviewer departures from standardization in the interview. To 

facilitate this, NASS should collect paradata on the frequency with which 

interviewers follow the order of the questionnaire, read questions as worded, 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 13 

provide clarification, and similar indications of departures from standardized 

procedures. 

NASS/ERS Response: To analyze departures from standardization by 

interviewers, the ARMS interviews would have to be recorded and analyzed. 

NASS currently does not have the systems in place to allow this, but is 

working with the Census Bureau to obtain the use of the computer audiorecorded interview (CARI) system developed by RTI International under 

contract to the Census Bureau. Once the system is available, it should be 

feasible to implement it on any NASS surveys collected by field enumerators 

on an iPad. 

In the interim, NASS and ERS invest in an annual national workshop for the 

field statisticians that focus on standard data collection, edit, and analysis 

procedures. Participants at the national workshop in turn hold local 

workshops where standard procedures are taught to the interviewers. These 

workshops provide a platform to strengthen the standardization efforts and, 

in turn, result in improved data quality through standardized editing. Future 

costs analysis will also be improved through standard data collection 

procedures. 

Status: In progress. See ARMS Research Plan for research NASS and ERS 

will conduct prior to implementing CARI. In the interim, NASS continues to 

instruct NASDA Supervisors to monitor data collection procedures and reinterview a small percentage of respondents for quality check purposes. 

NASDA Supervisors must complete the quality check forms and return them 

to the NASS Regional Field Office for review. Update as of May 2014: The 

ARMS User’s Guide is published and available on the ERS website. In 2014, 

NASS conducted a proof of concept project using the Census Bureau’s CARI 

system for the Agricultural Labor Survey. Although the project showed that 

a CARI system can provide beneficial information about the quality of 

questionnaire instruments and interviewer behavior, that particular CARI 

system does not fit into NASS’s information technology infrastructure. NASS 

is currently looking at other CARI systems that may fit our needs and 

integrate with our call centers’ telephone systems.

CNSTAT Recommendation 5.3: NASS should use available analytic tools, for 

example, cognitive interviews, interviewer debriefing, recording and coding of 

interviews, and re-interviews, to investigate the quality of survey responses. 

NASS/ERS Response: Enumerator training and quality assurance follow-ups 

have been expanded. 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 14 

NASS plans to initiate methodological research on total survey error once a 

PhD statistician is hired with this competency. The Research and 

Development Division has added a PhD statistician whose background is in 

quality control. One of his assignments is to assist in setting up quality 

control measures for our data collection process, especially with respect to 

the newly established NASS National Operations Center (NOC), which began 

collecting data during the last quarter of 2011 on a limited number of 

surveys. 

In FY 2013, NASS expects to use the NOC in lieu of the Census Bureau’s 

National Processing Center to process the ARMS data. Thus NASS will 

implement quality procedures developed for the NOC. Once CARI is 

operational and NASS has developed a computer-assisted interview 

instrument for ARMS, it will be feasible to record and code interviews. This 

may not occur until after the 2012 Census of Agriculture. See also the 

response to Recommendation 5.2. 

To complement the longer-term CARI solution and supplement cognitive 

interviews, NASS will utilize interviewer debriefing and training for field staff 

and interviewers to investigate and address the quality of survey responses. 

Status: In progress. Staff are in place to address this recommendation as 

the required systems become operational. See the ARMS Research Plan. 

Update as of June 2015: In May 2014, NASS trained field staff in several 

states on cognitive interviewing methods and procedures. Having a trained 

cadre of cognitive interviewers makes conducting cognitive interview projects 

more cost and time effective. So far, these trained staff have conducted 

cognitive interviews for several NASS surveys, and will likely be used for 

ARMS cognitive interviews in the future. Additional NASS field staff will be 

trained on cognitive interviewing in July 2015. 

CNSTAT Recommendation 5.4: NASS should move to computer-assisted 

interview and possibly Web-based data collection, after research and testing to 

determine possible effects of the collection mode on the data. Computer-assisted 

personal interviews and Web-based data collection will provide opportunities to 

increase timeliness, improve data quality, reduce cost, and obtain important 

paradata. 

NASS/ERS Response: Web-based data collection is available to about one 

half of the ARMS sample nationally. NASS utilized Morae usability testing 

software to test computer- based instruments. Implementation and testing of 

computer-assisted personal interviewing (CAPI) began in fall 2009 with the 

assignment of a CAPI project manager. Due to the complex nature of ARMS, 

with numerous tables and interrelated instrument designs, CAPI 

implementation will be incremental over the next few years. 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 15 

Status: In progress. NASS is in the process of putting this 

recommendation in place with its operational efficiency measures. The 

application of CAPI to ARMS is addressed in the ARMS Research Plan. 

Update as of June 2015: Completed. Web-based and CAPI data collection 

was available for all the 2014 ARMS sample and will be available in all future 

ARMS surveys. 

CNSTAT Recommendation 5.5: NASS and ERS should develop a program to 

define metadata and paradata for ARMS so that both can be used to identify 

measurement errors, facilitate analysis of data, and provide a basis for 

improvements to ARMS as part of the broader research and development program 

the panel recommends.

NASS/ERS Response: Start-up activities on defining and using paradata have 

begun with a preliminary literature search to determine state of the art, 

current applications, and general trends. This information will be used to 

inform decisions about how to organize this effort. 

Research has been directed to developing predictive models to identify 

operations highly likely to be non-respondents in ARMS and other surveys. 

These models use census of agriculture data as a proxy for ARMS 

respondents and non-respondents. During the 2011 ARMS, NASS is collecting 

information on how the data from the predictive models can be used in ARMS 

data collection. The potential impact of identified subsets of these likely nonrespondents on non-response bias has been evaluated and results have been 

documented in research reports and external conference presentations. 

In the future, expanded use of CAPI data collection on iPads and 

development of an ARMS CAPI questionnaire should facilitate the capture of 

additional paradata both directly and from interviewer observation. However, 

current ARMS data collection does not include the routine capture of 

paradata. Ultimately, NASS hopes to be able to use paradata to reduce nonresponse bias. 

Interim and complementary responses to recommendations 5.2 and 5.3 will 

be employed to provide a basis for improvement to ARMS until the longerterm solution can be implemented. 

Status: In progress. Technology is being developed to move this initiative 

forward. Until those systems are in place the use of paradata will not be very 

effective. Testing will continue on how to best use the information from the 

predictive models in data collection. See the ARMS Research Plan. 

 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 16 

Nonresponse, Imputation, and Estimation 

CNSTAT Recommendation 6.1: NASS should routinely report ARMS case 

dispositions linked across survey phases to provide the foundation for appropriate 

response rate calculations for Phases II and III. 

NASS/ERS Response: This information has been reported in past years for 

specific commodity versions of the ARMS Phase III survey in an internally 

published document. This information will be expanded to cover all versions 

of ARMS and will also be included in the Methodology and Quality Measures 

document released to the public in August 2012 and annually thereafter. 

The Farm Production Expenditures report published in August 2011 contains 

a statement and link to additional information on survey methodology and 

quality measures. Quality metrics include sample size, response rates, 

coefficients of variation, and percent of estimate from respondents. 

Status: Completed. ARMS case dispositions linked across survey phases 

have been compiled and maintained within NASS since 2006. Starting in 

August 2012, these tables will be included in the ARMS Phase III Survey 

Methodology and Quality Measures document that is published along with the 

Farm Production Expenditures report. These documents will be available to 

the general public through the NASS Web page.

CNSTAT Recommendation 6.2: All published ARMS response rates for Phase II 

and III should be calculated to reflect the nonresponse from the preceding 

phase(s). 

NASS/ERS Response: A new method of calculating the response rates to 

reflect the nonresponse from previous phases will be developed. NASS has 

always reported response rates for each individual phase of ARMS 

independently, but this new method will provide a response measure that 

covers all three phases. This information will be considered for inclusion in 

the Methodology and Quality Measures document released to the public in 

August 2012. 

Status: Completed. 

Update May 2014: NASS has calculated multiple “cumulative” response 

rates and will evaluate those response rates and will be considered for 

publication (pending any unforeseen issues) in the August 2014 Farm 

Production Expenditures Methodology and Quality Measures Document. 

Update June 2015: NASS will publish a “cumulative” response rate to the 

data user’s manual on http://www.max.gov
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 17 

CNSTAT Recommendation 6.3: The nature of the ARMS nonresponse bias should 

be a key focus of the research and development program the panel recommends. 

This research and development program should focus initially on understanding the 

characteristics of nonrespondents. 

NASS/ERS Response: NASS has and will continue to explore nonresponse 

bias using predictive models built using census of agriculture data with the 

current year ARMS data. These analyses have evaluated bias for key survey 

estimates and the effect of NASS weighting procedures on bias. These 

studies of nonresponse bias and additional analysis of respondent incentives 

have been conducted and results have been published. NASS continues to 

assess nonresponse bias. Studies to date have shown that current NASS 

weighting procedures reduce or eliminate bias for most key survey items, 

although, as described below in research conducted by ERS, the impact of 

nonresponse adjustment on estimates is sizeable for some measures. 

In 2008, research projects were completed to examine reasons for 

nonresponse in Phase III of the 2006 ARMS. Studies were completed in 

Washington and Louisiana, providing an opportunity to examine regional 

differences. Item nonresponse tabulations are routinely circulated among 

ARMS managers, and summary analyses are disseminated through a survey 

research Web page). 

Research is currently underway to evaluate the use of the information from 

predictive models in data collection both to increase response and to 

decrease non-response bias. Based on the results of current research, future 

activities may focus on ways to use the nonresponse models to supplement 

the current ARMS weighting procedures. 

ERS research on nonresponse bias has focused on economic variables that 

influence nonresponse, and the effects of nonresponse on economic analyses 

using ARMS data. The research uses census responses from ARMS 

nonrespondents, and finds that farm size plays an important role in 

nonresponse. Accounting for nonresponse has very minor effects on most 

coefficients analyzed in several econometric papers, but important (50%-

100%) impacts in a few. Moreover, standard econometric corrections for bias 

do not work in all cases of concern. Continuing work aims to isolate the types 

of measures for which bias will be important. 

Status: Completed. Update February 2014: NASS is currently using the 

nonresponse propensity models to identify likely non respondents. This 

information is utilized when assigning data collection methods. Targeted 

methods used for operations that were identified as likely non respondents 

included in-person recruitment by more experienced NASS staff and 

interviewers, providing publications and brochures, a drop off/pick up 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 18 

methodology and emphasizing data uses that apply to specific types of 

operations. 

CNSTAT Recommendation 6.4: The research and development program should 

continue NASS’s work on both public relations and incentives, and it should do so 

with a focus on nonresponse bias, not simply nonresponse rate. 

NASS/ERS Response: Much work has been done over the years on targeting 

public relations materials toward specific groups in ARMS—in some cases, 

those with historically low response rates. For several years, monetary 

incentives have been used and researched for the ARMS core sample with 

some success in incremental response rate increases. However, with the use 

of nonresponse propensity scores (see recommendation 6.3), we plan to 

utilize the incentive funds to conduct more targeted nonresponse avoidance 

activities in lieu of its use to manage the debit cards. 

Nonresponse propensity models can be used to identify highly likely nonrespondents before data collection begins. The models developed by NASS 

identify multiple subgroups of highly likely non-respondents according to 

farm production or operator characteristics. This will allow NASS to alter data 

collection procedures, develop targeted publicity materials and incentives, or 

alter interviewer assignments in focused nonresponse avoidance. Likely nonrespondents are currently being identified and a split sample experiment is 

being conducted to evaluate whether response rates can be improved for 

these operations. Similar to prior research, the impact of these nonrespondents on bias in key survey estimates will also be included. Results 

may indicate that some groups of likely non-respondents have greater 

impact on data quality and these would be the focus of future efforts. 

Status: Completed. Update February 2014: The models are being 

utilized in planning data collection and for follow-up non respondents. 

CNSTAT Recommendation 6.5: The research and development program should 

analyze whether there are differences in ARMS unit and item nonresponse rates 

between census and non-census years, with an eye toward deciding whether 

making ARMS mandatory would improve data quality. 

NASS/ERS Response: The Research and Development Division performed a 

detailed analysis of the item nonresponse rates for the 2006 and 2007 ARMS 

Phase III. The report summarizing the analysis, published June 2012, looks 

at item nonresponse in two different ways to account for the fact that 

collection procedures at the time did not permit differentiating between valid 

zeros, zeros that are imputed by an analyst, or zeros that were filled in by 

data entry staff when no value was available during keying. In addition, a 

change rate was calculated to examine the total number of changes to an 

item. The report contains these three calculations for all variables collected in 

ARMS Phase III and identifies the problematic items. 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 19 

A relatively small number of items did not meet the OMB threshold. However, 

the items that fell short were consistent across years. Most of these items 

dealt with landlord and contractor expenses, values that may not be readily 

available (or available at all) to the respondent (the operator). Some 

manually imputed items were imputed one hundred percent of the time, 

while one machine-imputation-eligible item, landlord’s property tax expense, 

was imputed over half the time. The analysis also discovered several dozen 

items that always get zero responses and many more that get only a few 

responses. These variables are being or have been addressed by the 

NASS/ERS Steering Committee in questionnaire design and editing 

procedures; they will be evaluated annually as part of post-data-collection 

and summary evaluation procedures. At this time, the Committee believes 

ARMS should remain a voluntary survey. 

Status: Completed. There is no current initiative or external effort to 

evaluate mandatory reporting (nor is there expected to be) on ARMS. 

CNSTAT Recommendation 6.6: The research and development program should 

examine how questionnaire design and interviewing changes could reduce item 

nonresponse. 

NASS/ERS Response: Questionnaires are routinely pretested to ensure that 

respondents can understand and answer ARMS items. In addition, field office 

staff submit comments and suggestions for changes after data collection that 

are used in the design of subsequent ARMS questionnaires and data 

collection procedures. 

Many questions that were the focus of testing and redesign on the 2012 

Census of Agriculture also appear on ARMS questionnaires. Work on the 

census has been done to identify the items with the most nonresponse and 

this was used to determine the areas of the questionnaire that were the 

focus of redesign. Item nonresponse in the 2012 Census of Agriculture will be 

compared to the 2007 Census to determine the impact of those changes. 

Status: Completed. Update February 2014: The testing and redesign of 

questions from the 2007 Census were integrated into the ARMS 

questionnaire. NASS and ERS continue to evaluate and make adjustments as 

needed. 

CNSTAT Recommendation 6.7: NASS and ERS should consider approaches for 

imputation of missing data that would be appropriate when analyzing the data 

using multivariate models. Methods for accounting for the variability due to using 

imputed values should be investigated. Such methods would depend on the 

imputation approach adopted. 

NASS/ERS Response: NASS and ERS staff have worked together over the 

past two years, and with academia as members of a cooperative research 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 20 

venture with the National Institute of Statistical Sciences (NISS), to develop 

an improved, multivariate approach to imputation for the ARMS Phase III 

data. They have developed new imputation methodology that will be 

incorporated operationally in the near future. The imputation procedure 

samples imputations from a joint model that is constructed from a sequence 

of conditional regression models known as iterative sequential regression. 

The procedure is conducive for high dimensional problems since it allows for 

flexible selection of a predictor function in each conditional model while 

maintaining a valid joint distribution. The procedure will jointly impute for 

more than 150 ARMS variables using models that were created using a 

combination of economic expertise and automated variable selection 

procedures. The product of the research is a system written in R 

programming language that will incorporate multivariate imputation for key 

ARMS variables into the NASS processing system. 

ERS has developed a cooperative agreement with researchers at North 

Carolina State University, to assist us in developing analyses and routines in 

support of the implementation of improved imputation methodologies for the 

variables that ERS imputes in the ARMS survey. These variable tend to be 

non-negative, clustered at zero, and highly skewed, and are therefore not 

directly amenable to the sort of iterative procedures being implemented by 

NASS for imputation. However, there are a set of transformations that may 

allow us to follow the NASS approach. ERS will engage with the NC State 

team during the Spring of 2013. 

Status: Research completed. R routine is being incorporated into the 

NASS processing system. Update as of February 2014: Testing is 

expected to occur in 2014 with implementation in 2015. Update as of June 

2015: Multivariate imputation was used operationally for the 2015 survey. 

CNSTAT Recommendation 6.8: All missing data that are imputed at any stage in

the survey should be flagged as such on files to be used for analysis. 

NASS/ERS Response: After the ARMS review, NASS initiated tracking of all 

item imputation computed by the machine imputation process. Any 

manual imputation currently done by a field office statistician is not 

traceable. If a field office statistician makes an update to the questionnaire 

before the data are entered into our system, that update cannot be 

differentiated from a value reported by the respondent. NASS business 

processes are being updated and it is anticipated that in the future original 

data will be captured before analyst review, which will result in all changes to 

the data being captured. This will occur as NASS moves to using Apple iPads 

for field interviewing, and as editing in CAPI becomes the same as editing by 

field office statisticians through training and supporting documentation. 

Forms returned by mail are scanned and keyed at a processing center so no 

editing occurs by a field office statistician. 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 21 

Status: Partially completed. System changes for machine imputation 

have been made to respond to this recommendation. Training procedures 

have been enhanced to emphasize appropriate use and notation of manual 

imputation. Development of a CAPI instrument for the Apple iPad is planned 

for 2015. See the ARMS Research Plan. Update as of June 2015: 

Completed. As of 2015, NASS systems allow for the tracking of the origin of 

the “current value” on the dataset for ARMS. This allows us distinguish 

between reported, updated, imputed, and edited values on the dataset 

moving forward. 

CNSTAT Recommendation 6.9: NASS and ERS should provide more clarification 

and transparency of the estimation process, specifically the effect of calibrations on 

the assignment of weights and the resulting estimates. 

NASS/ERS Response: NASS has assessed the impact of calibration weighting 

used for nonresponse adjustment on nonresponse bias for several years 

using census of agriculture data. These analyses show that calibration 

substantially reduced bias for most key ARMS estimates. NASS specialists 

have conducted seminars at ERS on the subject of calibration. 

The ARMS Phase III Methodology and Quality Measures document published 

for the first time for the 2010 survey contains a table that displays the 

percent of the survey estimate that came directly from the respondents. The 

converse of that number is the percent of the estimate that resulted from 

weight adjustments due to calibration, indicating the impact that calibration 

has on the survey estimates. Also included in the document are overall 

survey response rates and the coefficients of variation for each published 

estimate. Each provides the data user with a level of quality and precision in 

the ARMS Phase III estimates. The document is available to the public at the 

same time as the Farm Production Expenditures publication. Prior to this 

complete quality measures and methodology document, the coefficients of 

variation for the national estimates were included in the annual publication 

since the 2008 survey release. 

ARMS data summaries made available on the ERS Web site include a 

measure of statistical reliability for each variable presented. In addition, the 

site provides survey documentation including enumerator manuals, survey 

procedures, data dictionary, and other reference material. A data user's 

guide is under development, chapters of which are available upon request. 

Status: Completed. Update as of February 2014: See quality measures

on the NASS and ERS Web sites. Update as of May 2014: The ARMS User’s 

Guide is published and available on the ERS website. 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 22 

Methods of Analysis 

CNSTAT Recommendation 7.1: NASS should continue to provide sampling 

weights with the ARMS data set, combined with replication weights for variance 

estimation. 

NASS/ERS Response: A complete set of weights that include the calibrated 

weights, version specific weights, and masked weights along with their 

respective sets of replicate weights have been made available for many years 

and this practice will continue. 

Status: Completed. 

CNSTAT Recommendation 7.2: NASS and ERS should continue to recommend 

the design-weighted approach as appropriate for many of the analyses for users of 

ARMS data and as the best approach for univariate or descriptive statistics. 

NASS/ERS Response: NASS always recommends using the design-weighted 

approach when data users attempt to utilize micro level ARMS data in other 

research and analysis. All data requests are accompanied by an explanation 

of the weights we recommend and why utilizing another weighting method 

may not be accurate in representing the total population of farms at the 

state, regional, or national level. 

Status: Completed. 

CNSTAT Recommendation 7.3: NASS should investigate and implement 

improvements to the current jackknife replicates to make them more useful for the 

types of analyses performed by users in ERS and other organizations. In particular, 

NASS should increase the number of replicates and apply bounds to the magnitude 

of the weight adjustments. 

NASS/ERS Response: Based on research it conducted on this issue, NASS 

has taken several steps to improve the jackknife replicate method of variance 

estimation. Specifically: increasing the number of jackknife replicates from 

15 to 30, limiting the magnitude of weight adjustments in calibration and in 

the creation of replicate weights, posting a document describing methods 

that may be employed to sharpen analyses derived from the method, and 

undertaking research on improvements to the current method, with a 

particular focus on the applicability of the estimator for analyses of 

subsamples of the database. 

Status: Completed. 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 23 

CNSTAT Recommendation 7.4: NASS and ERS should investigate the feasibility 

of providing sufficient information on the design and nonresponse characteristics of 

ARMS, in order to perform design-based statistical analysis without using the 

replicate weights and to allow users to incorporate design and nonresponse 

information in model-based analyses. 

NASS/ERS Response: Documentation is planned that will serve as a guide for 

data users. (See response to recommendation 7.6.) A section of the guide 

will address alternative statistical procedures reflecting the improvements in 

imputation methods and use of replicates currently being investigated. 

Status: In progress. NASS and ERS will report on the status of this 

documentation to OMB in its PRA submission. This status document will be 

updated annually or more frequently as progress warrants. It will be posted 

to “the Independent Reviews” box on the NASS Surveys Web page. Status 

Change: Completed. Update as of February 2014: ARMS User’s Guide is 

in pre-publication phase at ERS and should be published online by May 2014. 

CNSTAT Recommendation 7.5: ERS should build an enhanced level of in-house 

survey statistics expertise, in cooperation with NASS. The specialized expertise in 

both econometrics and survey statistics needed to accomplish this is currently not 

present in ERS and is likely to require a significant effort in recruiting and training. 

NASS/ERS Response: ERS has hired new staff with survey research expertise 

and is continuing to recruit with econometric and survey research expertise 

in mind. Since the ARMS review, ERS has hired two economists with 

extensive survey experience or training, who have devoted part of their time 

to survey research efforts (on imputation and on nonresponse). ERS has 

hired a third economist with extensive survey experience to work on 

database development and integration at ERS. 

NASS has hired five doctoral-level survey statisticians and one survey 

methodologist. NASS has also used a PhD economist in ongoing research 

projects. 

Status: Initial effort completed. ERS and NASS will continue to build on 

this resource in accordance with other agency priorities. Update as of June 

2015: Completed. NASS and ERS completed a joint training in June of 2015 

for ERS researchers. 

CNSTAT Recommendation 7.6: ERS and NASS should collaborate on writing a 

Guide for Researchers for performing multivariable analyses using data from 

complex surveys, particularly data from ARMS. In areas in which expertise is not 

available for writing parts of such a guide, expertise should be sought from the 

statistics and economics community, especially those with experience in the 

analysis of survey data from complex survey designs. 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 24 

NASS/ERS Response: ERS now provides copies of the interviewer’s manual, 

as well as copies of the questionnaires, for each year of ARMS as part of the 

documentation on the public Web site. ERS also provides all users with a set 

of file documents on variable listings and definitions, estimation procedures, 

and the structure of financial accounts. An ARMS User’s Guide, which 

combines existing documentation memos and programs with new material 

and an annotated table of contents along with an executive summary in an 

organized framework, is being developed by ERS. Completed chapters are 

already offered to data users. Chapters are posted on the ERS intranet site 

for easy access by ERS staff, and are provided directly to external data 

users, who receive some chapters when they first inquire about data access 

and the others once they are granted access. 

In addition the DaTUM committee identified three types of users: 1) The 

casual user who wants to simply know what the survey is and wants access 

to basic public data; 2) the advanced user of public data who digs deeper 

into the data and studies more about the survey; 3) the researcher (both at 

ERS and outside of ERS) that is authorized to use record level data. As a 

departure from earlier work shown to OMB, the team now is working toward 

a substantial update to the website to address the needs of users 1 and 2. 

Mockups should be available for internal ERS review by the end of April 2013. 

The website will provide a basic overview of the survey and a clear pathway 

to the data for type 1 users and point to the large amount of publicly 

available information for type 2 users. The type 3 user will have all the public 

website material available and will be provided with additional resources that 

address all in-depth issues of record level data use at the passwordprotected site Max.gov. New documents are planned where needed and the 

team maintains its goal of launch no later than December 2013. 

Status: Completed. Update as of February 2014: ARMS User’s Guide is 

in pre-publication phase at ERS and should be published online by May 2014. 

Update as of May 2014: The ARMS User’s Guide is published and available 

on the ERS website. 

Dissemination 

CNSTAT Recommendation 8.1: ERS should continue to improve the ARMS Web 

tool by providing summaries on more variables and more subsets from ARMS, and 

to improve the ARMS extranet Web tool by adding the ability to link over years and 

to more sophisticated models. 

NASS/ERS Response: An ERS-NASS team is preparing a research and 

documentation Web-based search tool that will enable interested users to 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 25 

locate and download ERS and NASS survey research papers and annual 

ARMS metadata summaries. ERS recently added a new set of tables detailing 

participation in government programs to the Web tool. Additional tables are 

under consideration. 

Status: Initial effort completed. NASS and ERS will report on the Web 

tool in its PRA submission to OMB. 

CNSTAT Recommendation 8.2: USDA should consider extending the availability 

of ARMS microdata through the Census Bureau research data centers to increase 

access opportunities for using additional data sets and enabling researchers to 

match ARMS files with other data sets. 

NASS/ERS Response: ERS and NASS have joined the NORC Data Enclave 

program at the University of Chicago. The Data Enclave expands ARMS 

access opportunities for qualifying researchers in controlled on-campus 

environments. It provides a confidential, protected environment within which 

authorized researchers can access sensitive microdata remotely from their 

offices, an approach that combines good researcher access with researcher 

training and administrative support. 

Currently 18 researchers representing 15 academic institutions are using the 

Data Enclave to accomplish their research, increasing the value added of the 

ARMS data collection through high-quality analysis, deeper insights into key 

issues, and by tapping into a broader analytic community. These researchers 

are presenting their findings at conferences and publishing them in 

proceedings and journals. They are able to address questions at a more local 

level than can be done directly at ERS. Participants have achieved greater 

efficiency and lower costs by not having to undertake the time and expense 

of travel to USDA offices, and the support burden on these offices has been 

reduced. The Data Enclave is better suited than the Census Bureau research 

centers for ARMS data. Researchers are enthusiastic about how their 

analyses are facilitated, enabling them to collaborate with ERS in a more 

productive way. 

Status: Completed. NASS and ERS determined that the NORC Data 

Enclave program better suited the agencies’ and their researchers’ needs and 

have been developing this access mechanism. 

CNSTAT Recommendation 8.3: ERS should provide more training for new data 

users, including developing a data user manual, which also includes the 

recommended guide on statistical estimation, and offering training workshops. 

NASS/ERS Response: In 2010, ERS had an agency-wide two-day 

comprehensive training for ARMS users including participation from NASS 

and the Bureau of Economic Analysis. The workshop covered the uses of the 

survey, its components, the links between the survey’s goals and 
NASS and ERS Respond to 

ARMS Review Panel Recommendations

Revised April 2015 Page 26 

questionnaire design, and technical features of designing the survey, 

developing a research database, and analyzing the data. Topics included 

survey design and sample selection, weighting and calibration, data editing 

and imputation, inference with complex survey data, and the creation of farm 

income and wealth accounts from raw data. Another comprehensive training 

is planned for 2013. 

Status: Training will be offered based on demand and resources. 

NASS and ERS will report on training as applicable in its PRA submission.

Update as of May 2014: The ARMS User’s Guide is published and available 

on the ERS website. 

Update as April 2015: ERS will be again be conducting a formal ARMS 

training workshop, focused on the needs of new users, in June of 2015, with 

presentations from NASS and ERS staff. The workshop will be aligned with 

material from the ARMS User’s Guide, completed in 2014. ERS intends to 

post powerpoint presentations and record sessions, so that others can use 

the material. 

CNSTAT Recommendation 8.4: Database management practices should include 

a system for managing and reporting errors found by users, for ensuring the 

consistent labeling of the codes for raw variables, and for using consistent names of 

the ERS-created summary variables over time. 

NASS/ERS Response: ERS maintains the capability to receive email 

suggestions and notices regarding the ARMS data tool available on its home 

page. Responses are reviewed by staff. An email address and telephone 

number are provided for a member of the agency’s team for specific 

questions regarding access, special tabulations, or other questions regarding 

access and use of the data. See the ARMS Briefing Room. 

Status: Completed. 
Attachment A – ARMS Research Plan

 

National Agricultural Statistics Service 

and Economic Research Service 

Agricultural Resource Management 

Survey 

Research Plan 

FYs 2012 - 2016 

June 2012

 
Attachment A – ARMS Research Plan

Timeline

2012 

• Testing – Annual questionnaires are routinely pretested to ensure that respondents can 

understand and answer ARMS items. For 2012, conduct cognitive tests for soybeans and 

wheat. 

• Nonresponse bias analysis – Annual studies are now part of the operational process by 

our methods staff; in 2012, test 2011 data. 

• Test iterative sequential regression imputation methodology. 

• Large and complex farm project – Initial planning has begun and continues. 

• ERS and NASS will collaborate on research to explore the implications of expanding the 

value codes used in ARMS. 

• ERS and NASS will analyze differences in questionnaire reporting for specific items 

related to mode of data collection (mail versus personal interview). 

2013 

• Large and complex farm project -- Begin testing for ARMS and census. 

o Update February 2014: Pilot procedures for this project were developed and an 

internal steering committee was formed. 

• Testing – Annual questionnaires are routinely pretested to ensure that respondents can 

understand and answer ARMS items. For 2013, conduct cognitive tests for vegetables, 

rice, and peanuts. 

o Update February 2014: Testing was moved from Research and Development 

Division to the Standards and Survey Development Methodology Branch thus 

making it operational. 

• Nonresponse bias analysis – In 2013, evaluate bias in 2012 data. 

• Parallel test iterative sequential regression imputation methodology. 

o Update February 2014: Parallel testing is being conducting for the 2014 survey 

and will be operational in 2015. 

• Begin computer audio-recorded interviewing (CARI) system development, integration, 

and testing. \ 

o Update February 2014: NASS started exploring the feasibility of adoption of 

CARI and tested it on small surveys. Not operational at this time. 

• Develop three-phase response rate for ARMS. 

2014 

• Begin animated graphical Internet displays for ARMS work. 

• Nonresponse bias analysis – In 2014, test 2013 data. 
Attachment A – ARMS Research Plan

• Initiate research on linking ARMS data to administrative data available through USDA’s 

Acreage/Crop Reporting Streamlining Initiative (ACRSI). 

• Complete historic census data conversion for complex analysis. 

• Implement iterative sequential regression imputation methodology. 

o Update June 2014: Parallel test was conducted with iterative sequential 

regression imputation methodology and will be implemented for the ARMS 2015 

survey. 

2015 

• Begin to implement computer assisted personal interviewing (CAPI) for ARMS 

questionnaires with table in Blaise IS (Internet Services) software for 2015 data year. 

• Testing – Annual questionnaires are routinely pretested to ensure that respondents can 

understand and answer ARMS items. For 2015, the questions/commodities to be 

cognitive tested are still to be determined. Conduct census and ARMS evaluation for

census year. 

• Assess the coordination effort to synchronize ARMS questions with the 2012 Census of 

Agriculture report form. Use data from both the census of agriculture and ARMS to 

determine edit and imputation rates and evaluate nonresponse. 

• Nonresponse bias analysis – In 2015, test 2014 data year. 

2016 

• Use CARI for quality control in ARMS. 

• Testing – Annual questionnaires are routinely pretested to ensure that respondents can 

understand and answer ARMS items. For 2016, the questions/commodities to be 

cognitive tested are still to be determined. 

• Nonresponse bias analysis – In 2016, test 2015 data. 

• Automate collection of ARMS paradata. 

 
Attachment A – ARMS Research Plan

Plan

1. Sample and Questionnaire Design 

• Solicit stakeholder feedback 

o Prior to each reference period for data collection, NASS and ERS will engage 

stakeholders for input on the ARMS three-phase program. Resulting changes will 

be reported to the Office of Management and Budget (OMB) in its Paperwork 

Reduction Act (PRA) submissions. Public comments will be solicited from the 

Federal Register Notice, at NASS’s annual Data User’s Meeting, through the ERS 

ARMS Briefing Room, and other events and means. Internal comments are 

solicited through a formal request and response system and evaluated by the 

ERS/NASS ARMS Steering Committee. 

• Test questionnaires 

o Questionnaires are routinely pretested to ensure that respondents can understand 

and answer ARMS items. Questions are considered for evaluation and redesign 

each year. Questions can change depending on agricultural policies and structural 

changes. In addition, field office staff submits comments and suggestions for

changes using E-2 forms after each survey data collection period. This 

information is used to modify and make design changes to subsequent ARMS 

questionnaires and data collection procedures. The need for testing each year 

depends on content, timing, complexity, and resource constraints. 

When NASS conducts large-scale tests, we use an OMB-approved generic 

clearance docket (OMB Control # 0535-0248) to do testing and evaluation of 

NASS questionnaires. In years when only minor changes are made to any of the 

questionnaires, testing is limited to nine or fewer cognitive type interviews and is 

not submitted to OMB for approval. A variety of assessment methods, including 

cognitive testing, focus groups, split sample field tests, etc., are used to test 

ARMS and other NASS surveys. An experimental control group is used to 

evaluate differences between mail and field collected responses. Item nonresponse 

and survey design are examined. Varied data collection methods are evaluated for 

large and complex operations. The geographic dispersion of farm operators limits 

the use of cognitive laboratory testing. As is typical in establishment surveys, 

most testing is conducted with onsite visits. The OMB-approved generic 

clearance docket provides a venue to evaluate current instruments and practices 

and to test revised instruments before they are put into production. 

Prior to the 2012 Census of Agriculture, extensive coordination was done to 

synchronize questions on ARMS with the relevant census of agriculture 

questions. After the 2012 Census of Agriculture, NASS will assess these common 

questions using both census and ARMS data to determine edit and imputation 

rates and evaluate nonresponse in 2015. 
Attachment A – ARMS Research Plan

• Develop CAPI instrument for Apple iPad (field data collection) 

o Research and Development Division (RDD) is developing a data collection 

instrument utilizing Blaise IS software for the area data collection in 2012. The 

software is currently a beta version. Once this software is in production version 

and NASS has tested questionnaires with table format questions, NASS will begin 

full development of CAPI ARMS questionnaires. The plan date is 2015 for 2014 

data. 

• Use administrative data in lieu of collecting data 

o NASS is one of the four agencies involved with the Acreage/Crop Reporting 

Streamlining Initiative within USDA. ACRSI is establishing data standards to be 

used for the annual acreage reports collected by the Farm Service Agency (FSA) 

and the Risk Management Agency (RMA). The data collected under ACRSI 

standards will be available to NASS; however, these data will not be 

fundamentally different from what’s currently available from FSA and RMA. The

challenge in using the data will be in mapping FSA data to specific NASS 

operations sampled for ARMS. The research required to link FSA data to ARMS 

operations will begin by April 2014. 

• Analyze detail data to determine what questions work using CARI (2016) and paradata 

research related to the implementation of CARI. 

o RDD is currently investigating a beta version of CARI software. Once the Census 

Bureau accepts the software, NASS will integrate it into its system. NASS will 

begin to design procedures to select portions of CARI interviews for question 

review or interviewer coding as soon as software is available for research. Work 

is expected to be completed in 2016. 

2. Data Collection 

• Use of paradata 

o NASS does not yet have systems in place for the automated collection of paradata 

on interviewer assignments, interviewer characteristics, and their possible impact 

on data quality in the interview. Systems to allow this should be in place in 2015. 

NASS currently has little social science staff expertise to explore the impact of 

characteristics of interviewer and respondents. NASS expects to hire more 

research staff with this background and experience and to begin research in this 

area in 2016. 

• Large and complex farm project 

o After hiring a new staff member, RDD began initial planning in 2012 to 

investigate alternative methods for collecting data on large and complex 

agricultural operations. Initial implementation of any program for data collection 

from impact operations will begin with a small pilot set of operations. These 

operations will be selected by field office directors and other NASS staff. Indepth interviews and reviews of their relevant records, operating structures, and 
Attachment A – ARMS Research Plan

contact information could be conducted. Because all ARMS samples are 

coordinated with the census of agriculture in the census year, data collection for 

the pilot operations in 2013 will include ARMS and census data. 

• Quality – Design quality control procedures with CARI for interview verification 

o Once CARI software is in place in NASS, ongoing review of ARMS interviews 

will be possible. Samples of interviews can be captured with CARI software and 

interviewer and respondent behavior can be coded and analyzed. This will allow

evaluation of both questionnaires and interviewers. We expect this to begin in 

2016. 

• Quality – Develop three-phase response rate 

o The Statistical Methods Branch will use the 2011 ARMS survey data to derive 

and compute a multi-phase response rate that will accurately reflect the 

nonresponse from each preceding phase of the ARMS program. This new 

computation will also be tested on the 2009 and 2010 ARMS survey data. All 

testing will be completed by January 2013. Once successful tests have been 

executed for 2009 through 2011, this program will be implemented into the 

operational program for the 2012 ARMS survey cycle. Results will be published 

August 2013 in the 2012 ARMS Quality Measures and Methodology document, 

published annually. 

• Quality – Conduct nonresponse bias analyses 

o Nonresponse bias analysis has been developed by research staff, and is now 

integrated into ongoing post-data-collection activities. Bias analysis can be 

conducted annually. 

3. Nonresponse, Imputation, and Estimation 

• Incorporate multivariate imputation into the edit/imputation system 

o Iterative Sequential Regression (ISR) imputation methodology. NASS will test 

ISR in 2012 after the 2011 ARMS Phase III data are processed. Parallel test ISR 

imputation methodology in 2013. Operational use of ISR is expected to start in 

2014. 

4. Analysis of Complex Systems, Data Preparation 

o NASS obtained historic census of agriculture data files from the Census Bureau 

for the years 1964, 1969, 1974, and 1978. Before use, these data files need to be 

converted into formats that are readable by modern processing systems. 

Unfortunately, the conversion process is complicated by two facts: (a) the same 

conversion process is not workable for all years, and (b) the record layouts for the 

historic files are not always available. Work to convert the historic data files to 

the extent possible continues. However, data will probably not be recovered for 

all years or for all states. Also, once recovered, data may not be able to be 

mapped to more recent censuses and ARMS operations because of differences in 
Attachment A – ARMS Research Plan

operation identification numbers. RDD expects to complete the process to convert 

the historic data files (to the extent possible) and map data to operations (also to 

the extent possible) by January 1, 2014. 

 5. Dissemination 

o Work has begun on providing animated graphical displays of data for NASS’s 

Web site. Once the technology has been adopted, ARMS Web-animated graphical 

displays will be a high priority. Work on ARMS is expected to begin in 2014. 

 6. Data User Resources 

o ERS is developing an ARMS User’s Guide. A topics-based outline is nearly 

Complete; ERS will evaluate whether to post the outline to its website.

 7. Staff Development 

o Managers in NASS and ERS will continue to support the Joint Program on 

Statistical Methodology or other professional associations and encourage staff 

involvement in the program to enhance staff skills. 
Attachment B – ARMS Summary and Variables

[Type text] Page 8 

ARMS 

Listing and Description of Farm Business and Farm Operator Household 

Summary and Classification Variables, 1991-2010