Behavior Coding of 

October 2017 

Agricultural Labor 

Survey

Heather Ridolfo, David Biagas Jr., 

Emilola J. Abayomi, and Joseph 

Rodhouse

United States 

Department of

Agriculture

National

Agricultural

Statistics

Service

Research and

Development Division

Washington DC 20250

RDD Research Report

Number RDD-20-01

September 2020

The findings and conclusions in this report are those of the authors and should not be construed to 

represent any official USDA or U.S. Government determination or policy. 
i

EXECUTIVE SUMMARY

In 2016, cognitive testing of the Agricultural Labor Survey’s computer-assisted telephone

interviewing (CATI) instrument detected a number of issues with questions used to categorize 

workers. During cognitive testing, respondents had difficulty selecting an appropriate category 

for their workers and often wanted to report their workers under more than one category. In their 

attempts to classify workers into a single category, respondents often selected the first category 

that appeared applicable, leading to response bias. Finally, respondents with multiple workers 

sometimes had difficulty keeping track of the workers they had already reported (Sloan 2017).

Following the 2016 cognitive testing performed in 2016, a new CATI script was developed. In 

the new CATI script, respondents were instructed to report each worker under one category, 

some worker categories were renamed, and the worker categories were presented in a different 

order to mitigate the response order effects found in previous testing. Transition text was also 

added to improve questionnaire flow. 

In October 2017, a field test and a behavior coding study were conducted to evaluate the new 

CATI script. The majority of the sample received the original CATI script and a subsample 

received the revised CATI script. Behavior coding was conducted on a subset of the original and 

revised versions of the survey. Behavior coding is an objective, quantitative method for studying 

the interviewer and respondent interaction (Fowler 2011). Ninety interviews were selected for 

behavior coding. Forty-five of the interviews were conducted using the original script and fortyfive were conducted using the revised script. 

Behavior coding revealed that neither of the instruments were administered appropriately at an 

acceptable rate—a pattern that was exacerbated in the administration of questions for the second 

reference period. Interviewers often made major changes to the survey questions, which altered 

the construct being measured. Respondents were often encouraged by interviewers to report the 

same information for the second quarter as they had for the previous quarter. Interviewers also 

shortcutted questions at alarmingly high rates. That is, they entered responses without asking the 

survey questions. 

When directly comparing the performance of both instruments to one another, the revised 

instrument appeared to perform slightly better. For the most part, questions designed to classify 

agricultural workers into different worker codes were administered as worded more often when 

interviewers used the revised instrument compared to the original instrument. However, the new 

transition text on the revised instrument was often not read as worded. 

Based on these results, one recommendation is to implement the format of collecting number of 

workers and type of workers used on the revised script. However, a careful review of the entire 

revised CATI script is recommended to reduce the amount of text interviewers are required to 

read and to improve the flow of the instrument. Additionally, reviewing these results in 
ii

conjunction with the field test data may better inform the performance of these CATI 

questionnaires and the impact on data quality. Furthermore, another recommendation is that 

interviewers be retrained on the importance of reading survey questions as worded and 

discourage the usage of conversational interviewing to reduce interviewer variation. It is also 

recommended to continue use of the Verint system for interviewer monitoring and behavior 

coding research. 
iii

RECOMMENDATIONS

1) Use the revised method of collecting number of workers and worker type on the 

Agricultural Labor CATI script for the April 2018 data collection.

2) Review the revised CATI script to identify additional ways the script can be improved for 

CATI.

3) Examine the number and type of agricultural workers reported on this survey cycle and 

use this information along with these behavior coding results to inform the questionnaire 

design of future Agricultural Labor surveys.

4) Conduct enumerator training that focuses on the importance of reading survey questions 

as worded and discourages the usage of conversational interviewing to reduce interviewer 

variation.

5) Continue to conduct behavior coding on future cycles of the Agricultural Labor Survey.

6) Expand the usage of behavior coding to more data collection centers.

7) Expand the usage of behavior coding to evaluate other surveys conducted at NASS.
iv

TABLE OF CONTENTS

1. Introduction..........................................................................................................................1

2. Methods...............................................................................................................................2

2.1 Sample............................................................................................................................2

2.2 Behavior Coding .............................................................................................................2

2.3 Questionnaire Versions....................................................................................................4

2.4 Analytic Strategy.............................................................................................................7

3. Results.................................................................................................................................8

3.1 Peak Number of Workers ................................................................................................8

3.2 H-2A ............................................................................................................................ 11

3.3 Introduction Screens...................................................................................................... 13

3.4 Screener Questions........................................................................................................ 16

3.5 Worker Category Introduction ....................................................................................... 20

3.6 Worker Categories......................................................................................................... 23

3.7 Worker Subcategory Introduction .................................................................................. 26

3.8 Worker Subcategories.................................................................................................... 28

3.8.1 Supervisor Subcategory Questions........................................................................... 29

3.8.2 Livestock Subcategory Questions............................................................................ 31

3.8.3 Field Worker Subcategory Questions....................................................................... 32

3.8.4 Other Worker Subcategory Questions (original instrument only).............................. 34

3.9 Original Instrument Number of Workers........................................................................ 35

3.10 Hours Worked............................................................................................................. 37

3.11 Gross Wages............................................................................................................... 44

3.12 Revised Instrument Other Worker Questions................................................................ 50

3.12.1Contract, Custom and Value-Added Workers (revised instrument only)................... 51

3.12.2 Agricultural Inspectors (revised instrument only)................................................... 53

3.13 Revised Instrument Remaining Workers ...................................................................... 55

3.14 Total Number of Workers Confirmation (original instrument only)............................... 57

3.15 150-Day Question ....................................................................................................... 58

4. Conclusion and Recommendations...................................................................................... 61

5. References.......................................................................................................................... 62

APPENDIX A: Original CATI Script...................................................................................... 64

APPENDIX B: Revised CATI Script .................................................................................... 125
1

Behavior Coding of the October 2017 Agricultural Labor Survey

Heather Ridolfo, David Biagas Jr., Emilola J. Abayomi, and Joseph Rodhouse

Abstract

In 2016, cognitive testing of the Agricultural Labor Survey’s computer-assisted 

telephone interviewing (CATI) script detected a number of issues with the 

questions used to categorize workers. As a result, revisions were made to the 

CATI script and a field test was conducted to compare performance of the revised 

script to the original CATI script. Behavior coding was also performed on a 

subset of interviews conducted using the original and revised versions of the 

survey. The goal of the behavior coding was to evaluate interviewer behavior 

across the two versions of the survey. Results of the behavior coding showed that 

neither of the instruments were administered appropriately at an acceptable rate. 

However, questions designed to categorize agricultural workers into major 

categories and subcategories were administered as worded more often when 

interviewers used the revised instrument.

Key Words: Behavior Coding, Interviewer-Respondent Interaction, Data Quality

1. INTRODUCTION 

The Agricultural Labor Survey is a biannual survey, designed to measure characteristics of farm 

laborers. During each data collection, respondents are asked to report the type and number of 

workers on their payroll, hours worked, and wages paid for two reference weeks. When reporting 

type of workers, respondents are first asked to categorize their workers into four main categories: 

field workers, livestock workers, supervisors, and other workers. They are then asked to further 

categorize their workers into subcategories (e.g., agricultural equipment operators, farmworkers, 

graders and sorters, hand packers and packagers, and all other livestock workers). A total of 16 

subcategories are offered.

In 2016, cognitive testing of the Agricultural Labor Survey’s computer-assisted telephone 

interviewing (CATI) script found a number of issues with the questions used to categorize 

workers. During cognitive testing, respondents had difficulty selecting an appropriate category 

for their workers and often wanted to report their workers under more than one category. In their 

efforts to choose a single worker category, respondents often selected the first category that 

appeared applicable, leading to response bias. Finally, respondents with multiple workers 

sometimes had difficulty keeping track of the workers they had already reported (Sloan 2017).

Following cognitive testing performed in 2016, a new CATI script was developed. In the new 

CATI script, respondents were instructed to report each worker under one category, some worker 
2

categories were renamed, and the worker categories were presented in a different order to 

mitigate the response order effects found in previous testing. Transition text was also added to 

improve questionnaire flow. This script was evaluated using cognitive testing and found to 

perform well (Sloan 2018). 

In October 2017, a field test and a behavior coding study were conducted to further evaluate the 

new CATI script. The majority of the sample received the original CATI script and a subset of 

cases received the revised CATI script. The revised script was administered in the Southern and 

Northwest Regional Field Office (for more information see Sloan, Pick, Sirkis, and McGovern, 

forthcoming). 

Additional changes were made to both the original and revised CATI scripts in response to 

comments from field and HQ staff. The peak number of workers and H-2A questions, which 

historically have appeared in Section 4 of the survey, have high nonresponse. Field staff 

hypothesized that this was due to their placement at the end of the survey and a request was 

made to move them to the beginning of the survey. Subsequently, these two questions were

moved to Section 1 of the survey on both versions. 

Behavior coding was conducted on a subset of the original and revised versions of the survey. 

Behavior coding is an objective, quantitative method for studying the interviewer and respondent 

interaction (Fowler 2011). The goal of the behavior coding was to evaluate interviewer and 

respondent behavior across the two versions of the Agricultural Labor Survey. In particular, it 

was of interest to learn if there were differences between the two versions in terms of the 

interviewers’ ability to administer the questions in a standardized manner, and respondents’

requests for clarification and the ability to provide an adequate response.

2. METHODS

2.1 Sample

As mentioned above, the revised version of the October 2017 Agricultural Labor Survey was 

only administered in the Southern and Northwestern Regional Field Offices. Therefore, only 

calls from these offices were included in this study. Audio of all calls was recorded using the 

Verint system. Screen capture was not available at the time of data collection. Ninety interviews 

were selected for behavior coding. Nine had no labor during the reference periods. Half of the 

interviews were conducted using the original CATI script and half were conducted using the 

revised script. Although behavior coding was performed on the entire interview, these analyses 

focus specifically on questions designed to elicit the reporting of agricultural workers. 

2.2 Behavior Coding

In behavior coding, each turn in the interview can be coded. A turn begins when the first person 

begins speaking and ends when the second person begins speaking. A pair of turns is referred to 
3

as an exchange (Ongena and Dijkstra 2006). During the administration of a single question, there 

can be a number of exchanges before a final answer to the question is provided. In an ideal 

situation, there would be one exchange between the interviewer and the respondent. That is, the 

interviewer would ask the survey question and the respondent would provide an adequate 

answer. However, multiple exchanges can occur when the respondent needs clarification or 

provides an inadequate response. Coding each exchange can be time consuming and research has 

found there to be diminishing returns to coding all exchanges for a single question (Oksenberg, 

Cannell, and Kalton 1991). Therefore, only the first level exchange and the final response given

by the respondent were coded. 

For the purposes of this study, both the interviewer and respondent behavior were coded. In this 

report, the focus was only on the interviewer behavior (i.e., the administration of the survey 

questions). Although the study evaluated interviewers’ behavior, it is important to remember the 

goal of the study was not to rate their performance but rather to identify aspects of the questions 

that may make it difficult for the interviewers to administer them in a standardized way.

The codes used to assess the question asking behavior of interviewers are summarized in Table 1 

below.

Table 1. Behavior Codes for Interviewer Behavior

Code Description

ES Exact wording

VER Verified response

MC Major change

SC Shortcutting

If the interviewer read the question as worded with only slight or minimal changes, the ES code 

was applied. Questions were coded as MC if interviewers read the questions in a manner that 

substantially altered the question meaning. Verification occurred when interviewers verified a 

response that respondents preemptively provided in previous questions. Finally, questions were 

coded as shortcutting if interviewers failed to read the question entirely or failed to verify a 

response. Major changes and shortcutting are considered to be problematic behavior. When these 

codes are applied to a question at least 15 percent of the time, it is an indication that there is a 

problem with the survey question (Fowler, 2011).

Three researchers trained in behavior coding coded the interviews. Before coding began, 

Cohen’s kappa was calculated to ensure consistency across coding. Five interviews were selected 

at random for the assessment of interrater reliability. Two of these interviews were conducted 

using the original instrument while three were conducted using the revised instrument. Each 

researcher coded the five interviews independently. Cohen’s kappa (Cohen 1960) was calculated 

for all possible coder pairs. The overall average of these kappa combinations was 0.73, 

indicating that there was substantial agreement among the three coders (Landis and Koch 1977). 
4

The coding scheme initially tried to account for cases in which interviewers read the question as 

worded and the optional text associated with each question. However, interviewers only read 

optional text 44 times or 1.53 percent of the time, so cases in which the question was read as 

worded but the optional text was not read were recoded as ES. In addition, any behaviors that 

could not be accounted for with the coding scheme were coded as other (OTH). This code only 

occurred 13 times or 0.45 percent of the time, so they are not reported in the results.

2.3 Questionnaire Versions

Two versions of the Agricultural Labor questionnaire were evaluated (see Appendix A for a copy 

of the original CATI script and Appendix B for a copy of the revised CATI script). The 

instruments varied substantially in how they collected information about the type and number of 

employees hired to perform agricultural labor (see Table 2). The original instrument presented

the worker categories of interest and their descriptions followed by an open-ended question that 

asked respondents to describe the type of work their workers were hired to do. To classify these 

workers into more refined subcategories, a second question asked respondents to report the 

specific type of work their workers were hired to do. Respondents were then asked how many 

workers performed that type of work.

1 Once workers were subcategorized, respondents were

asked to report the total hours worked and wages paid to those types of workers. After collecting 

information about the first type of workers reported (e.g., number of workers, hours worked, and

wages paid), respondents were asked if they had any other agricultural workers on the payroll for 

the reference week. If they had other types of workers, the process was repeated until the 

respondents accounted for all workers.

Table 2. Agricultural Labor CATI Questionnaire Versions

Original Instrument Revised Instrument

Peak Worker Peak Workers

H-2A Workers H-2A Workers

Screener October Reference Period Introduction to October Questions

Worker Categories Introduction Screener October Reference Period

Worker Categories Number of Workers in Reference Week

Field Worker (FW) Subcategories Worker Categories Introduction

FW Subcategory 1 – Open Ended Description 

of Work

Number of Supervisors

FW Subcategory 1- Number of Workers Number of Livestock Workers

FW Subcategory 1 – Total Hours Number of Field Workers

FW Subcategory 1 – Total Gross Wages Number of Other Workers

Any More Workers Intro to Supervisor Subcategories

 

1 Note in the original version of the questionnaire, the instrument did not use the subcategory selected by the 

respondent in the survey questions asking about number, hours worked, and wages paid but instead used the main 

worker type the subcategory fell under. For example, instead of asking ‘how many livestock equipment operators 

were paid during the week’ the question asked, ‘how many livestock workers were paid during the week.’
5

(FW Subcategory Questions repeat for each 

FW subcategory selected)

Number of Supervisor Subcategory 1

Livestock Worker (LW) Subcategories Number of Supervisor Subcategory 2

LW Subcategory 1 – Open Ended Description 

of Work

Supervisor Subcategory selected – Open 

Ended Description of Work

LW Subcategory 1- Number of Workers Supervisor Subcategory selected – Total 

Hours

LW Subcategory 1 – Total Hours Supervisor Subcategory selected – Gross 

Wages

LW Subcategory 1 – Total Gross Wages (Supervisor Subcategory Questions repeat for 

each Supervisor subcategory selected)

Any More Workers Intro to Livestock Subcategories 

(LW Subcategory Questions repeat for each 

LW subcategory selected)

Number of Livestock Subcategory 1

Supervisor Subcategories Number of Livestock Subcategory 2

Supervisor Subcategory 1 – Open Ended 

Description of Work

Number of Livestock Subcategory 3

Supervisor Subcategory 1- Number of 

Workers

Number of Livestock Subcategory 4

Supervisor Subcategory 1 – Total Hours Number of Livestock Subcategory 5

Supervisor Subcategory 1 – Total Gross 

Wages

Number of Livestock Subcategory 6

Any More Workers Livestock Subcategory selected – Open 

Ended Description of Work

(Supervisor Subcategory Questions repeat for 

each Supervisor subcategory selected)

Livestock Subcategory selected – Total Hours

“Other Workers” Subcategories Livestock Subcategory selected – Gross 

Wages

“Other Workers” Subcategory 1 – Open 

Ended Description of Work

(Livestock Subcategory Questions repeat for 

each Livestock subcategory selected)

“Other Workers” Subcategory 1- Number of 

Workers

Intro to Fieldworker Subcategories 

“Other Workers” Subcategory 1 – Total 

Hours

Number of Fieldworker Subcategory 1

“Other Workers” Subcategory 1 – Total Gross 

Wages

Number of Fieldworker Subcategory 2

Any More Workers Number of Fieldworker Subcategory 3

(Other Workers Subcategory Questions repeat 

for each Other Worker subcategory selected)

Number of Fieldworker Subcategory 4

Confirmation of total workers in October Number of Fieldworker Subcategory 5

150 Days Question Number of Fieldworker Subcategory 6

149 Days Question Fieldworker Subcategory selected – Open 

Ended Description of Work
6

July Reference Period Screener Fieldworker Subcategory selected – Total 

Hours

[Repeats sequencing of questions] Fieldworker Subcategory selected – Gross 

Wages

(Fieldworker Subcategory Questions repeat 

for each Fieldworker subcategory selected)

Any Custom, Contract, Retail Workers

Number of Custom, Contract, Retail Workers

Instruction to Exclude Custom, Contract, 

Retail Workers

Any Agricultural Inspectors

Number of Agricultural Inspectors

Agricultural Inspectors – Open Ended 

Description of Work

Agricultural Inspectors – Total Hours

Agricultural Inspectors – Gross Wages

Remaining Workers – Open Ended 

Description of Work

Remaining Workers – Total Hours

Remaining Workers – Gross Wages

150 days Question

149 days Question

July Reference Period Screener

[Repeats sequencing of questions]

The revised instrument’s departure from the original instrument began with a question at the 

beginning of the interview that asked respondents to report how many agricultural workers were

employed on the operation during the reference week. The worker categories of interest were

introduced in a similar manner to the original instrument; however, in the revised instrument, the 

reporting of workers was decomposed by worker categories with the use of separate CATI 

screens for each worker category. The questions asking about subcategories of workers were also 

decomposed in a similar fashion. This was designed to ensure that all worker categories were 

presented to the respondent and to better elicit the more accurate classification and enumeration

of the workers on the operation. In addition, contrary to the original instrument, which collected

information for different types of workers based on the order that they were reported in the openended question, the revised instrument first collected information for managers and supervisors, 

followed by livestock workers, crop, nursery and greenhouse workers, and other types of 

workers.

For example, if respondents stated that their operation had 15 agricultural workers on the payroll 

during the reference week, they were then asked how many of the 15 workers were managers or 

supervisors, livestock workers, crop, nursery or greenhouse workers, or other workers in an 
7

iterative fashion with the use of four individual questions on separate CATI screens. If they

reported that 2 of the 15 workers were managers or supervisors, and 13 of the workers were 

livestock workers, they were asked a series of questions about the managers and supervisors and 

then asked a series of questions about the livestock workers. 

The subcategory questions were administered in a similar manner. For example, for the 

supervisor and managers subcategories, the respondent was first read an introduction to the 

supervisor subcategories and then asked how many of the 3 managers and supervisors were

farmers, ranchers, and other agricultural managers on one CATI screen, and how many of the 3 

managers and supervisors were first line supervisors of farm workers on the subsequent screen. 

Interviewers then collected other relevant information of interest (e.g., hours worked and gross 

wages paid) for the supervisor subcategories selected. The interviewer would then ask the 

respondent to subcategorize the livestock workers.

The two instruments differed in how they asked about “other” workers. In the original 

instrument, “other workers” was a main worker type and included the following subcategories: 

agricultural inspectors, animal breeders, pesticide handlers and sprayers, and any other worker 

not listed. Respondents were asked about these workers in the same fashion as the other 

subcategories. In the revised instrument, “animal breeders” were removed from the “other 

workers” subcategories and added to the livestock workers subcategories. Pesticide handlers 

were also removed from the “other workers” and added to the crop, nursery and greenhouse

workers subcategories.

Similar to the original instrument, respondents were asked how many “other type of workers” 

they had after being asked about the number of supervisors, livestock, and crop, nursery and 

greenhouse workers. However, after reporting detailed information on the supervisor, livestock 

and crop workers, they were asked if any other workers were custom, contract, or value-added 

workers. If the respondent indicated that they were, they were asked how many of the workers 

fell under these categories and were then informed that the survey does not measure these types 

of workers. 

If respondents still had other workers, they were asked if any of the remaining workers were

agricultural inspectors. If agricultural inspectors were reported, they were asked the number, 

description of work, hours, and wages for that worker type. Finally, they were asked about the 

remaining workers that were unaccounted for in the worker types presented.

Finally, in the revised instrument “field workers” was changed to “crop, nursery and greenhouse 

workers” in all relevant questions. Other smaller differences in the questionnaire versions are 

described in the results below. 

2.4 Analytic Strategy
8

To the extent possible, this report will examine comparable questions from the two versions of 

the Agricultural Labor CATI questionnaire. The results will begin by comparing the manner in 

which the questions were administered. In general, questions that are administered with major 

changes or that are not administered at all at a rate higher than 15 percent are considered 

problematic (Fowler 2011; Oksenberg, Cannell and Kalton 1991). In addition to examining the 

distribution of behavior codes, these analyses also summarize the most common ways in which 

these questions were administered with major changes. In instances where major changes were 

made to the question at a high rate, the types of changes made to the question are summarized. 

3. RESULTS

The analyses begin by comparing the distribution of behavior codes applied to the administration 

of the survey questions. Whenever possible, these analyses are deconstructed by the reference 

period of administration (e.g. October and July). Questions that are not deconstructed by 

reference period either did not have enough cases to permit doing so or were only asked once 

during the entire interview.

3.1 Peak Number of Workers

The peak number of workers question historically appeared at the end of the Agricultural Labor 

Survey (Figure 1). In 2017, this question was moved to the front of the questionnaire and was the 

first question asked on both versions of the CATI instrument. There was no difference in 

wording between the two versions. 

Figure 1. Peak Number of Workers 

Interviewers administered this question the same regardless of the instrument they were using. 

This question was administered with a major change the majority of the time (Original: 78%, 

Revised: 80%) and was only administered as worded 13 percent of the time when using either 

instrument (Table 3).
9

Table 3. Peak Workers Interviewer Behavior

Code Original Revised

ES 13% 13%

VER 2% 0%

MC 78% 80%

SC 7% 7%

Total 45 45

The most common major change made to this question was not reading the introductory text or 

the include/exclude statements. That is, interviewers only read the survey question. A couple of 

interviewers, who did not read the introductory text or the include/exclude statements, also did 

not read “will be” in the survey question (e.g. “What was the largest number of hired workers on 

your payroll for any one day this year?”). One interviewer did not read the introductory text or 

the exclude statements but did read the include statements: “During 2017, what was or will be 

the largest number of hired workers on the payroll on any one day? When reporting this number 

include hired workers that were directly paid by the farm operation, part time workers, paid 

family members, and hired managers.” 

In a couple of interviews, the interviewers read the introductory text and some but not all of the 

include/exclude statements (e.g., “The agricultural labor survey is the only federal survey that 

provides employment and wage estimates for all workers in the U.S. that are on a farm or ranch. 

You know, hired workers, part time workers, paid family workers. During 2017, what will be the 

largest number of hired workers on the payroll in any one day?”). In several interviews, the 

interviewer read the introductory text and survey question but none of the include/exclude 

statements.

Some respondents, who did not have the include/exclude statements read to them, expressed 

confusion regarding which workers to report. For example, in one interview where the 

interviewer read the introductory text and question but none of the include/exclude statements, 

the respondent initially reported three workers but then asked if he should include a part-time 

worker. After the interviewer confirmed that he should include part-time workers, the respondent 

changed his answer to six. In another interview, the interviewer only read the survey question 

and the respondent was unsure whether or not he should include H-2A workers: 

I: What would be the highest number of employees or hired workers that you may have had on 

your payroll on any one day of this year? 

R: Okay. So, let's talk about payroll. We are strictly H-2A, work visa, immigrant laborers. With 

that being said, do you want to talk about labor because they are all from the same ethnic 

community and they are all on a work visa, at the longest a 10-month program? 
10

I: Okay. 

R: And I don't know how [many] of the people that you're calling have a setup like that. But that 

is a federalized program that we're allowed to bring workers in. Are you familiar with that, sir? 

I: Actually, that's going to be my next question. So what's the largest number that would have 

been on any one day?

R: Of those type of workers? 

I: Yes, sir. It could be those type or whoever may have been employed. 

R: Okay. Are you looking at what we said last season? 

I: Well this is for this year. 

R: Yes, sir. What did we say last year? 

I: They don't give us access to those records, sir. I apologize. 

R: You know, probably 10.

The interviewer never correctly clarified the respondent’s question. H-2A should be included if 

they are paid directly by the farm operation. However, if they are hired through a contractor, they 

should not be included. This could have been clarified by reading the include/exclude statements. 

Not reading the include/exclude statements can also add additional unnecessary burden. For 

example, in one interview the interviewer did not initially read the introductory statement or the 

include/exclude statements:

I: During this calendar year, during 2017, what will be the largest number of hired workers on 

the payroll on any one day?

R: For the whole year? Uh, we had 16 employees. That would be our max.

I: And let me tell you this year, what we're excluding on this. We are not needing any of the 

retail workers. Okay? What we're looking for would beR: Labor. Manual labor

I: Exactly. 

R: Yeah, okay. Well see, my parents- this is where- I'll just be honest with you. My mom and 

dad own it. They consult. My mom comes in at the end of the quarter, helps me close out. So, we 

have two that are management that are- and my dad helps with sales because he's known in the 

industry. A lot of people call him and he's kind of my liaison. But they don't- it's not manual 

labor. Other than that, my highest number is 14. 

I: Okay. And what I was going to say is we actually have a place for any management. We have 

management, we have- actually there's four. There's what we call field workers that are actually 

doing the hands-on and then we have - that's one bracket. Another bracket is for foreman type of 

people or managers like that. And then we have the office workers. So you got regular worker, 

and then people who do supervision of those workers and then we have the third one, which is 

the office and within those categories, if people operate equipment, like in some locations they 

have tractors and what have you. We try to- We actually have a place where the people who are 

doing the work can- if they also do equipment we put that sort of in a bracket there because it's 

normally more of a senior person that's doing the- that's allowed to get on the equipment, you 

know. 
11

R: Yeah. So that would be my husband and we only have him doing that. And then we have the 

highest number of field workers this year would be 14 and then three management. 

I: Okay. So that would be 14 plus your husband? 

R: I'm sorry. No, it would be 13- 13 would be the highest for field workers, my husband is like 

management, operates the machinery and does the driving and he's the only one that does that. 

and then there's me, my mom, and my dad that does office. 

I: Okay. soR: Three 

I: So, three office. So that would be see a total of 17. 13 plus 1 plus 3 would be 17. 

R: Yeah. 

3.2 H-2A

Like the peak number question, the H-2A question was moved to the front of the questionnaire. 

The question was designed to capture employees hired through the H-2A program and was 

identical on both versions of the survey (Figure 2). 

Figure 2. H-2A Question 

The administration of the H-2A question was comparable across both instruments (Table 4). For 

both versions of the questionnaire, interviewers administered the questions as worded nearly 60 

percent of the time. However, it was administered with fewer major changes (31% vs. 40%) but 

shortcutted more often (7% vs. 0%) when interviewers used the original instrument. 
12

Table 4. H-2A Interviewer Behavior

Code Original Revised

ES 58% 58%

VER 4% 2%

MC 31% 40%

SC 7% 0%

Total 45 45

When interviewers were administering the H-2A question on the original instrument, the most 

common major change made to this question was rewording the question to ask if the workers 

reported in the previous question were H-2A workers (e.g., “Is that one person an H-2A 

agricultural worker?”) instead of asking if any of the hired workers the respondent has or will

have in 2017 are H-2A. Similarly, in other interviews, the interviewers changed the question to 

ask if the respondent currently has any H-2A workers (e.g., “Do you have any H-2A temporary 

workers?”). Others asked if the respondent has or will have any H-2A workers but did not read 

the reference year when asking the question. 

As in the original instrument administration, when interviewers made major changes to the H-2A 

question in the revised instrument they most often reworded the question to ask if the workers 

reported on the survey were H-2A and did not collect information on whether the respondent will 

have H-2A workers in 2017. Other interviewers also changed the question to ask if the 

respondent currently has any H-2A workers and some omitted the reference date. In a few 

interviews the interviewers asked if the respondent had ever had H-2A workers. Additionally, in 

some interviews, the interviewers failed to read the term H-2A when asking the question (e.g., 

“Of the hired workers you have or will have on the payroll in 2017, were any of them temporary 

agricultural workers or like migrant workers?”). Failing to read the term H-2A or focusing on 

“temporary” or other terms such as “immigrant workers” when administering this question is 

problematic as it could lead respondents to falsely report workers who are not part of the H-2A 

program. For example, in one interview, the interviewer asked, “By any chance, do you have any 

workers on the payroll that fall under the H-2A temporary - you know the immigrant program?” 

To which the respondents replied, “Oh ummm. Me, my husband, and parents are the only ones 

that are not immigrants.” Finally, in a few interviews, the interviewers presumed that the 

employees on payroll were not part of the H-2A program (e.g., “I am assuming he/she wasn't an 

H-2A temporary agricultural worker.” “And you don't hire any temporary workers either do 

you?”). 

Aside from changes the interviewers made to this question, the placement of the H-2A question 

at the start of the interview led to confusion for some respondents who employed H-2A workers. 

These respondents were unsure if the questions asking about workers during the specific 

reference periods were referring to H-2A workers only or all agricultural workers on their 

operation. This is demonstrated in the following exchanges. 
13

Example1: 

I: And more specifically, what type of work were they hired to agricultural equipment operators, 

farmworkers, graders, sorters, hand packers and packagers, or agricultural workers? 

R: This is just the H-2A workers or is this all of them? 

Example 2: 

I: And how many employees did this operation have on the payroll to do agricultural work the 

week of Sunday the 8th-

R: Everybody. 

I: Everybody? So, 45?

R: Uh-huh

I: And for the 45 paid workers I’d like to record the type of work they were hired to do and the 

total workR: Wait a minute. On that question is it U.S. workers or H-2A workers? 

3.3 Introduction Screens

Unlike the original instrument, the revised instrument begins with an introductory screen prior to 

asking questions about the workers employed during each reference period. Previous cognitive 

testing had found that respondents did not understand why they were asked to report payroll 

information for two reference periods (Sloan 2017). This revised introductory text was

developed to describe the basic task and inform respondents why NASS collects information for 

two reference periods. The introductory text for the October reference period is shown in Figure 

3 and the introductory text for the July reference period is shown in Figure 4. 

Figure 3. Revised Instrument - Introductory Screen for October Reference Period
14

Figure 4. Revised Instrument - Introductory Screen for July Reference Period

The introductory screen was read as worded 29 percent of the time for the October reference 

period and only 9 percent of the time for the July reference period (Table 5). Major changes were 

made to the question at a high rate for both reference periods at 29 and 34 percent, respectively. 

In addition, the introductory screen was shortcutted at unacceptably high rates of 42 percent for 

the October reference period and 57 percent for the July reference period.

Table 5. Introductory Screen Revised Instrument

Interviewer Behavior

Code October July

ES 29% 9%

MC 29% 34%

SC 42% 57%

Total 45 44

In interviews where the interviewer made a major change to the introductory text for the October 

reference period, none of the interviewers read the second sentence. Half of the interviewers read 

the first sentence as worded and then omitted the second sentence (e.g., “Now I will ask you 

some questions about the agricultural workers that you had on the payroll for a specific week in 

October.”). One interviewer, who only read the first sentence, also did not read the last phrase of 

that sentence (“for a specific week in October”). Other interviewers paraphrased the first 

sentence, indicating to the respondents they would be asking them about a specific week but did 

not mention “agricultural workers” and did not read the second sentence. For example, one 

interviewer said: “Now we're looking at a specific week in October, which would be last week.”

One interviewer paraphrased the first sentence and then told the respondent that these questions 

would not apply to him: “We are asking questions specifically for a week in October but that 

doesn't apply to you.” Another interviewer encouraged the respondent to report that he had the 

same workers in October and July: “We are going to take a snapshot of a week in October 8th

through the 14th and a week in July and if it's the same then I can do the second one after we get 

off the phone to save time.” 
15

Similar to the administration of the October introductory text, when interviewers made major 

changes to the July introductory text, they always dropped the second sentence explaining why 

respondents would be asked about workers in July. Instead of reading the text as worded, most 

interviewers just indicated to the respondent that they would be asking questions about July: 

“They are going to ask about a week in July now.” 

“Now I'm going to ask you about July.” 

“Now they want to go through July 9th through the 15th.” 

Additionally, some interviewers specified that they would be asking about workers in July: 

“Now I'm going to ask about your workers for July.”

“And it's going to ask about agricultural workers that you had on your operation in July." 

A few interviewers indicated that the questions about July would be the same as the questions 

asking about October labor: 

“And it's asking for agricultural work that you had for- this is about July. It's basically the 

same questions for July if you can- um Sunday, July 9th through Saturday July 15th.” 

“We're asking those same questions in July.” 

“Now we're going to go to a week in July and ask the same questions.” 

By making these types of changes, interviewers failed to convey the reasoning for asking about 

workers in two different reference periods. 

Additionally, when reading this introductory text, some interviewers changed the wording in 

ways that may encourage respondents to satisfice. When reading this introductory text, some 

interviewers asked the respondents if they wanted to report the same information for July as they 

did for October:

“Now I am going to ask about the agricultural workers that you had on your operation in 

July. Was it the same for July?” 

“And now Sunday July 9th through Saturday July 15, it's going to be all the same 

questions. You want to use all the same answers?”

These types of major changes may have encouraged respondents to report the same worker 

characteristics across the two reference periods instead of using cognitive effort to recall July 
16

workers or accessing records to determine the precise worker characteristics for the July 

reference period.

Finally, one interviewer completely changed the wording of this question and in doing so 

encouraged the respondent to not answer this series of questions. 

I: Now I'm going to ask about July. Do you think you could remember? July 9th through July 

15th?

R. No. But what's the question?

I: It's the same thing. We'd be asking the same questions. 

R: Oh no. I couldn't remember. I think that should be generally the whole thing. 

I: You think it would be about the same? 

R: Yeah. 

I: Well, you know, I'm just going to say unknown.

The introduction screens were revised to address respondents’ confusion in cognitive interviews 

regarding the collection of payroll information for two reference periods. In approximately half 

of the interviews, interviewers skipped these revised screens entirely and when the text was read, 

in approximately one-third of the interviews, the interviewers changed the wording substantially. 

The two most common changes made were failing to read the second sentence, which explained 

why NASS was collecting information for two reference periods and may have encouraged 

respondents to report the same information for both reference periods. Subsequently, in 

approximately 70 percent of the interviews that were coded, concerns raised in previous testing 

were not addressed with the addition of this new introductory text. 

3.4 Screener Questions

Prior to enumerating the workers on the operation for each reference period, interviewers 

administered a screener question to detect if respondents had any workers on the payroll during 

the reference periods. This screener appeared on both versions of the survey, but is different in 

important ways. The original instrument did not require that interviewers read the inclusion 

criteria—which appears in purple text in Figure 5 below. 
17

Figure 5. Original Instrument – October Screener Question 

The revised instrument required that the inclusion criteria be read. Additionally, optional text 

was added to the revised instrument to provide further guidance on what workers should be 

included in responses to this question (Figure 6).

Figure 6. Revised Instrument – October Screener Question 

There are notable differences in the administration of the screener questions across the two 

versions of the survey and the reference period. The screener question was more likely to be read 

as worded for both reference periods when using the original instrument (e.g. for October,

Original: 67%, Revised: 20%) (Table 6). Also, this question was administered with major 

changes more often when using the revised version compared to the original version for both 

reference periods (e.g., for October, Original: 20%, Revised: 69%), which is not surprising given 

that the optional text on the original instrument was required on the revised instrument.
18

Table 6. Screener Interviewer Behavior

Code October July

Original Revised Original Revised

ES 67% 20% 58% 0%

VER 7% 4% 7% 7%

MC 20% 69% 29% 77%

SC 7% 7% 7% 16%

Total 45 45 45 44

When administering the original version of the questionnaire, the most common major change 

made to the October screener question was not reading “to do agricultural work.” The next most 

common change made to the October screener was not reading the reference date when asking 

the question. Furthermore, some interviewers did not read the clause “to do agricultural work” or 

the reference date when administering the question: 

“Did you have any hired workers last week?”

“Did the operation have anyone on the payroll that work week?”

When changes were made to the July screener question in the original instrument, interviewers 

often changed the question to ask if the respondent had the same workers on payroll in July as 

they did in October: 

“Did those same two guys work the week of July 9th through July 15th?

”

“Did you have anyone else on the payroll to do agricultural work the week of Sunday 

July the 9th through Saturday July the 5th?”

“Did you have the same one worker?”

In some interviews, the interviewer did ask the survey question but, as in the October screener 

questions, they failed to read the clause “to do agricultural work.” 

When administering the revised instrument, the most common major change made to the October 

screener was not reading the inclusion criteria following the question, (i.e., Include part-time 

workers, paid family members, and hired managers. Only report hired workers that were directly 

paid by the farm operation.). In addition to omitting these criteria, in some interviews, key parts 

of the survey question were not read to the respondent. For example, in three interviews, the 

interviewer only read the question but did not say “to do agricultural work” when asking the 

question: “Did you have employees on the payroll the week of Sunday October 8 through 

Saturday October 14?” In another interview, the interviewer did not ask about agricultural work 

or if the employees were on the payroll: “Did you have any October 8th through the 14th?

”

In 
19

another interview, the interviewer read everything except the last sentence instructing the 

respondent to only report those they directly hired. 

In three interviews, the interviewer did not read the instructions following the question, left out 

key phrases from the survey question, and asked the question in a leading way (e.g., “The week 

of Sunday October 8 through Saturday October 15, you had employees doing agricultural work, 

you folks did have people on the payroll between October 8

th and October 14th

, correct?”). In one 

of these interviews, the interviewer asked the respondent if the workers he discussed earlier in 

the interview were working during the reference week: “We want to think back to last week. 

Sunday October 8th through Saturday October 14th did you have any of those two workers to 

help you last week?” This is a substantially different question since the operation may have had 

other workers (aside from the 2 reported earlier) during the reference period. 

Finally, instead of asking if the respondent had agricultural employees on the payroll that week, 

one interviewer changed the question to ask: “About how many people did you have working 

that week?”

Interviewers made similar changes to the revised July screener. Once again, the most common 

major change made to the survey question was not reading the inclusion criteria following the 

survey question. Many interviewers also did not read the clause “to do agricultural work.” 

Additionally, some interviewers asked the respondent if they had the same workers in July as 

they did October when administering the question: 

“Think back a couple of months ago. Saturday July 9th through Sunday July 15th. Did 

you have him working for you then?”

“Did he just do the same thing and work about 10 hours?”

“Would it be the same for July 9th through the 15th?”

Finally, in a couple of interviews, the interviewers asked if the respondent could recall the 

workers they had in July. This is problematic as it encourages the respondent to not report 

workers in July or to report the same workers as in October, as demonstrated in the interaction 

below. 

I: Do you think you can remember? July 9th through July 15th?

R: Uhhh, no. (Laughs) But what's the question?

I: The same thing

R: Oh no I couldn't remember... that should be generally the whole thing 

I: You think it would be about the same?

R: Yeah.

I: Well, you know, I'm just going to say unknown though.
20

As noted above, far more major changes were made in the administration of the revised screener 

questions than in the administration of the original screeners. The majority of the changes made 

in the revised screener were not reading the inclusion criteria following the survey question. The 

addition of the inclusion criteria was the most significant change to the revised instrument to 

ensure that respondents were including the correct type of workers in their responses. 

Additionally, not asking about workers “hired to do agricultural work” is problematic because 

farm operations may have employees on their payroll that should not be reported in this survey. 

Previous cognitive testing has found that this phrase is important to specifying the types of 

workers that should be included. The following example shows how respondents can misreport if 

this information is not provided. In this interview conducted using the revised instrument, the 

interviewer did not read the reference period or the inclusion criteria. 

I: During that time, did you have any workers on the payroll to do agricultural work?

R: I had one guy and a temp.

I: Okay so you had one guy and a temp, okay. 

R: Yeah, I just hired him through a temporary employment outfit. 

The interviewer then went on to include the temporary employee who was hired through a third 

party in the count of agricultural workers for the October reference period.

3.5 Worker Category Introduction

Questions used to categorize workers into the four categories of interest are critical as estimates 

for these workers (e.g. average hourly pay and total hours worked) are published by worker type. 

As noted above, the worker categories were introduced and presented to respondents differently

on the two questionnaires. On the original instrument, respondents were introduced to the worker 

categories and then asked to categorize their workers using an open-ended question. In contrast, 

the revised instrument followed the worker category introduction screen with iterative questions 

that asked respondents if any of the workers they reported corresponded to each of the four 

worker categories.

We first compare the administration of the screens that introduce respondents to the four worker 

categories of interest and provide a basic description of how to categorize workers who perform 

multiple functions on the operation. This introductory text served to inform respondents that a) 

interviewers would be asking about type of work, hours worked, wages paid, b) workers needed 

to be separated into categories that described the work they were hired to do, c) if workers fit 

into multiple categories they should be reported under the category that best describes the work 

they were hired to do, d) there are four categories to choose from, and e) the description of those 

categories. The main difference between the two versions of this introduction is in the original 

instrument interviewers were instructed to explain the categories to respondents – a task that was 

open to interpretation (Figure 7). However, in the revised version, interviewers are required to 

read the four worker categories (Figure 8).
21

Figure 7. Original Instrument - Introduction to Worker Categories
22

Figure 8. Revised Instrument - Introduction to Worker Categories

The results in Table 7 show that although neither instrument performed adequately, the 

introductory screen on the revised instrument performed better than that of the original

instrument. In general, the revised instrument was more likely to be administered with major 

changes than shortcutted. In addition, the introduction on the original instrument was more likely 

to be shortcutted than read with major changes across both reference periods. That is, rather than 

being skipped over altogether, the interviewers were at least administering some of the worker 

category introductory screens across both reference periods when using the revised instrument.

Table 7. Worker Category Introduction Interviewer 

Behavior

Code October July

Original Revised Original Revised

ES 19% 21% 0% 11%

VER 0% 0% 0% 0%

MC 19% 58% 9% 32%

SC 62% 21% 91% 57%

Total 37 33 34 28
23

When interviewers made major changes to the original version of introductory text for the 

worker categories, they most often only read some of the information presented on the screen, 

omitting important information. For example, some interviewers only read the first sentence and 

did not read the rest of the text indicating to respondents to report workers only once, or to report 

them under the one category that best described the type of work they were hired to do, or the list 

of worker categories. 

Major changes were made to the worker categories introductory text for the July reference period

in three interviews. In one interview, the interviewer only read some of the information on the 

screen. In the two other interviews, the interviewers asked the respondents if they could recall 

their payroll from July, which encouraged these respondents to not report July payroll 

information. For example, in one interview the interviewer said, “Now we have to go through it 

again. Can you remember that far?” To which the respondent answered, “Not without my 

records.” In another interview, the interviewer said, “Now they are asking about the paid workers 

for that week that I just asked you about. I can put question marks if you don't know the correct 

answer.” The interviewer then skipped all of the July questions. 

The biggest change interviewers made to the introduction to the October worker category 

questions in the revised instrument was only reading some of the sentences on the screen or 

paraphrasing the introduction (e.g., “We are going to separate the workers based on the type of 

work they do, total hours worked, and gross wages you paid for that week.”). Some interviewers 

read the text on the screen; however, they changed the last sentence into a question (e.g., “Now 

were they field workers, livestock workers, or managers?”).

Similar to the revised introduction of worker categories for October, when administering the 

introduction for July, interviewers sometimes only read parts of the introductory text or 

paraphrased the text omitting important information (e.g., “They want us to record the type of 

work they were hired to do.”). A couple of interviewers did not read the text but instead 

indicated they would be asking the same questions again (e.g., “I'm going to go through the same 

questions again.”) Finally, a couple of interviewers once again changed the last sentence into a 

question (e.g., “What do they do?”)

3.6 Worker Categories

The questions designed to subsequently classify agricultural workers into the four worker 

categories appear to be deficient on both instruments (See Figures 9 and 10 below for examples 

from the original and revised instruments). When using the original instrument, the open-ended 

question asking what type of work the workers were hired to do in October was only asked as 

worded 40 percent of the time and was administered with a major change 25 percent of the time

(Table 8). Major changes for the iterative questions used to classify workers into the four worker 

categories in October on the revised instrument were much higher (40%), and these questions 
24

were only asked as worded 14 percent of the time. The workers category questions were coded 

as shortcutting equivalently for both instruments.

Figure 9. Original Instrument - Worker Categories
25

Figure 10. Revised Instrument - Number of Supervisors 

Table 8. Interviewer Behavior for Worker 

Categories

Code October Total July Total

Original Revised Original Revised

ES 40% 14% 14% 4%

VER 13% 25% 32% 33%

MC 25% 40% 7% 18%

SC 23% 21% 48% 46%

Total 53 129 44 114

This pattern of administration is demonstrably worse for both instruments for the July worker 

category questions. The rate at which interviewers administered these questions as worded 

dropped significantly from 40 percent and 14 percent in October to 14 percent and 4 percent in 

July, while the shortcutting rates more than doubled to just under 50 percent for both 

instruments. Although the original instrument performed better than the revised instrument, 

neither of the two were administered correctly at an acceptable rate.

When interviewers made major changes to this question on the original instrument, they often 

paraphrased the question and, in doing so, left off important elements of the question. For 

example, instead of asking “What type of work were they hired to do,” they would ask questions 
26

such as: “What were your guys doing?” or “What type of workers?” This is problematic because 

workers may have been performing work during that reference period that does not represent the

type of work they were hired to do and may not reflect their skill set or job series. For example, a 

manager may have been helping repair fences during that reference period. The second most 

common change interviewers made to this question was asking whether employees were 

performing specific jobs: 

“Were all of them field workers or were some supervisors?” 

“Would you consider them livestock workers or manager or all around?” 

As noted above, major changes occurred more often when interviewers were using the revised 

instrument. Across this series of questions, the most common changes interviewers made to these 

questions were not reading the reference period and/or not reading the number of workers 

reported (e.g., “Of the 5 workers, how many were manager or supervisors?” or “Any of them 

hired as a manager or supervisor?”)

Finally, when reading the question that measured the number of crop, nursery and greenhouse

workers, several interviewers inserted “livestock” into the question. For example, one 

interviewer asked, “How many were hired to be crops, nursery or livestock workers?”

3.7 Worker Subcategory Introduction

Next, we examined the manner in which the subcategory questions were administered for each 

worker type. In the revised instrument, an introduction was added to the questionnaire to inform 

respondents that they would be asked to categorize their workers into more specific categories

(Figure 11). 
27

 Figure 11. Revised Instrument - Supervisor Subcategory Introduction

When administering the October worker subcategory introductions, interviewers were less likely 

to read the question as worded as they moved through the instrument (Table 9). Interviewers 

made major changes (supervisor 31%, livestock 43%, field workers 35%) to the subcategory 

introductions and shortcutted (supervisor 39%, livestock 36%, fieldworkers 55%) the 

introduction at high rates.

Table 9. Introduction to Workers Subcategories Interviewer Behavior (Revised Instrument)

Supervisors Livestock Field Workers

Code October July October July October July

ES 31% 0% 21% 0% 10% 0%

VER 0% 0% 0% 0% 0% 0%

MC 31% 13% 43% 10% 35% 23%

SC 39% 88% 36% 90% 55% 77%

Total 13 8 14 10 20 13

No interviewers read the worker subcategory introductions as worded for the July reference 

period. Once again interviewers made major changes to the introduction, but the majority of the 

time they shortcutted the introduction (supervisor 88%, livestock 90%, field workers 77%). 

The types of major changes interviewers made to this introduction were similar across the 

question types and reference periods. The most common major change made to this introduction 

was only reading the first sentence on the screen (Now I have some more questions about the 

[insert number] [insert worker types] you had during the week). The next most common type of 

change made was paraphrasing the text on the screen and then attempting to categorize the 
28

workers instead of asking the questions that followed on subsequent screens. For example, one 

interviewer read part of the first sentence and then added a question to it: 

I: Now I have questions about the 2 managers or supervisors. Do they work directly with the 

people who work with the crops or do they oversee, plan, and coordinate? 

R: I would say directly with. 

The interviewer then used this information to code the workers in the next two questions without 

asking the respondent the survey questions.

In other administrations, the interviewers told the respondents what category they thought the 

workers would fall under (e.g., “So now we're going to break it down even more. Did they work 

on equipment, were they graders and sorters, hand packers, sprayed pesticides or farmworkers 

which sounds to me that may be, manually plant, cultivate and harvest vegetables, fruits, nuts. 

So, you think it might have been that one?”). 

In other interviews, the interviewers paraphrased the text on the screen leaving off important 

information: 

“Now I have more questions. There are 6 categories.” 

“Now we're talking about the 12 livestock workers and we have different categories for 

them.”

In two interviews, the interviewers read all of the text on the screen but did not read the 

categories. 

Finally, one interviewer expressed his frustration with the CATI script and did not read the text 

on the screen: “Good lord! This thing is so convoluted it's not even funny. What type work were 

they doing? Instead of me reading all this stuff off to you.”

3.8 Worker Subcategories

For the worker subcategories questions, we present the data for each worker type by version 

(Figures 12 and 13) and reference period collapsing across subcategories. 
29

Figure 12. Original Instrument - Worker Subcategories

 Figure 13. Revised Instrument - Number of Farmers, Ranchers and Other Agricultural 

 Managers 

3.8.1 Supervisor Subcategory Questions
30

Very few respondents who were administered the original instrument reported having 

supervisors or managers, so this question was only asked a total of 11 times across the two

reference periods. As a result, observations were aggregated across reference periods for both 

instruments for the sake of comparison. 

Interviewers tended to engage in more desirable interviewer behaviors (i.e. ES and VER) when 

asking about the first supervisor subcategory than the second (Table 10). That is, interviewers 

tended to engage in shortcutting less often when they asked about the first subcategory compared 

to the second for both instruments. In general, it also appears that, despite not performing well, 

the revised instrument outperformed the original instrument on supervisor subcategory questions 

as they were asked as worded more frequently and shortcutted less frequently across both 

reference periods.

Table 10. Supervisor Subcategory Interviewer Behavior

Original Revised

Code 1

st Subcategory 2nd Subcategory Managers

1

st Line 

Supervisors

ES 0% 0% 29% 19%

VER 14% 25% 5% 10%

MC 29% 0% 19% 14%

SC 57% 75% 48% 57%

Total 7 4 21 21

In the two instances where major changes were made to the supervisor subcategory question in 

the original instrument, both interviewers only offered the second subcategory to respondents

(e.g., “Now are they first line supervisors?”).

In the revised instrument, when asking about the first subcategory of supervisors (How many of 

the [insert number] hired managers were hired as farmers, ranchers, and other agricultural 

managers), some interviewers combined this question with the second subcategory question 

(How many of the [insert number] hired managers were hired as first-line supervisors of farm 

workers). In the example below, the interviewer only offered first line supervisors to the 

respondent, but then read the description for farmers, ranchers, and other agricultural managers:

I: Were they first-line supervisors?

R: First-line you said?

I: Plan, direct, or coordinate the management or operation of farms, ranches, greenhouses, 

agricultural operations, nurseries, tree farms, or other agricultural establishments. Is it that one?

Another interviewer read the first subcategory question as worded and the respondent indicated 

that this category described what he himself did. The interviewer then moved on to the next 

subcategory question and said, “Okay. So, I guess there’s two choices here. There’s the manager-
31

I have a farmers, ranchers and other agricultural managers and the other one is first line 

supervisors of farm workers. Is it the first one?” To which the respondent replied, “Yeah.” 

The last example of a major change made to the supervisor subcategory questions, is a good 

example of how slight changes to the question wording can significantly affect question 

comprehension. In one interview, the interviewer changed the first subcategory question to: 

“Now the one hired for a manager, was he hired for farm, ranches, or any other managers?” To 

which the respondent replied, “He's just solely for us,” conveying that he is the sole employer of

that individual. 

3.8.2 Livestock Subcategories

The livestock subcategory questions did not perform well on either instrument. For the October 

reference period, the revised instrument outperformed the original instrument as it was 

administered as worded more often (26% vs. 11%) and shortcutted less often (32% vs. 56%)

(Table 11). It is difficult to determine which of the two performed better for the July reference 

period. The revised instrument was shortcutted less often (63% vs. 73%), but was also 

administered exactly as worded less frequently (2% vs. 18%). However, it is worth noting that 

interviewers verified the responses much more often on the revised instrument than the original

instrument for the July reference period (27% vs. 9%). Hence, there is evidence that interviewers 

were at least attempting to make sure that the livestock workers present on the operation were

being classified into the correct subcategories on the revised instrument—which is preferable to 

not administering the question entirely.

Table 11. Livestock Subcategory Interviewer 

Behavior

Code October July

Original Revised Original Revised

ES 11% 26% 18% 2%

VER 22% 19% 9% 27%

MC 11% 23% 0% 8%

SC 56% 32% 73% 63%

Total 9 78 11 60

Only one major change was made to a livestock subcategory question during the administration 

of the original instrument (although these questions were not administered very often). Instead of 

asking “More specifically, which type of work were they hired to do. Report each worker only 

once” and then offering the respondent all five livestock subcategories, the interviewer asked, 

“Was anyone working with equipment or was it specifically with the animals?” To which the 

respondent answered, “90 percent was with the animals.” 

More major changes were seen in the administration of the livestock subcategories when the 

revised instrument was used, and these major changes were more prevalent further along into the 
32

series of questions (Table 11). The most common major change to the livestock subcategory

question on the revised instrument was that interviewers did not read the part of the question that 

asked about the number of livestock workers (i.e., How many of the [insert number] livestock 

workers...]. Instead, they reworded the survey question to ask if respondents had any workers 

that fell under the subcategories: 

“Were there any livestock products grader or sorters?” 

“How about livestock products, graders, and sorters?” 

Or they asked if the workers performed any the specific duties (e.g., “Do they drive or control 

any heavy farm equipment?”). Often, the only thing interviewers offered to the respondent was 

the subcategory itself: 

“Pack or package by hand a variety of products?” 

“Livestock farmworkers. Attend to live farm, ranch, aquacultural animals, including 

cattle, sheep, swine?"

3.8.3 Field Worker Subcategories

Although the revised instrument did a better job of asking about supervisor and livestock worker 

subcategories, this pattern is reversed for field workers (Table 12). When administering the 

October questions, the questions designed to classify field workers into subcategories on the 

original instrument were asked as worded more often than comparable questions on the revised 

instrument (19% vs. 3%). In addition, interviewers tended to engage in shortcutting more often 

when administering these questions on the revised instrument as well (62% vs. 14%). This 

pattern persists for the July reference period. However, the differences were dampened as the 

original instrument was still shortcutted less often (66% vs. 72%) and administered as a 

verification more often (28% vs. 12%). 

Table 12. Field Worker Subcategory Interviewer 

Behavior

Code October July

Original Revised Original Revised

ES 19% 3% 0% 4%

VER 35% 13% 28% 12%

MC 31% 22% 6% 12%

SC 14% 62% 66% 72%

Total 11 120 12 74

Major changes to the field workers subcategory question on the original instrument varied 

substantially. Some interviewers changed the wording of the survey question and did not offer 
33

any of the response options. For example, instead of asking “what type of work were they hired 

to do? Report each worker only once,” interviewers asked respondents what type of work the 

workers were “doing” and did not read any of the response options that were listed: 

“What exactly was he doing?” 

“What type of work were they doing? Farming or harvesting?” 

Similarly, a couple of interviewers changed the question wording to ask what type of work the 

workers were performing and then offered some, but not all, of the response options (e.g., “What 

type of work do they do in the field? Are they operating equipment with the crops? Are they 

doing farm work as far as just by hand, I guess?”)

Other interviewers read the survey question as worded but only offered some of the response 

options: 

“More specifically what type of work were they hired to do? Agricultural equipment?” 

“The next question asks the type of work that they were hired to do. Under farmworkers 

that would be tilling the soil, applying fertilizer, transplanting, weeding, thinning, 

applying pesticides, constructing trellises, repairing fences, and then I also have graders 

and sorters, which that would be to sort and classify by size, weight, color, etcetera, and 

then I have hand packers and packages- packagers and that’s pretty self-explanatory, 

handling a wide variety of products and materials and then we have all other crop, 

nursery and greenhouse. So where would we put those that were working that week of 

October 8th through the 14th?”

Finally, others did not read the survey question but instead just read the response options or some 

version of them: “Is it farm work crop work or graders and sorters crop work?”

When major changes were made to the fieldwork subcategory questions in the revised

instrument, interviewers often did not read the part of the question that asked about the number 

of workers (e.g., How many of the [insert number] crop, nursery, and greenhouse workers): 

“How many were graders and sorters,”

“Any hand packers or packagers?” 

Other interviewers reworded the questions to ask if the workers performed duties listed in the 

survey questions: 

“Does he operate any equipment?” 
34

“Does he pack?”

“Do they do any spraying of pesticides?”

Others did not read the question stem and reworded the worker subcategories in significant ways: 

“And they were just general field workers?” 

“Does he do any nursery, greenhouse?”

“Of the 15, were any workers not included in the other categories? Anyone working on a 

specialized crop?”

“And would they have just been general workers as well?” 

3.8.4 Other Worker Subcategory Questions (original instrument only)

As described above, the “other workers” questions varied greatly between the two instruments. 

Due to this, the “other workers” questions for the revised instrument will be presented separately 

(see section 3.7). The “other workers” subcategories question (Figure 14) was only administered 

11 times using the original instrument. Therefore, we will collapse results across the reference 

periods for all “other workers” questions. The “other workers” subcategory question was 

shortcutted 100 percent of the time.
35

Figure 14. Original Insturment – Other Workers Subcategories

3.9 Original Instrument Number of Workers

Although workers were counted and categorized simultaneously on the revised instrument (e.g.,

Figure 13), respondents were asked to report the number of workers they had for each 

subcategory after they categorized their workers on the original instrument. That is, after they 

reported having a specific type of worker and described what those workers did, they were asked 

to report how many of those workers they had, as shown in Figure 15. They accounted for the 

number of workers they employed in this manner for all worker subcategories.
36

Figure 15. Original Instrument - Number of Livestock Workers

The administration of this question varied significantly by worker subcategory on the original 

instrument (Table 13). Although none of the questions were administered as worded more than 

20 percent of the time, the number of supervisors question was never administered as worded. 

The number of “other workers” question was administered with major changes most often, while 

the number of field workers question was verified most frequently. Lastly, the number of 

supervisors question was shortcutted much more frequently than the other worker types at 82 

percent, followed by the number of livestock workers question at 48 percent. The number of 

field and “other workers” questions were shortcutted about a third of the time.

Table 13. Original Instrument Number of Workers 

Interviewer Behavior by Type

Code Supervisors Livestock Field Other

ES 0% 9% 15% 18%

VER 18% 30% 44% 9%

MC 0% 13% 8% 36%

SC 82% 48% 33% 36%

Total 11 23 61 11
37

There were also differences in the administration of the number of worker subcategories across 

the two reference periods (Table 14). In general, interviewers tended to administer the questions 

as worded more frequently for the October reference period compared to the July reference 

period (15% to 9%), and were less likely to engage in shortcutting as well (33% to 52%). Rather 

than skip over these questions altogether like they did for the July reference period, interviewers 

administered them with major changes instead (17% to 4%). These questions were administered 

as verifications at equal rates across the two reference periods (35%). 

Table 14. Original Instrument Number of Workers Interviewer Behavior by Reference Periods

Code October July

ES 15% 9%

VER 35% 35%

MC 17% 4%

SC 33% 52%

Total 60 46

When making major changes to these questions, interviewers changed these questions in two 

general ways. Some interviewers did not ask how many of the specific type of workers were paid

that week. Instead they asked how many workers the respondent had or how many workers were 

performing the work: 

“And how many workers?” 

“How many of those were working on that?” 

In one interview, the interviewer asked how many workers were supposed to be performing the 

work: “How many field workers were supposed to work that week?”

3.10 Hours Worked

The main differences between the hours worked question in the two versions is the original 

instrument used a generic question for all subcategories reported (Figure 16). Whereas in the 

revised instrument, the hours worked question was customized to each subcategory (Figure 17).
38

Figure 16. Original Instrument - Hours Worked
39

Figure 17. Revised Version – Livestock Equipment Operator Hours Worked

The supervisor hours question was read as worded less often on the original instrument for both 

reference periods (e.g., 43% vs. 93% coded as ES for October) (Table 15). During the 

administration of the October questions, this question was asked with major changes more often

using the original instrument than the revised instrument (43% vs. 7%). Conversely, the hours 

question for the July reference period was asked with major changes less often when using the 

original instrument versus the revised instrument (0% vs. 11%). However, for the July reference 

period, the question was shortcutted at a higher frequency when using the original instrument 

compared to the revised instrument (50% vs. 11%). 

Table 15. Supervisor Workers Hours Worked 

Interviewer Behavior

Code October July

Original Revised Original Revised

ES 43% 93% 25% 33%

VER 14% 0% 25% 44%

MC 43% 7% 0% 11%

SC 0% 0% 50% 11%

Total 7 14 4 9
40

The livestock worker hours were read as worded the majority of the time on both versions (73% 

vs. 78%) (Table 16). However, interviewers made more major changes to this question for the 

October reference period when using the original instrument compared to the revised instrument

(27% vs. 11%). Interviewers read this question as worded less frequently for the July reference 

period. For July, the original question was more likely to be verified (42%), whereas the revised 

question was more likely to be read with major changes. 

Table 16. Livestock Workers Hours Worked Interviewer 

Behavior

Code October July

Original Revised Original Revised

ES 73% 78% 25% 25%

VER 0% 11% 42% 25%

MC 27% 11% 17% 38%

SC 0% 0% 17% 13%

Total 11 18 12 16

The field workers hours question did not perform well on either instrument (Table 17). It was 

read as worded only 50 percent of the time on the original instrument and 40 percent of the time 

on the revised instrument for the October reference period, and even less often for the July 

reference period (26% original vs. 40% revised). The question was read with major changes at a 

high rate regardless of the questionnaire. However, the revised version was read with major 

changes at a higher rate than the original version for the October reference period. For the July 

reference period, both versions of this question were administered with major changes at a high 

rate. However, the original hours question was shortcutted more often than the revised hours 

question (22% vs. 0%). 

Table 17. Field Workers Hours Worked Interviewer 

Behavior

Code October July

Original Revised Original Revised

ES 50% 40% 26% 40%

VER 24% 0% 19% 27%

MC 24% 60% 33% 33%

SC 3% 0% 22% 0%

Total 34 25 27 15

The other workers hours question was only administered eight times for the October reference 

period and three times for the July reference period. Interviewers asked this question as worded 

55 percent of the time and verified the question 9 percent of the time across both reference 

periods. It was shortcutted 27 percent of the time and had a major change made to it 9 percent of 

the time. 
41

Since the hours worked question was exactly the same for each subcategory in the original 

instrument and was very similar (with the exception of the subcategory name) for each 

subcategory in the revised instrument, the results are presented for the types of major changes 

made to these questions by version type and reference period and collapsing across subcategories

of workers. 

When interviewers made a major change to the October hours question using the original 

instrument, it most commonly consisted of not asking for the “total” number of hours. This is 

especially problematic when more than one worker is reported under the subcategory as it does 

not convey to the respondent that the question is asking for the total for all workers and could 

produce inaccurate results when calculating a wage rate. Additionally, a few interviewers asked 

for the average number of hours instead of the total (e.g., “How many hours would they- did they 

average per week during that particular time period?”).

The second most common type of major change made to this question was not reading the 

number of workers reported (e.g., “How many total hours did your workers work?”) The number 

in the question serves to remind respondents of the number of workers for which they need to 

provide a total. 

Other changes made included replacing “did” with “would” (e.g., “How much would you say the 

one on the salary worked?”) and asking respondents if they could recall how many hours their 

workers worked (e.g., “Do you know what the total hours were that the 16 workers worked?”). 

Asking this question in this way presumes that the respondent may not know the answer to the 

question and may encourage respondents to answer “don’t know.” In fact, when interviewers 

changed the wording to “do you know...” in this question respondents always replied “no.” 

In other interviews, the interviewer asked the original October hours question in a leading way: 

“Do both those workers work 40 hours a week?” 

“How many total hours did your workers work? Was it a 40-hour work week or more?” 

In one interview, the interviewer asked the question as worded and then added an additional 

question to the end of the question: “How many total hours did this one worker work? Is he a 

salaried employee?”

In another interview, the interviewer changed the question to ask about the number of hours 

worked per day: “How many hours a day did these people work?”

Finally, interviewers often made a combination of the changes described above: 
42

“How many hours would they -- did they average per week during that particular time 

period?” 

“How many hours a week did they work?” 

The major changes interviewers made to the July hours question in the original instrument were 

similar to the changes made to the October hours question, and interviewers often made more 

than one major change to the question. 

Most often interviewers did not ask for the total number of hours and/or did not read the number 

of the workers: 

“How many hours did they work that week?” 

“How many hours?” 

Similar to October, interviewers also replaced “did” with “would” and asked respondents if they 

knew how many hours their workers worked (e.g., “And you don't have any idea on the top of 

your head how many hours they worked that week do you?”) Other interviewers also presumed 

that this information would not be accessible to the respondent but asked the question in slightly 

different ways. For example, one interviewer asked: “Now do you remember, ‘cause it asks for 

the total hours but if you can give me how many hours one worker worked if they all work the 

same amount of hours, I can multiply it.” The respondent indicated that his workers did not all 

work the same number of hours that week. Another interviewer reworded the question in a way 

that encouraged the respondent to provide an estimate rather than an exact figure: “For that week 

- we take guesstimates- how many hours did they work?” As seen in other questions, one 

interviewer asked the question in a manner that encouraged the respondent to report the same 

number of hours as he did in the previous reference period: “Will they be working the same 

hours that week?”

Finally, one interviewer asked about the number of hours worked per day: “And they worked 

how many hours a day?”

The worker subcategory was added to the hours question in the revised questionnaire to remind 

respondents which group of workers the current question was asking about. However, when 

major changes were made to this question for the October reference period, the most common 

change made was not reading this addition (e.g., “How many total hours did these 161 workers 

work?”) Moreover, when a couple of interviewers read this question, they read the incorrect 

subcategory to the respondent. For example, instead of asking how many total hours the four 

pesticide handlers worked, the interviewer asked, “How many total hours did these 14 packers 

work?”
43

The next most common change found was not reading “total” when administering the question. 

One interviewer also reworded the question to ask how many hours each worker worked: “Now 

they do ask about how many hours each of those workers worked last week?”

Instead of asking for the exact total for the reference week, some interviewers reworded the 

question in a way to encourage respondent to provide estimates or averages: 

“About how many hours would that one person average for that time period?”

“How many total hours would they work in a week?” 

“How many total hours did those 12 workers work approximately?” 

“How many hours do you think those three worked during that week of October 8th14th?”

Similar to the original instrument, when administering the revised hours question for October, 

some interviewers asked respondents if they knew how many hours their workers worked (e.g., 

“Do you know about how many hours they worked?”), to which some respondents replied “no.”

In a few interviews, the interviewer did not reference the number of workers listed under the 

subcategory. 

Finally, in some interviews, the interviewers asked the question in a leading way: 

“Do they work 40 hours a week? 

“How many total hours did these six crops, nursery and greenhouse equipment operators 

work? Like did they work all day every day?” 

When administering the July hours question using the revised instrument, the most common 

major change made was not reading the worker subcategory. In two interviews, the interviewer 

changed the wording of the worker subcategory. For example, instead of asking first line 

supervisors, one interviewer asked: “How many hours do you think... well he was a foreman... 

how many hours would you say that he worked?” In another interview instead of asking about 

livestock graders and sorters, one interviewer asked: “Your livestock workers, about how many 

hours did they work?” The next most common major changes made were not asking about the 

number of workers listed under the specific worker codes and omitting the word “total” from the 

question or asking for an average (e.g., “How many average hours did they work?”). Often 

interviewers made several of these major changes when administering the question (e.g., “And 

how many hours did they work?”).
44

Finally, as in the original instrument and the previous reference period for the revised instrument, 

interviewers reworded the question in ways that implied that the response may not be known. For 

example, in a couple of interviews the interviewers asked how many hours the respondent 

thought their workers had worked (e.g., “Total hours for the equipment operator. How many do 

you think he worked that week - that July week?”). In other interviews, the interviewer asked the 

respondents if they knew how many hours their workers worked (e.g., “During that week do you 

have any idea of how many hours they worked?”). 

3.11 Gross Wages

Like the hours question, the gross wages question was exactly the same for each subcategory in 

the original instrument (Figure 18) and was very similar (with the exception of the subcategory 

name) for each subcategory in the revised instrument (Figure 19). Here the results are shown for

the types of major changes made to these questions by version type and reference period and 

collapsing across subcategories. 

Figure 18. Original Instrument Gross Wages
45

Figure 19. Revised Instrument Gross Wages

The supervisor gross wages question was read as worded at a very low rate regardless of 

instrument across both reference periods (Table 18). Interviewers read this question as worded 

20 percent of the time using the original instrument and 36 percent of the time using the revised 

instrument for the October reference period. In addition, this question was never read as worded 

using either instrument for the July reference period. The majority of the time the question was 

asked with major changes. However, for the July reference period, it was shortcutted 50 percent 

of the time when interviewers were using the original instrument.

Table 18. Supervisor Workers Gross Wages Worked Interviewer Behavior

Code October July

Original Revised Original Revised

ES 20% 36% 0% 0%

VER 20% 0% 25% 29%

MC 60% 57% 25% 57%

SC 0% 7% 50% 14%

Total 5 14 4 7

The livestock gross wages question for the October reference period was administered as worded 

more about half the time (original: 50% and revised: 56%) (Table 19). However, it was still 

administered with major changes at a high rate (33% using each version). For the July reference 

period, this question was never read as worded using either version. Interviewers made major 
46

changes (original: 25%, revised: 40%) and shortcutted (original: 42%, revised 20%) this question 

at high rates. 

Table 19. Livestock Workers Gross Wages Interviewer Behavior

Code October July

Original Revised Original Revised

ES 50% 56% 0% 0%

VER 17% 11% 33% 40%

MC 33% 33% 25% 40%

SC 0% 0% 42% 20%

Total 12 18 12 15

Similar to the supervisor and livestock gross wages questions, the field workers gross wages 

question was not read as worded very often (Table 20). When asking about the October reference 

period, interviewers made major changes to the gross wages question 53 percent of the time 

using the original instrument and 80 percent of the time using the revised instrument. Major 

changes were also made to the question when asking about the July reference period (original: 

33%, revised 57%). Interviewers also shortcutted this question for the July reference period at a 

high rate when administering the original instrument (26%). 

Table 20. Field Workers Gross Wages Interviewer Behavior

Code October July

Original Revised Original Revised

ES 32% 8% 15% 14%

VER 9% 12% 22% 14%

MC 53% 80% 33% 57%

SC 6% 0% 26% 14%

OTH 0% 0% 4% 0%

n 34 25 27 14

Once again, the other workers gross wages were collapsed across both reference periods due to 

the limited amount of times it was administered (8 times for October and 3 times for July). 

Across the two reference periods, this question was asked as worded 27 percent of the time and 

verified 9 percent of the time. It was shortcutted 27 percent of the time and was administered 

with a major change 36 percent of the time. 

In all interviews where interviewers made a major change to the wage question for the October 

reference period using the original instrument, they never read the first sentence in this question 

(We would now like to determine how much you paid this one worker). The next most common 

major change made to this question was combining the wage question with the interviewer note. 

This note appears below the question and indicates to interviewers that they can input different 

forms of payment on the following CATI screen ([ENUM] Unit that will be asked in the next 
47

question. Following are the possible units that can be used to answer the next question: total 

amount paid, hourly wage, average weekly salary per worker, average monthly salary per 

worker, average annual salary per worker). This change was made by almost half of the 

interviewers who made a major change to the wage question for the October reference period. 

Examples included: 

“What’s their hourly wage?”

“And you paid them all that week. What was their hourly wage, their weekly wage, their 

monthly wage?”

Although some interviewers asked respondents how much their workers were paid, most asked 

what their hourly rate was. This is problematic because respondents often pay their workers 

different hourly rates, making this rewording of the question difficult for respondents to answer. 

For example, in one interview the interviewer asked: "And you paid them all that week. What 

was their hourly wage, their weekly wage, their monthly wage?" The respondent replied that 

some of his workers are paid $11 per hour, one is paid $12 per hour, another is paid $13 per hour 

and two are paid $16 per hour. Instead of asking for the total amount paid to those workers that 

week, the interviewer left a note for the data reviewers. In other cases, when interviewers asked 

how much the workers were paid per hour, the respondents corrected the interviewers as their 

workers were paid a salary as demonstrated in the example below:

I: What did each worker receive as a wage? Was it an hourly wage?

R: My laborers are salary. One employee gets $3,000 a month, I get $1,300 I think, and my dad 

$1,000 a month.

Other changes made by the interviewers included not reading “that week” (e.g., “What were the 

wages for the 19 workers?”), and not reading the number of workers listed under the worker 

category. Sometimes interviewers omitted multiple things from the question when reading it 

(e.g., “What was the wages?”).

As with the hours questions, in some interviews the interviewer changed the question in a way 

that implied that the respondent would not know the answer to the question (e.g., “Do you 

happen to know what her wage was for that week?”). Again, this is problematic because it may 

encourage some respondents to satisfice and say no. In fact, in some instances, when asked the 

question in this way, respondents answered “no.” Similarly, in one interview the interviewer 

asked for the average wages: “What would you say were the average wages for that person that 

week?”

The changes made to the July wages questions using the original instrument were very similar to 

those made to the October wages questions. Among the interviewers who made major changes to 

the July gross wages questions, all but one omitted the first sentence in this question. 

Interviewers also combined this question with the interviewer note regarding units of pay (e.g., 
48

“And what were the wages for that week per hour?” “How much did they get paid an hour?”). 

Interviewers made changes similar to those made to the October wages questions, including not 

reading “that week,” and not reading the number of workers listed under the worker category. In 

one interview, the interviewer asked the question in a leading way that implied the respondent 

did not know the answer: “What did he get? The total wages? If you know what that is.” In 

another interview, the interviewer asked if the July wages were the same as October: “Now 

would they all be under the same salary this time?” Finally, some interviewers altered the 

question significantly leaving out multiple key terms:

“And again, the pay?” 

“What was the pay for the 12-hour workers?”

“Can you share what your cost would have been as far as one of the two for the time you 

had them hired?”

“We would like to determine how much you paid those two. The wages or whatever.”

In the revised instrument, the worker type was added to the survey question to clarify which 

group of workers the question was asked about. However, in the majority of interviews where 

interviewers made major changes to the October wage question, the interviewer did not read the 

worker type. Additionally, in three administrations of the question, the interviewer read the 

wrong worker type or changed the wording of the worker type. This is illustrated in the 

following exchange: 

I: About how much did you pay the 17 graders and sorters?

R: I don't have any graders or sorters.

Other changes were similar to what was found in the administration of the original instrument.

Almost no interviewers read the first sentence in this question. However, in one interview that is 

all the interviewer read: “Now we would like to determine what he was paid during that week.” 

As in the original instrument, many interviewers also did not read the number of workers or “that 

week.” 

Interviewers also combined this question with the interviewer note regarding units of pay (e.g., 

“And what was his hourly wage?”). Above we note how this is problematic for respondents who 

pay their workers different hourly rates. Additional issues with this wording change emerged 

when this type of change was made to the revised question. First, interviewers often assumed 

that respondents pay their workers by the hour. However, some respondents paid their workers

monthly, a salary, or by piece. 
49

This led to additional unnecessary exchanges between the interviewer and respondent and made 

the question more difficult for the respondent to answer. This is illustrated in the following 

exchange: 

I: And what was the wage? Do you pay them by the hour or by the week or?

R: They're paid by the bin... they're apple pickers.

I: “Approximately, how much do they make by the hour or how much does it average to?

R: (Respondent asks husband and he yells, “I don't know!”) I don't know the answer to that.

Another reason this change is problematic is respondents do not always pay their employees the 

same unit of pay. That is, respondents will pay some workers a salary and others an hourly rate. 

For example, in one interview the interviewer asked: “Then they determine the approximate 

wages that you paid those workers. Did you pay like per hour, salary, however?” The respondent 

then replied, “Both.” 

In a couple of interviews, the interviewer changed the question to ask the respondent if they were 

paying all their workers the same wages (e.g., “What were their wage? Are you paying them all 

the same wage?”). Instead of providing the total wages paid to the workers the respondent 

provided wage rates for different workers that then had to be computed into total wages (note 

that this respondent did not exclude contract labor in his response): 

I: What were the wages for the two workers? Did they make different wages?

R: Oh, yeah

I: How much did they make?

R: The good usable guy, I'm paying him 17 an hour and the other guy I pay through the temp 

agency... they pay him 12 but it's costing me about 15.

As in the administration of the original instrument, some interviewers asked respondents if they 

knew how much they paid their employees that week, to which some respondents replied “no” 

(e.g., I: “Do you know how much they were paid that week?” R: “I do not.”).

An issue not seen in other administrations of this question using the original instrument was that 

several interviewers asked respondents for an approximate or estimate:

“Approximately, how much did you pay those 17 equipment operators that week?” 

“They want an approximate idea of what they made - each of them.” 

“What were the wages for these 4 workers? Can you estimate it?”). 

Similarly, a couple of interviewers asked for an average wage rather than a total (e.g., “Okay and 

his average wage?”)
50

Finally, instead of asking this question as worded, one interviewer asked if the respondent 

wanted to refuse to answer this question: “You want to refuse the pay question for this one too?”

Interviewers made similar changes to the revised wage question for the July reference period. All 

but one of the interviewers who made major changes to this question did not read the first 

sentence in this question. Many interviewers also did not read the worker type or the number of 

workers when administering this question. A couple of interviewers also did not read “that 

week.”

As in other administrations of this question, interviewers combined this question with the 

interviewer instruction regarding unit of pay. Once again, some interviewers assumed the 

workers were paid an hourly rate and in a couple interviews, the respondent had to correct them 

(e.g., I: “What do you pay them an hour?” Respondent said they pay daily.)

Interviewers continued to assume respondents may not know the answer to this question. In 

some cases, they asked the respondent if they knew (e.g., “Do you know what their wages were 

that week?”) and in other cases they asked for an approximate wage: “Approximately, what did 

they get paid - that equipment operator for that week?”

In a couple interviews, the interviewer once again asked for an average wage instead of total 

wages for that week:

“They want to know the average wages for these workers.”

“How much was he paid do you think on average per week” 

One interviewer asked if the respondent paid the workers the same amount as they did in the

previous reference period: “And was it the same pay?”

Interviewers also often made several major changes to the survey question at once:

I: Do you remember what you paid them? How much per hour? 

R: I don't pay by the hour. I pay by the piece.

I: The piece? 

The respondent then clarified that he pays by the bushel, adding that he pays a contractor, but he 

does not know how much the contractor pays the workers. He said he pays six dollars and a half' 

a bushel. Note once again that contract labor was not excluded from his response. 

Finally, one interviewer combined this question with the hours question: “What wage and total 

hours for those two?”

3.12 Revised Instrument Other Worker Questions
51

As mentioned above in the revised instrument the other worker questions were revised 

significantly. The results of those changes separate from the other worker types in the original 

instrument are shown in this section. 

3.12.1 Contract, Custom and Value-Added Workers (revised only)

In the revised instrument, after collecting information on any supervisor, livestock, or field 

workers the interviewers asked a series of questions about other worker types, beginning with 

contract, custom, retail, and value-added workers (Figure 20). 

 Figure 20. Revised Instrument: Contract, Custom, Retail and Value-Added Workers

This question was only administered nine times across the two reference periods. Interviewers 

only asked this question as worded three times, verified the response once, asked the question 

with major changes four times, and shortcutted the question one time. 

Interviewers made different types of major changes to this question. A couple of interviewers did 

not read the entirety of the question and only asked about specific types of workers. For example, 

one interviewer only asked about contractor labor: “Is it contract labor or is he under your 

payroll?” Another interviewer only asked about custom and contract workers. A third 

interviewer changed the question to include office workers: “The remaining two workers, what 

did we say those were? Office bookkeeper, custom work, retailer were – it says how many of the 

two workers were custom work, retail or (voice trails off)?” The respondent replied to this: 

“There are two of us that are office workers.” It appears the fourth interviewer perceived this 
52

question to be an introduction screen. Instead of reading the question as worded she said, “Now 

let's discuss the nine remaining workers” and then entered “1”, which is how interviewers 

typically advance from an introduction screen. 

Respondents who indicated that they had contract, custom or value-added workers on the 

payroll, were then asked how many of those they had (Figure 21). Two respondents should have 

been asked this question. In one interview, the interviewer verified the answer; and in the other 

interview, the interviewer shortcutted this question. The interviewer who verified the answer was 

the same interviewer who perceived the previous question to be introductory text and was clearly 

confused by this question: “How many of the nine workers were on custom or contract worker, 

retail workers or value-added workers? Zero right? I don’t know why they’re doing this. This is a 

weird one, right? And it’s different than it was.” 

 Figure 21. Revised Instrument: Number of Contract, Custom, Retail, and Value-Added 

 Workers

Respondents, who received the previous question, were instructed that this survey did not 

include these types of workers and interviewers had to go back and remove these workers from 

their worker counts (Figure 22). Two respondents should have been administered this instruction. 

One interviewer administered it with major changes and another interviewer shortcutted these 

instructions. The interviewer who made a change to this question was the same interviewer who 

perceived the initial question in this series as introductory text. When she got to this question she 

changed the instruction to “The survey does not include these type of workers, so we will 

exclude those - yeah okay” and moved on to the next screen. 
53

Figure 22. Revised Instrument: Exclusion Instruction

3.12.2 Agricultural Inspectors (revised only)

The agricultural inspector question (Figure 23) on the revised instrument was only administered 

eight times across the two reference periods. Therefore, the results shown here are for the two 

reference periods combined. This question was read as worded two times, verified three times 

without receiving previous information from the respondents (e.g., “Were any of the 9 

inspectors... no right?”), read with major changes twice, and shortcutted twice. In the one 

instance where this question was read with major changes, the interviewer changed the question 

to “In a sentence, what describes the work the five were hired for?” 
54

 Figure 23. Revised Instrument: Number of Agricultural Inspectors

Only one respondent was filtered to the question that asked how many agricultural inspectors 

were on payroll (Figure 24). The interviewer shortcutted this question. Subsequently, no 

respondents received the subsequent questions regarding agricultural inspectors.

 Figure 24. Revised Instrument: Number of Agricultural Inspectors
55

3.13 Remaining Workers (revised instrument only)

If the respondent had not accounted for all workers, they were asked a series of questions 

regarding the remaining workers beginning with an opened-ended question that asked about what 

the remaining workers were hired to do (Figure 25). This question was only administered nine

times across the two reference periods. Therefore, the results presented here were collapsed

across reference periods. This question was read as worded one time and verified twice. 

Interviewers read this question with major changes two times and shortcutted the question four 

times. In the interviews where the interviewer made a major change to this question, they did not 

read the full text on the screen. One interviewer asked: “What were those two hired to do?” The 

other interviewer asked, “So they were hired to work with the bees?” 

 Figure 25. Revised Instrument: Remaining Workers

The next question asked how many hours the remaining workers worked (Figure 26). In the nine 

times that this question was administered, four interviewers read the question exactly as worded.

In five administrations, the interviewers made major changes to the question. One interviewer’s 

behavior was marked as “other” for this question. 
56

Figure 26. Revised Instrument Total Hours Worked by Remaining Workers

In four of the interviews where the interviewer made a major change, the interviewers asked the 

respondent if they knew how many hours those remaining workers worked (e.g., “Do you know 

how many hours they worked?”). In two of these interviews, the respondent indicated that they

did not know how many hours the workers worked. In the fifth interview where an interviewer 

made a major change, the interviewer failed to read “total” when administering the question. 

Next, respondents were asked to report the wages paid to these remaining workers (Figure 27). 

In one interview, the interviewer read the question exactly as worded, and in another interview,

the interviewer verified the answer. Interviewers administered this question with a major change

five times. Interviewers made similar changes made to other wage questions in the instrument. 

One interviewer asked if the respondent was paying the remaining workers the same wages: 

“What were their wage? Are you paying them all the same wage?” Another interviewer asked 

how much the respondent paid the remaining worker per hour: “How much does she make per 

hour?” One interviewer asked, “How much did you pay that worker,” leaving off the 

introductory text and “that week.” Another interviewer indicated that the question was asking for 

an average rather than a total: “They want to know the average wages for these workers.” 

Finally, one interviewer asked if the respondent could recall what the wages for the remaining

worker were: “Do you know about what their wages were?”
57

Figure 27. Revised Instrument: Wages Paid to Remaining Workers

3.14 Total Number of Workers Confirmation (original instrument only)

In the original instrument, respondents are asked to confirm the total number of workers for the 

reference week after all detailed information about the agricultural workers for the reference 

period were provided (Figure 28). This helped ensure that all agricultural workers were

accounted for. 
58

Figure 28. Original Instrument: Total Number of Workers Verification

This question was read as worded slightly more often for the October reference period (43%) 

than for the July reference period (30%) (Table 21). Interviewers shortcutted this question at 

high rates when administering this question for both reference periods (October 40%, July 58%). 

Table 21. Original Instrument Number of Workers 

Confirmation

Code October July

ES 43% 30%

VER 10% 7%

MC 7% 4%

SC 40% 58%

N 30 27

3.15 150-Day Question

Following each reference period, interviewers are tasked with determining whether employees 

reported for the reference week work for 150 days or more during the calendar year. This is 

accomplished with two questions shown in Figures 29 and 30 below. The first question asked

respondents to specify how many of the workers they reported will be paid for 150 days or more. 

The second question asked how many will be paid for 149 days or less. The same questions are 

asked on both versions of the instruments. 
59

 Figure 29. Number of Workers Paid for 150 Days or More

 Figure 30. Number of Workers Paid for 149 Days or Less

The question asking if workers will be paid for 150 days or more was administered as worded 

over 50% of the time for the October reference period, regardless of what instrument was used 

(Table 22). Both instruments were administered with major changes approximately 40% of the 

time in October. Very rarely were they administered as a verification or shortcutted.

The 150 days question was read as worded less often for the July reference period than for the 

October reference period. For this reference period, the question was only read as worded about 

a fifth of the time on both instruments and verified over 20 percent of the time. There were

differences in the instrument with respect to major changes and shortcutting. When 
60

administering the original instrument, this question is more likely to be shortcutted (42%) than 

administered with a major change (12%). However, the opposite is true when administering the 

revised instrument (23% vs. 35%).

Table 22. 150 Days or More Interviewer Behavior

Code October July

Original Revised Original Revised

ES 50% 56% 21% 19%

VER 6% 3% 24% 23%

MC 39% 41% 12% 35%

SC 6% 0% 42% 23%

Total 36 32 33 26

Interviewers made similar major changes across the two reference periods when administering 

the 150 days question. The most common major changes were omitting important pieces of the 

question such as “during 2017,” “by this operation,” and replacing the word “paid” with “work.” 

Often, interviewers made several of these changes at once (e.g., “Do all of those 12 - for that 

week - work more than 150 days per year?”). Other changes commonly made included not 

reading the number of workers when asking the question (e.g., “Will any of them be paid for 150 

days or more of work?”). Interviewers also changed the question to ask “more than 150 days” 

(e.g., “About how many of those five would work more than 150 days during 2017?”); or

combined this question with the next one (e.g., “Does he work more than 150 days out of the 

year or would he work less than 150?”). This changed the question to ask if the employees work 

full-time (e.g., “For the five employees, how many are full-time and will work 150 days or 

more?”) or conversely if the workers were temporary (e.g., “Any of those workers work for 150 

days or more or how many of them worked for you? Are they just temporary?”). Some 

interviewers also asked this survey question in a leading way: “Neither one of these gentlemen, 

work 150 days or more?” “They would not have worked more than 150 hours this year?” (Note 

this interviewer also changed days to hours). 

The question asking if workers are paid for 149 days or less fared much worse. It was rarely 

administered as worded across both reference periods, regardless of the instrument used (Table 

23). In addition, shortcutting was much higher for the original instrument than the revised 

instrument across both reference periods. As expected, shortcutting was also higher in July than 

in October (58% vs. 72% for the original instrument and 17% and 54% for the revised 

instrument).
61

Table 23. 149 Days or Less Interviewer Behavior

Code October July

Original Revised Original Revised

ES 8% 0% 6% 0%

VER 25% 58% 17% 23%

MC 8% 25% 6% 23%

SC 58% 17% 72% 54%

Total 24 12 18 13

Similar to the previous question, when major changes were made to this question, interviewers 

most often omitted important pieces of the question such as “during 2017,” “by this operation,” 

and replacing the word paid with “work,” and the number of workers reported (e.g., “How many 

of them will work 150 days or less?”). 

4. CONCLUSION AND RECOMMENDATIONS

This research report compares the administration of questions from two Agricultural Labor 

Survey CATI instruments that are designed to classify agricultural workers into various worker 

categories. Behavior coding was performed on the administration of the survey questions to 

identify problematic questions across the instruments. Survey questions that are administered 

with major changes or that are not administered at all at a rate higher than 15 percent are 

considered problematic (Fowler 2011; Oksenberg, Cannell and Kalton 1991). In general, neither 

of the instruments was administered appropriately at an acceptable rate—a pattern that was 

exacerbated in the administration of questions for the second reference period. Interviewers often 

made major changes to the survey questions, which altered the construct being measured. 

Respondents were often encouraged by interviewers to report the same information for the 

second quarter as they had for the previous quarter. Interviewers also shortcutted questions 

(entering responses without asking the survey question) at alarmingly high rates, often exceeding 

50 percent and, in some cases, shortcutting the questions more than 70 percent of the time. 

When directly comparing the performance of both instruments to one another, a clear case can be 

made for the revised instrument. For the most part, when it came to questions designed to 

classify agricultural workers into major categories and subcategories, the revised instrument was 

administered as worded more often than the original instrument. Moreover, as a whole, the 

results showed that interviewers were also less likely to engage in shortcutting with the revised 

instrument. However, introductory screens added and edited on the revised script were often not 

read as worded. These screens were read as worded for the first reference period between 10 and 

31 percent of the time, and between 0 and 3 percent of the time for the second reference period. 
62

Based on these results, it is recommended moving forward with the format of collecting number 

of workers and type of workers on the revised script. However, a careful review of the entire 

revised CATI script is recommended to reduce the amount of text interviewers are required to 

read to respondents and to improve the flow of the instrument. Additionally, reviewing these 

results in conjunction with the field test data may better inform the performance of these CATI 

questionnaires and their impact on data quality. 

Another recommendation is to continue using the Verint system to conduct behavior coding and 

interviewer monitoring. The results from this study indicate that interviewers deviate from the 

CATI scripts at very high rates, which could have a substantial impact on the data collected. 

And, it is recommended that interviewers be retrained on the importance of reading survey 

questions as worded and discouraged from engaging in conversational interviewing to reduce 

interviewer variation.

One limitation of this research was that images of the interviewers’ screens were not available 

during behavior coding. This made it very difficult for the behavior coders to determine which 

question the interviewer was attempting to administer, particularly when the interviewers 

deviated from the script. This slowed down the behavior coding process, as behavior coders 

often had to listen to interviews several times to determine the interviewers’ location in the 

instrument and opened the coding up to possible error. 

Despite these limitations, there is sufficient evidence to make the following recommendations: 

1) Use the revised method of collecting number of workers and worker type on the 

Agricultural Labor CATI script for the April 2018 data collection.

2) Review the revised CATI script to identify additional ways the script can be improved for 

CATI.

3) Examine the number and type of agricultural workers reported on this survey cycle and 

use this information along with these behavior coding results to inform the questionnaire 

design of future Agricultural Labor surveys.

4) Conduct enumerator training that focuses on the importance of reading survey questions 

as worded and discourages the usage of conversational interviewing to reduce interviewer 

variation.

5) Continue to conduct behavior coding on future cycles of the Agricultural Labor Survey.

6) Expand the usage of behavior coding to more data collection centers.

7) Expand the usage of behavior coding to evaluate other surveys conducted at NASS.

5. REFERENCES

Cohen, J. (1960), “Coefficient of Agreement in Nominal Scales,” Educational and Psychological 

Measurement, 20, 37-46.
63

Fowler, F. J. (2011), “Coding the Behavior of Interviewers and Respondents to Evaluate Survey 

Questions,” in Question Evaluation Methods: Contributing to the Science of Data Quality eds. J. 

Madans, K. Miller, A. Maitland, and G. Willis, Hoboken, NJ: John Wiley & Sons, pp. 5-21.

Landis, J. R. and Koch, G. G., (1977), “The Measurement of Observer Agreement for 

Categorical Data,” Biometrics, 33, 159-174. 

Oksenberg, L., Cannell, C., and Kalton, G., (1991), “New Strategies for Pretesting Survey 

Questions,” Journal of Official Statistics, 7, 349-365.

Ongena, Y. P., and Dijkstra, W., (2006), “Methods of Behavior Coding of Survey Instruments,” 

Journal of Official Statistics, 22, 419-451.

Sloan, R., (2017), “Agricultural Labor Survey Cognitive Interview Report,” Washington, DC: 

National Agricultural Statistics Service. 

Sloan, R., (2018), “2017 Agricultural Labor Survey Cognitive Interview Report,” Washington, 

DC: National Agricultural Statistics Service.

Sloan, R. E., Pick, K. M., Sirkis, R., and McGovern, P. D. (forthcoming). “Considering Lessons 

Learned from a Bridge Study for a Business Survey,” Washington, DC: National Agricultural 

Statistics Service. 
64

APPENDIX A: Original CATI Script

Ag Labor – Original Questionnaire – October 2017
65
66
67
68
69
70
71
72
73
74

BEGIN CODING

S1Q1 (largest number)

S1Q2 (H2A)
75

S2Q1 (October labor)

S2Q2 (Labor intro)
76

S2Q3a (Oct Type – first time asking) (Interviewer should cycle back to this for each worker type)
77

OCTOBER FIELD WORKERS

S2Q4a1 (Oct Field Workers-Type) (each type of field workers is reported on different row. Interviewer 

should cycle back through this battery of questions for each type. For additional rows, replace the 

number on the end of the variable name. e.g. S2Q4a2, S2Q4a3, S2Q4a4, S2Q4a5.
78

S2Q5a1 (Oct field workers- describe) (S2Q5a2, S2Q5a3, S2Q5a4, S2Q5a5)

S2Q6a1 (Oct Field workers – number) (S2Q6a2, S2Q6a3, S2Q6a4, S2Q6a5)
79

S2Q7a1 (Oct Field workers – hours) (S2Q7a2, S2Q7a3, S2Q7a4, S2Q7a5)

S2Q8a1 (Oct Field workers – wages) (S2Q8a2, S2Q8a3, S2Q8a4, S2Q8a5)
80

S2Q9a1 (Oct Field workers – unit) (S2Q9a2, S2Q9a3, S2Q9a4, S2Q9a5)

If respondent reported total wages the following question is skipped and then go to any more

S2Q10a1 (Oct Field workers - Confirm wage) (S2Q10a2, S2Q10a3, S2Q10a4, S2Q10a5)
81

S2Q11a (Oct Any more - 1)

S2Q3b (Oct Type)
82

OCTOBER LIVESTOCK WORKERS

S2Q4b1 (Oct Livestock workers – Type) (each type of livestock workers is reported on different row. 

Interviewer should cycle back through this battery of questions for each type. For additional rows,

replace the number on the end of the variable name. e.g. S2Q4b2, S2Q4b3, S2Q4b4, S2Q4b5)
83

S2Q5b1 (Oct Livestock – describe) (S2Q5b2, S2Q5b3, S2Q5b4, S2Q5b5)

S2Q6b1 (Oct Livestock workers – number) (S2Q6b2, S2Q6b3, S2Q6b4, S2Q6b5)
84

S2Q7b1 (Oct Livestock workers – hours) (S2Q7b2, S2Q7b3, S2Q7b4, S2Q7b5)

S2Q8b1 (Oct Livestock workers – Wages) (S2Q8b2, S2Q8b3, S2Q8b4, S2Q8b5)
85

S2Q9b1 (Oct Livestock Workers – unit) (S2Q9b2, S2Q9b3, S2Q9b4, S2Q9b5)

If respondent reported total wages the following question is skipped and then go to any more

S2Q10b1 (Oct Livestock workers - Confirm wage) (S2Q10b2, S2Q10b3, S2Q10b4, S2Q10b5)
86

S2Q11b (Oct Any more)

S2Q3c (Oct Type)
87

OCTOBER SUPERVISOR WORKERS

S2Q4c1 (Oct Supervisors – Type) (each type of supervisor is reported on different row. Interviewer 

should cycle back through this battery of questions for each type. For additional rows, replace the 

number on the end of the variable name. e.g. S2Q4c2)
88

S2Q5c1 (Oct Supervisor – describe) (S2Q5c2)

S2Q6c1 (Oct Supervisor workers – number) (S2Q6c2)
89

S2Q7C1 (Oct Supervisor hours) (S2Q7C2)

S2Q8C1 (Oct Supervisor wages) (S2Q8C2)
90

S2QQ9C1 (Oct supervisor unit) (S2QQ9C2)

If respondent reported total wages the following question is skipped and then go to any more

S2Q10C1 (Oct Confirm wage) (S2Q10C2)
91

S2Q11C (Oct any more)

S2Q3d (Oct Type)
92

OCTOBER OTHER WORKERS

S2Q4d1 (Oct Other – Type) each type of other worker is reported on different row. Interviewer should 

cycle back through this battery of questions for each type. For additional rows, replace the number on 

the end of the variable name. e.g. S2Q4d2, S2Q4d3, S2Q4d4)
93

S2Q5d1 (Oct Other – Describe) (S2Q5d2, S2Q5d3, S2Q5d4)

S2Q6d1 (Oct Other workers – number) (S2Q6d2, S2Q6d3, S2Q6d4)
94

S2Q7d1 (Oct other hours) (S2Q7d2, S2Q7d3, S2Q7d4)

S2Q8d1 (Oct Other wages) (S2Q8d2, S2Q8d3, S2Q8d4)
95

S2Q9d1 (Oct Other unit) (S2Q9d2, S2QQ9d3, S2Q9d4)

If respondent reported total wages the following question is skipped and then go to any more

S2Q10d1 (Oct other Confirm wage) (S2Q10d2, S2Q10d3, S2Q10d4)
96

S2Q11d (Oct any more)

S2Q3e (Oct Type)
97

S2Q12 (Oct Confirm total workers)

S2Q13 (Oct 150 days)
98

S2Q14 (Oct 149 days)
99

JULY LABOR

S3Q1 (July labor)

S3Q2 (July intro)
100

S3Q3a (July Type)
101

JULY FIELD WORKERS

S3Q4a1 (July Field workers – Type) (each type of field workers is reported on different row. 

Interviewer should cycle back through this battery of questions for each type. For additional rows,

replace the number on the end of the variable name. e.g. S3Q4a2, S3Q4a3, S3Q4a4, S3Q4a5.
102

S3Q5a1 (July field workers – describe) (S3Q5a2, S3Q5a3, S3Q5a4, S3Q5a5)

S3Q6a1 (July field workers – number) (S3Q6a2, S3Q6a3, S3Q6a4, S3Q6a5)

S3Q7a1 (July Field workers hours) (S3Q7a2, S3Q7a3, S3Q7a4, S3Q7a5)
103

S3Q8a1 (July field workers wages) (S3Q8a2, S3Q8a3, S3Q8a4, S3Q8a5)
104

S3Q9a1 (July field workers unit) (S3Q9a2, S3Q9a3, S3Q9a4, S3Q9a5)

If respondent reported total wages the following question is skipped and then go to any more

S3Q10a (July field workers confirm wage)
105

S3Q11a (July field workers any more)

S3Q3b (July Type) 
106

JULY LIVESTOCK WORKERS

S3Q4b1 (July Livestock workers – Type) (each type of field workers is reported on different row. 

Interviewer should cycle back through this battery of questions for each type. For additional rows,

replace the number on the end of the variable name. e.g. S3Q4b2, S3Q4b3, S3Q4b4, S3Q4b5)
107

S3Q5b1 (July livestock workers – describe) (S3Q5b2, S3Q5b3, S3Q5b4, S3Q5b5)

S3Q6b1 (July livestock workers – number) (S3Q6b2, S3Q6b3, S3Q6b4, S3Q6b5)
108

S3Q7b1 (July livestock hours) (S3Q7b2, S3Q7b3, S3Q7b4, S3Q7b5)

S3Q8b1 (July livestock wages) (S3Q8b2, S3Q8b3, S3Q8b4, S3Q8b5)
109

S3Q9b1 (July livestock unit) (S3Q9b2, S3Q9b3, S3Q9b4, S3Q9b5)

S3Q10b1 (July livestock confirm wage) (S3Q10b2, S3Q10b3, S3Q10b4, S3Q10b5)
110

S3Q11b (July Livestock any more)

S3Q3c (July Type) 
111

JULY SUPERVISORS

S3Q4c1 (July Supervisors– Type) (each type of field workers is reported on different row. Interviewer 

should cycle back through this battery of questions for each type. For additional rows, replace the 

number on the end of the variable name. e.g. S3Q4c2)
112

S3Q5c1 (July Supervisors – describe) (S3Q5c2)

S3Q6c1 (July supervisor workers – number) (S3Q6c2)
113

S3Q7c1 (July Supervisor hours) (S3Q7c2)

S3Q8C1 (July supervisor wages) (S3Q8C2)
114

S3Q9c1 (July supervisor unit) (S3Q9c2)

If respondent reported total wages the following question is skipped and then go to any more

S3Q10c1 (July supervisor confirm wage) (S3Q10c2)
115

S3Q11c (July supervisor any more)

S3Q3d (July Type) 
116

JULY OTHER WORKERS

S3Q4d1 (July Other – type) (each type of field workers is reported on different row. Interviewer 

should cycle back through this battery of questions for each type. For additional rows, replace the 

number on the end of the variable name. e.g. S3Q4d2, S3Q4d3, S3Q4d4)
117

S3Q5d1 (July other – describe) (S3Q5d2, S3Q5d3, S3Q5d4)

S3Q6d1 (July other workers – number, S3Q6d2, S3Q6d3, S3Q6d4)
118

S3Q7d1 (July other hours) (S3Q7d2, S3Q7d3, S3Q7d4)

S3Q8d1 (July other wages) (S3Q8d2, S3Q8d3, S3Q8d4)
119

S3Q9d1 (July other unit) (S3Q9d2, S3Q9d3, S3Q9d4)

If respondent reported total wages the following question is skipped and then go to any more

S3Q10d1 (July other confirm wage) (S3Q10d2, S3Q10d3, S3Q10d4)
120

S3Q11d (July other any more)

S3Q3d (July Type) 
121

S3Q12 (July Confirm total workers)

S3Q13 (July 150 days)
122

S3Q14 (July 149 days)

STOP CODING
123

S3Q15 (Gross value of sales)

S3Q16 
124

S3Q17

S3Q18
125

APPENDIX B: Revised CATI Script

Ag Labor – Revised Questionnaire – October 2017
126
127

If operator, skips straight to this screen.

If not correct, respondent gets the following screen:
128

If changed, respondent gets the following screen:
129

If no to previous, respondent gets this screen:
130

If no to previous, respondent gets this screen:

If no to previous, respondent gets this screen:
131

If no to previous, respondent gets this screen:

If no to previous, respondent gets this screen:
132
133

If yes to agricultural questions, respondent gets this screen:

If partners, respondent gets the following screen:
134

If more than 1, respondent gets the following screen for each additional partner:

If individual operator or hired manager, respondent gets the following screen:
135

If person is no longer the hired manager, respondent gets the following screen:
136

BEGIN CODING

BS1Q1 (largest number)

BS1Q2 (H2A)
137

OCTOBER Labor

BS2Q1 (October Labor intro)

BS2Q2a (October labor)
138

Note: If BS2Q2a = yes ask BS2Q2b

BS2Q2b (October Number of workers)

BS2Q3 (October Type intro) 
139

Note: Questions BS2Q4, BS2Q5, BS2Q6, BS2Q7 are all asked regardless of the type of workers 

respondent report having in these questions. 

BS2Q4 (October Supervisors)

BS2Q5 (October Livestock)
140

BS2Q6 (October Crops)

BS2Q7 (October Other workers)
141

OCTOBER SUPERVISORS

Note: BS2Q8a- BS2Q8c asked if reported having Supervisors. 

BS2Q8a (October Supervisor – Type Intro)
142

Note: If reported having supervisors, Rs are asked BOTH BS2Q8b and BS2Q8c

BS2Q8b (October Supervisor – Number Farm, ranch, ag managers)

BS2Q8c (October Supervisor – Number of first line supervisors)
143

Note: BS2Q9a-BS2Q9e are only asked if respondent reported having farmers, ranchers, and other ag 

managers

BS2Q9a (October Supervisor – farmers, ranchers, ag managers describe)

BS2Q9b (October Supervisor – farmers, ranchers, ag managers hours)
144

BS2Q9c (October Supervisor – farmers, ranchers, ag managers wages)

BS2Q9d (October Supervisor – farmers, ranchers, ag managers unit)
145

BS2Q9e (October Supervisor – farmers, ranchers, ag managers Confirm wage)

Note: BS2Q10a-BS2Q10e are only asked if respondent reported having first line supervisors

BS2Q10a (October Supervisor – First line supervisor Describe)
146

BS2Q10b (October Supervisor – First line supervisor Hours)

BS2Q10c (October Supervisor – First line supervisor wages)
147

BS2Q10d (October Supervisor – First line supervisor unit)

BS2Q10e (October Supervisor – First line supervisor confirm wage)
148

OCTOBER LIVESTOCK WORKERS

Note: If reported having Livestock workers, Rs are asked BS2Q11a- BS2Q11g

BS2Q11a (October Livestock – Type Intro)
149

BS2Q11b (October Livestock – Number Livestock equipment operators)

BS2Q11c (October Livestock – Number Livestock products graders and sorters)
150

BS2Q11d (October Livestock – Number Livestock products hand packers and packagers)

BS2Q11e (October Livestock – Number of animal breeders)
151

BS2Q11f (October Livestock – Number of farmworkers)

BS2Q11g (October Livestock – Number of other specialized livestock workers not included in other 

cats)
152

Note: BS2Q12a- BS2Q12e only asked if respondent reported having livestock equipment operators

BS2Q12a (October Livestock – livestock equipment operators describe)

BS2Q12b (October Livestock – livestock equipment operators hours)
153

BS2Q12c (October Livestock – livestock equipment operators wages)

BS2Q12d (October Livestock – livestock equipment operators unit)
154

BS2Q12e (October Livestock – livestock equipment operators confirm wages)
155

Note: BS2Q13a- BS2Q13e only asked if respondent reported having livestock graders and sorters

BS2Q13a (October Livestock – livestock graders and sorters describe)

BS2Q13b (October Livestock – livestock graders and sorters hours)
156

BS2Q13c (October Livestock – livestock graders and sorters wages)

BS2Q13d (October Livestock – livestock graders and sorters unit)
157

BS2Q13e (October Livestock – livestock graders and sorters confirm wage)
158

Note: BS2Q14a- BS2Q14e only asked if respondent reported having livestock handpacker and 

packager

BS2Q14a (October Livestock – livestock handpacker and packager describe)

BS2Q14b (October Livestock – livestock handpacker and packager hours)
159

BS2Q14c (October Livestock – livestock handpacker and packager wages)

BS2Q14d (October Livestock – livestock handpacker and packager unit)
160

BS2Q14e (October Livestock – livestock handpacker and packager confirm wage)
161

Note: BS2Q15a- BS2Q15e only asked if respondent reported having livestock animal breeders

BS2Q15a (October Livestock – livestock animal breeder describe)
162

BS2Q15b (October Livestock – livestock animal breeder hours)

BS2Q15c (October Livestock – livestock animal breeder wages)
163

BS2Q15d (October Livestock – livestock animal breeder unit)

BS2Q15e (October Livestock – livestock animal breeder confirm wages)
164

Note: BS2Q16a- BS2Q16e only asked if respondent reported having livestock farmworker

BS2Q16a (October Livestock – livestock farmworker describe)
165

BS2Q16b (October Livestock – livestock farmworker hours)

BS2Q16c (October Livestock – livestock farmworker wages)

BS2Q16d (October Livestock – livestock farmworker unit)
166

BS2Q16e (October Livestock – livestock farmworker confirm wage)
167

Note: BS2Q17a- BS2Q17e only asked if respondent reported having livestock other specialized worker

BS2Q17a (October Livestock – livestock other specialized worker describe)
168

BS2Q17b (October Livestock – livestock other specialized worker hours)

BS2Q17c (October Livestock – livestock other specialized worker wages)
169

BS2Q17d (October Livestock – livestock other specialized worker unit)

BS2Q17e (October Livestock – livestock other specialized worker confirm wages)
170

OCTOBER CROPS

Note: If reported having crops workers, Rs are asked BS2Q18a- BS2Q18g
171

BS2Q18a (October Crops – Type intro **see next two screenshots)

BS2Q18b (October Crops – number equipment operators)
172

BS2Q18c (October Crops – number crop, nursey, greenhouse graders and sorters)

BS2Q18d (October Crops – number crop, nursey, greenhouse packers and packagers)
173

BS2Q18e (October Crops – number pesticide handlers and sprayers)

BS2Q18f (October Crops – number crop, nursery, greenhouse farmworkers)
174

BS2Q18g (October Crops – number all other specialized workers)

Note: BS2Q19a- BS2Q19e only asked if reported having crop nursery greenhouse equipment operators
175

BS2Q19a (October Crops – crop nursery greenhouse equipment operators describe)

BS2Q19b (October Crops – crop nursery greenhouse equipment operators hours)

BS2Q19c (October Crops – crop nursery greenhouse equipment operators wages)
176

BS2Q19d (October Crops – crop nursery greenhouse equipment operators unit)

BS2Q19e (October Crops – crop nursery greenhouse equipment operators confirm wage)
177

Note: BS2Q20a- BS2Q20e only asked if respondent reported having crops graders and sorters)
178

BS2Q20a (October Crops – graders and sorters describe)

BS2Q20b (October Crops – graders and sorters hours)

BS2Q20c (October Crops – graders and sorters wages)
179

BS2Q20d (October Crops – graders and sorters unit)

BS2Q20e (October Crops – graders and sorters confirm wages)
180

Note: BS2Q21a - BS2Q21e Only asked if reported having Crops – handpackers and packager
181

BS2Q21a (October Crops – handpackers and packager describe)

BS2Q21b (October Crops – handpackers and packager hours)

BS2Q21c (October Crops – handpackers and packager wages)
182

BS2Q21d (October Crops – handpackers and packager unit)

BS2Q21e (October Crops – handpackers and packager confirm wages)
183

Note: BS2Q22a- BS2Q22e only asked if reported having crops pesticide handler
184

BS2Q22a (October Crops – pesticide handler describe)

BS2Q22b (October Crops – pesticide handler hours)

BS2Q22c (October Crops – pesticide handler wages)
185

BS2Q22d (October Crops – pesticide handler unit)

BS2Q22e (October Crops – pesticide handler confirm wages)
186

Note: BS2Q23a- BS2Q23e only asked if reported having crop nursery greenhouse farmworker
187

BS2Q23a (October Crops – crop nursery greenhouse farmworker describe)

BS2Q23b (October Crops – crop nursery greenhouse farmworker hours)
188

BS2Q23c (October Crops – crop nursery greenhouse farmworker wages)

BS2Q23d (October Crops – crop nursery greenhouse farmworker unit)
189

BS2Q23e (October Crops – crop nursery greenhouse farmworker confirm wages)

Note: BS2Q24a- BS2Q24e only asked if reported having Crops – other specialized worker
190

BS2Q24a (October Crops – other specialized worker describe)

BS2Q24b (October Crops – other specialized worker hours)

BS2Q24c (October Crops – other specialized worker wages)
191

BS2Q24d (October Crops – other specialized worker unit)

BS2Q24e (October Crops – other specialized worker confirm wages)
192

OCTOBER CONTRACT OR CUSTOM WORKERS
193

Note: Asked of all

BS2Q25a (October Contract or Custom Workers, Retail or Value added)

BS2Q25b (October – Number of Contract or Custom Workers, Retail or Value added)

BS2Q25c (October – Exclude Contract or Custom Workers, Retail or Value added)
194

OCTOBER AGRICULTURAL INSPECTORS
195

BS2Q26a (October – Agricultural Inspectors)

Note: if BS2Q26a = yes then BS2Q26b- BS2Q26g asked 

BS2Q26b (October – Number of Agricultural Inspectors)

BS2Q26c (October –Agricultural Inspectors describe)
196

BS2Q26d (October –Agricultural Inspectors hours)

BS2Q26e (October –Agricultural Inspectors wages)
197

BS2Q26f (October –Agricultural Inspectors unit)

BS2Q26g (October –Agricultural Inspectors confirm wages)
198

OCTOBER REMAINING WORKERS
199

BS2Q27a October Remaining workers Describe

BS2Q27b October Remaining workers hours

BS2Q27c October Remaining workers wages
200

BS2Q27d October Remaining workers unit

BS2Q27e October Remaining workers confirm wages
201

BS2Q28 (October 150 days)

BS2Q29 (October 149 days)
202

JULY LABOR
203

BS3Q1 (July Labor intro)

BS3Q2 (July Labor)

Note: If BS3Q2a = yes ask BS3Q2b
204

BS3Q2b (July Labor- Number)

BS3Q3 (July Labor- Type Intro)
205

Note: Questions BS3Q3, BS3Q4, BS3Q5, BS3Q6 are all asked regardless of the type of workers 

respondent report having in these questions. 

BS3Q3 (July Supervisors)

BS3Q4 (July Livestock workers)
206

BS3Q5 (July Crop, nursery, greenhouse workers)

BS3Q6 (July Other type of workers)
207

JULY SUPERVISORS

Note: Supervisor subquestions are only asked if respondent reported having supervisors in BS3Q3

BS3Q6 (July Supervisor Intro)
208

Note: If reported having supervisors, Rs are asked BOTH BS3Q7a and BS3Q7b

BS3Q7a (July Supervisors – Number of Farmers, ranchers, and other ag manager)

BS3Q7b (July Supervisors – Number of First line supervisors)
209

Note: BS3Q8a- BS3Q8e are only asked if reported having farmers, ranchers and other ag managers

BS3Q8a (July Supervisors – farmers, ranchers and other ag managers describe)

BS3Q8b (July Supervisors – farmers, ranchers and other ag managers hours)
210

BS3Q8c (July Supervisors – farmers, ranchers and other ag managers wages)

BS3Q8d (July Supervisors – farmers, ranchers and other ag managers unit)
211

BS3Q8e (July Supervisors – farmers, ranchers and other ag managers confirm wages)
212

Notes: BS3Q9a- BS3Q9e only asked if reported having first line supervisors

BS3Q9a (July Supervisors – First line supervisors describe)

BS3Q9b (July Supervisors – First line supervisors hours)
213

BS3Q9c (July Supervisors – First line supervisors wages)

BS3Q9d (July Supervisors – First line supervisors unit)
214

BS3Q9e (July Supervisors – First line supervisors confirm wages)
215

JULY LIVESTOCK WORKERS

Note: Livestock subquestions are only asked if respondent report having Livestock workers in BS3Q4

BS3Q10a (July Livestock Intro)
216

If R reported having livestock workers respondent are asked BS3Q10b- BS3Q10g

BS3Q10b (July Livestock Number of equipment operators)

BS3Q10c (July Livestock Number of graders and sorters)
217

BS3Q10d (July Livestock Number of handpackers and packagers)

BS3Q10e (July Livestock Number of animal breeders)
218

BS3Q10f (July Livestock Number of livestock farmworkers)

BS3Q10g (July Livestock Number of specialized workers)
219

BS3Q11a- BS3Q11e asked if reported having livestock equipment operators

BS3Q11a (July Livestock equipment operators describe)

BS3Q11b (July Livestock equipment operators hours)
220

BS3Q11c (July Livestock equipment operators wages)

BS3Q11d (July Livestock equipment operators unit)
221

BS3Q11e (July Livestock equipment operators confirm wage)
222

BS3Q12a- BS3Q12e asked if reported having livestock graders and sorters

BS3Q12a (July Livestock graders and sorters describe)

BS3Q12b (July Livestock graders and sorters hours)
223

BS3Q12c (July Livestock graders and sorters wages)

BS3Q12d (July Livestock graders and sorters unit)
224

BS3Q12e (July Livestock graders and sorters confirm wage)
225

BS3Q13a- BS3Q13e only asked if reported having handpacker and packager

BS3Q13a (July Livestock handpacker and packager describe)

BS3Q13b (July Livestock handpacker and packager hours)
226

BS3Q13c (July Livestock handpacker and packager wages)

BS3Q13d (July Livestock handpacker and packager unit)
227

BS3Q13e (July Livestock handpacker and packager Confirm wages)
228

BS3Q14a- BS3Q14e only asked if reported having animal breeder

BS3Q14a(July Livestock animal breeder describe)

BS3Q14b (July Livestock animal breeder hours)
229

BS3Q14c (July Livestock animal breeder wages)

BS3Q14d (July Livestock animal breeder unit)
230

BS3Q14e (July Livestock animal breeder confirm wages)
231

BS3Q15a- BS3Q15e is only asked if R reported having a livestock farmworker

BS3Q15a (July Livestock farmworker describe)

BS3Q15b (July Livestock farmworker hours)
232

BS3Q15c (July Livestock farmworker wages)

BS3Q15d (July Livestock farmworker unit)
233

BS3Q15e (July Livestock farmworker confirm wages)
234

BS3Q16a- BS3Q16e are asked if reported having specialized livestock workers

BS3Q16a (July Livestock specialized workers describe)

BS3Q16b (July Livestock specialized workers hours)
235

BS3Q16c (July Livestock specialized workers wages)

BS3Q16d (July Livestock specialized workers unit)
236

BS3Q16e (July Livestock specialized workers Confirm wage)
237

JULY CROPS WORKERS

Note: Crops subquestions are only asked if respondent report having Crops workers in BS3Q5

BS3Q17a (July Crops Intro)
238

If R reported having crops workers respondent are asked BS3Q17b- BS3Q17g

BS3Q17b (July Crops Number of crop, nursery, and greenhouse equipment operators)

BS3Q17c (July Crops graders and sorters)
239

BS3Q17d (July Crops hand packers and packagers)

BS3Q17e (July Crops pesticide handlers and sprayers)
240

BS3Q17f (July Crops farmworkers)

BS3Q17g (July Crops specialized workers)
241

If reported having crops equipment operators, Rs asked BS3Q18a- BS3Q18e

BS3Q18a (July Crops equipment operators describe)

BS3Q18b (July Crops equipment operators hours)
242

BS3Q18c (July Crops equipment operators wages)

BS3Q18d (July Crops equipment operators unit)
243

BS3Q18e (July Crops equipment operators confirm wages)
244

If R reported having crops graders and sorter respondent are asked BS3Q19a- BS3Q19e

BS3Q19a (July Crops graders and sorters describe)

BS3Q19b (July Crops graders and sorters hours)
245

BS3Q19c (July Crops graders and sorters wages)

BS3Q19d (July Crops graders and sorters unit)
246

BS3Q19e (July Crops graders and sorters confirm wages)
247

If R reported having crops handpackers and packager, respondent are asked BS3Q20a-20e

BS3Q20a (July Crops handpackers and packager describe)

BS3Q20b (July Crops handpackers and packager hours)
248

BS3Q20c (July Crops handpackers and packager wages)

BS3Q20d (July Crops handpackers and packager unit)
249

BS3Q20e (July Crops handpackers and packager confirm wage)
250

If R reports pesticide handler, respondent are asked BS3Q21a- BS3Q21e

BS3Q21a (July Crops pesticide handler describe)

BS3Q21b (July Crops pesticide handler hours)
251

BS3Q21c (July Crops pesticide handler wages)

BS3Q21d (July Crops pesticide handler unit)
252

BS3Q21e (July Crops pesticide handler confirm wage)
253

If R reports crops farmworker, respondent are asked BS3Q22a- BS3Q22e

BS3Q22a (July Crops Farmworker describe)

BS3Q22b (July Crops Farmworker hours)
254

BS3Q22c (July Crops Farmworker wages)

BS3Q22d (July Crops Farmworker unit)
255

BS3Q22e (July Crops Farmworker confirm wage)
256

If R reports specialized worker, respondent are asked BS3Q23a -BS3Q23e

BS3Q23a (July Crops specialized worker describe)

BS3Q23b(July Crops specialized worker hours)
257

BS3Q23c (July Crops specialized worker wages)

BS3Q23d (July Crops specialized worker unit)
258

BS3Q23e (July Crops specialized worker confirm wages)
259

JULY CONTRACT AND CUSTOM WORKERS

Note: Asked of all

BS3Q24a (July Contract or Custom Workers, Retail or Value added)

BS3Q24b (July Number of Contract or Custom Workers, Retail or Value added)
260

BS3Q24c (July Exclude Contract or Custom Workers, Retail or Value added)
261

JULY AGRICULTURAL INSPECTORS

BS3Q25a (July– Agricultural Inspectors)

Note: if BS3Q25a = yes then BS3Q25b- BS3Q25g asked 

BS3Q25b (July – Number of Agricultural Inspectors)
262

BS3Q25c (July –Agricultural Inspectors - Describe)

BS3Q25d (July –Agricultural Inspectors - hours)
263

BS3Q25e (July –Agricultural Inspectors - wages)

BS3Q25f (July –Agricultural Inspectors - unit)
264

BS3Q25g (July –Agricultural Inspectors – confirm wages)
265

JULY REMAINING WORKERS

BS3Q26a (July Remaining workers describe)

BS3Q26b (July Remaining workers Hours)
266

BS3Q26c (July Remaining workers wages)

BS3Q26d (July Remaining workers unit)
267

BS3Q26e (July Remaining workers confirm wages)

BS3Q27 (July 150 days)
268

BS3Q28 (July 149 days)

STOP CODING
269
270
271