Behavior Coding of the 

April 2018 Agricultural 

Labor Survey

Heather Ridolfo, David Biagas Jr., 

Emilola J. Abayomi, and Joseph 

Rodhouse

United States

Department of

Agricultural

National

Agricultural

Statistics

Service

Research and

Development Division

Washington, DC 20250

RDD Research Report

Number RDD-21-01

April 2021

The findings and conclusions in this report are those of the author and should not be construed to represent any

official USDA or U.S. Government determination or policy.
ii

EXECUTIVE SUMMARY

In April 2018, two new questions were proposed to be added to the Agricultural Labor Survey. 

In addition to asking respondents to report gross wages paid, the survey questionnaire was 

revised to accommodate two new questions on base wages and incentive/overtime wages, and a 

field test was conducted to evaluate the impact of these questions on data quality and usability.1

In addition to the new wage questions, several changes were made to the revised computerassisted telephone interviewing (CATI) instrument based on previous research (Sloan 2017; 

Sloan 2018; Ridolfo et al. 2020).

In conjunction with the field test, behavior coding was conducted to compare the administration 

of the original Agricultural Labor CATI instrument and the revised instrument. In particular, the 

behavior coding focused on determining whether there were differences between the two 

versions in terms of the interviewers’ ability to administer the questions in a standardized 

manner, respondents’ requests for clarification, and respondents’ ability to provide an adequate 

response. Forty-one interviews were selected for behavior coding. Twenty-two of the interviews 

were conducted using the original CATI instrument by the National Operation Center (NOC).

2

Nineteen of the interviews were conducted using the revised CATI script by the Oklahoma data 

collection center. Although respondent behavior was coded, this report focuses solely on 

interviewer behavior. 

Results from the behavior coding showed that neither of the instruments were administered 

appropriately at an acceptable rate. Interviewers failed to verify responses, made major changes 

to survey questions, and shortcutted survey questions (did not ask them at all) at high rates. 

Interviewers’ lack of adherence to the CATI instrument was particularly pronounced in questions 

that asked about the second reference period. Furthermore, the new questions on wages were 

often not asked as worded or not asked at all. This made it difficult to evaluate the new survey 

questions. 

In addition, when interviewers made major changes to survey questions and responded to 

respondents’ requests for clarification, they did not demonstrate an adequate understanding of 

the constructs being measured. For example, interviewers were observed providing inaccurate 

definitions and inclusion criteria, and selecting inappropriate worker types. This led to workers 

being inappropriately screened into and out of the survey, and classified as the wrong worker 

types, which could have larger implications for published wage rates. 

Based on the results of this and previous research (Ridolfo et al. 2020), further work is needed to 

improve the usability of the Agricultural Labor Survey CATI instruments. Additionally, 

interviewers would benefit from additional training on the content of the Agricultural Labor 

Survey and standardized interviewing.

1

For more information on the field test, see Reist et al. 2018). 

2 Two interviews conducted using the original questionnaire did not have labor during either reference period. 

These interviews were retained and two more were added that did have labor during the reference periods. 
iii

RECOMMENDATIONS

1) Conduct enumerator training that focuses on the importance of reading survey questions 

as worded and discourages the use of conversational interviewing to reduce interviewer 

variation. 

2) Conduct enumerator training on the content of the Agricultural Labor Survey.

3) Continue to conduct behavior coding on future cycles of the Agricultural Labor Survey.
iv

TABLE OF CONTENTS

1. Introduction..........................................................................................................................1

2. Methods...............................................................................................................................3

2.1 Sample............................................................................................................................3

2.2 Behavior Coding .............................................................................................................3

3. Results.................................................................................................................................4

3.1 Screener..........................................................................................................................4

3.2 Worker Type Introduction ...............................................................................................6

3.3 Worker Type ..................................................................................................................7

3.4 Worker Subtypes ............................................................................................................9

3.5 Total Hours .................................................................................................................. 10

3.6 Gross Wages ................................................................................................................ 11

3.7 Base Wages .................................................................................................................. 13

3.8 Incentive/Overtime ....................................................................................................... 14

3.9 150 & 149 Day Questions ............................................................................................. 15

3.10 Peak Number of Workers ............................................................................................ 16

3.11 H-2A .......................................................................................................................... 17

4. Conclusions and Recommendations .................................................................................... 19

5. References.......................................................................................................................... 20

APPENDIX A: Original CATI Script...................................................................................... 22

APPENDIX B: Revised CATI Script ...................................................................................... 55
1

Behavior Coding of the April 2018 Agricultural Labor Survey

Heather Ridolfo, David Biagas Jr., Emilola J. Abayomi, and Joseph Rodhouse

Abstract

In 2018, new wage questions were proposed to be added to the Agricultural Labor Survey. In 

addition, several changes were made to the revised computer-assisted telephone interviewing

(CATI) script based on previous research. To evaluate the addition of the new wage questions 

and revisions to the CATI script, a field test was conducted using the original Agricultural Labor 

Survey and a revised version. Behavior coding was conducted on a subset of the interviews 

conducted in NASS telephone data collection centers. The purpose of this behavior coding was

to compare interviewers’ administration of the original Agricultural Labor CATI instrument and 

the revised instrument. Results showed that neither of the instruments was administered 

appropriately at an acceptable rate. Additionally, the new wages questions were often 

administered with major changes or not at all. These findings indicate a need for additional 

enumerator training and a review of the Agricultural Labor CATI instruments. 

Key Words: Behavior Coding, Interviewer-Respondent Interaction, Data Quality

1. INTRODUCTION

The Agricultural Labor Survey collects information on agricultural labor performed on farms and 

ranches for four reference weeks in each calendar year. Respondents are asked to report the 

number of workers on payroll, type of work performed, total number of hours worked, and total 

gross wages paid to these workers. In April 2018, two new questions were proposed to be added 

to the Agricultural Labor Survey. In addition to asking respondents to report gross wages paid, 

the survey questionnaire was revised to accommodate two new questions on base wages and 

incentive/overtime wages, and a field test was conducted to evaluate the impact of these 

questions on data quality and usability.3

In addition to the new wage questions, several changes were made to the revised computerassisted telephone interviewing (CATI) instrument based on previous research (Sloan 2017; 

Sloan 2018; Ridolfo et al. 2020). The most substantial changes were to the questions asking 

about the number and type of workers on the payroll during the reference weeks. In the original 

instrument, respondents were first screened on labor during the reference week and then were

asked about the type of workers on payroll that week. Respondents could choose between four 

options: field workers, livestock workers, supervisors, and other type of workers. Respondents 

could report more than one type. However, the type they indicated first (or the interviewer 

selected to ask about first) was the first type respondents were asked to report detailed 

information on (e.g., number, hours worked, wages paid). After reporting detailed information on 

this group of workers, interviewers were then instructed to ask respondents if they had any

additional workers. If they had additional workers, interviewers were instructed to cycle back 

through the worker types and questions on hours worked and wages paid. 

3

For more information on the field test, see Reist et al. 2018). 
2

Based on previous research, there was concern that respondents were underreporting livestock 

workers, supervisors, and other types of workers given that respondents were often asked about 

their field workers first (Sloan 2017). It was hypothesized that after completing a lengthy battery 

of questions on their field workers, respondents would lose motivation to report other types of 

workers and instead satisfice when asked if they had any additional workers, and that 

respondents may be misreporting workers under field workers. Previous research has also found 

respondents report fewer workers and fewer types of workers when the labor survey is 

interviewer administered versus self-administered (Biagas et al. 2019). 

To obtain more complete reporting for all worker types, in the revised instrument, respondents 

were screened for labor in the reference week and then asked to report the total number of 

workers in the reference week, followed by the type of work performed, hours worked, and 

wages paid. Asking respondents to report the total number of workers for the reference period 

before providing detailed information on those workers, allowed interviewers to see how many 

workers were still unaccounted for when collecting the detailed information. 

Two questions in the original instrument were also removed from the revised instrument. In the 

original instrument, respondents were asked to report the type of work their workers were hired 

to do and offered four response options. Respondents were also asked to describe the type of 

work their workers were hired to do. This open-ended question was removed from the revised 

instrument. 

The second question removed from the revised instrument was the question that asked whether 

workers worked for 149 days or less. Previous behavior coding found that this question was 

rarely administered (Ridolfo et al. 2020). This question was removed from the revised instrument 

because it could be derived by subtracting the number of workers who worked 150 days or more 

from the total number of workers reported for the reference period. 

The placement of some questions was also changed on the revised instrument. In the original 

instrument, the questions on whether workers worked 150 days or more and whether they 

worked 149 days or less were asked for each reference week after respondents reported all 

detailed information for workers on the payroll during the reference period. On the revised 

instrument, the 150-day question was placed after the question that asked about the number of 

workers on the payroll during the reference week, and before detailed information was collected 

on those workers. The questions on peak number of workers and H-2A workers (non-immigrant 

foreign workers brought to the United States under the H-2A program) were also moved on the 

revised instrument. In the original instrument, these questions appear at the very end of the 

survey after respondents answered the farm-type question. In the revised instrument, these 

questions appeared before the value of sales question. 

In addition, other changes were made to the CATI instrument in both versions to make it easier 

for interviewers to read the text on the screen. Significant changes to the questionnaire wording 

or formatting are described in the results where relevant. A copy of the original CATI instrument 

can be found in Appendix A and copy of the revised CATI instrument can be found in Appendix 

B. 
3

In conjunction with the field test, behavior coding was conducted to compare the administration 

of the original Agricultural Labor CATI instrument and the revised instrument. In particular, the 

behavior coding focused on determining whether the two versions differed in terms of the 

interviewers’ ability to administer the questions in a standardized manner, respondents’ requests 

for clarification, and respondents’ ability to provide an adequate response. However, this report 

focuses solely on interviewers’ administration of the questions. 

2. METHODS

2.1 Sample

The original CATI instrument was administered in NASS’s National Operation Center (NOC). 

The revised script was administered in the Oklahoma data collection center. Audio of all calls 

was recorded using the Verint system. Screen capture was not available at the time of data 

collection. Forty-one interviews were selected for behavior coding. Twenty-two of the interviews 

were conducted using the original CATI instrument by the NOC.4 Nineteen of the interviews 

were conducted using the revised CATI script by the Oklahoma data collection center. 

2.2 Behavior Coding

In behavior coding, each turn in the interview can be coded. A turn begins when the first person 

begins speaking and ends when the second person begins speaking. A pair of turns is referred to 

as an exchange (Ongena and Dijkstra 2006). During the administration of a single question a 

number of exchanges can occur before a final answer to the question is given. Ideally, in each 

question administration, there would be one exchange between the interviewer and the 

respondent. Previous research has found there to be diminishing returns to coding all exchanges 

for a single question (Oksenberg, Cannell, and Kalton 1991). Therefore, only the first exchange 

and final response are provided. For the purposes of this study, both the interviewer and 

respondent behaviors were coded. In this report, we focus on the interviewer behavior (i.e. the 

administration of the survey questions) only. 

The codes used to assess the question asking behavior of interviewers are summarized in Table 1 

below.

Table 1. Behavior Codes for Interviewer 

Behavior

Code Description

ES Exact wording/slight change

VER Verified response

NOVER Failure to verify a response

MC Major change

SC Shortcutting

4 Two interviews conducted using the original questionnaire did not have labor during either reference period. 

These interviews were retained and two more were added that did have labor during the reference periods. 
4

If the interviewer read the question as worded with only slight or minimal changes, the exact 

wording/slight change (ES) code was applied. Verification (VER) occurred when interviewers 

verified a response that respondents preemptively provided in previous questions. Failure to 

verify a response (NOVER) occurred when respondents provided enough information in a 

previous question to answer the current survey question, but the interviewer failed to verify the 

information and proceeded to enter a response. Questions were coded as a major change (MC) if 

interviewers read a question in a manner that substantially altered the question meaning. Finally, 

interviewer behavior was coded as shortcutting (SC) if the interviewer entered a response 

without asking the survey question, and the respondent had not provided sufficient information 

prior to the current question to code the behavior as NOVER. Major changes, failure to verify, 

and shortcutting are considered to be problematic behavior. When these codes are applied to a 

question at least 15 percent of the time, it is an indication that there is a problem with the survey 

question. (Fowler, 2011).

Four researchers trained in behavior coding coded the interviews. Before coding began, Cohen’s 

kappa was calculated to ensure consistency across coding. Five interviews were selected for the 

kappa calculation. Two of these interviews were conducted using the original instrument, while 

three were conducted using the revised instrument. Each researcher coded the five interviews 

independently. Cohen’s kappa (Cohen 1960) was calculated for all possible coder pairs. The 

overall average of these kappa combinations was 0.77, indicating that there was substantial 

agreement among the three coders (Landis and Koch 1977). 

In the following results, the distribution of interviewer behavior codes is presented. In instances 

where major changes were made to the question at a high rate, the types of changes made to the 

question are summarized. 

3. RESULTS

3.1 Screener

Prior to enumerating the workers on the operation for each reference period, interviewers 

administered a screener question to determine whether respondents had any workers on the 

payroll during the reference periods. The screener was revised in 2018 and the same layout was 

used in both versions of the survey. Prior to 2018, the types of workers respondents should report 

were displayed in an optional paragraph (Figure 1). In both versions for the April 2018 data 

collection, the types of workers that should be reported were bulleted and were required text 

(Figure 2). 
5

Figure 1. October 2017 Screener Question

Figure 2. April 2018 Screener Question

Interviewers very rarely administered the screener questions as worded when using the original 

script (April: 18%, January: 9%) and never administered these questions as worded when using 

the revised script (Table 2). Instead, interviewers made major changes to the screener questions 

at very high rates across quarters and versions. 

Table 2. Screener Questions Interviewer Behavior

Code April January

Original Revised Original Revised

ES 18% 0% 9% 0%

VER 0% 0% 0% 0%

NOVER 0% 0% 0% 0%

MC 82% 95% 91% 89%

SC 0% 5% 0% 11%

Total 22 19 22 19

Interviewers often made multiple major changes to this question when administering it. The most 

common change made was not reading the include/exclude statements that followed the 
6

question. As mentioned above, the interviewers were required to read the include/exclude 

statements on both versions of the instrument, and the statements were reformatted to make them 

easier for interviewers to read. Other common changes to these questions were not reading “to do 

agricultural work” and not reading the reference period or changing the wording of the reference 

period (e.g., “last week”) when asking the question. These three elements (inclusion/exclusion 

criteria, agricultural work, and the reference period) are all essential to ensuring that respondents

report the correct types of workers. Additionally, interviewers sometimes asked these questions 

in a leading way, particularly for the second quarter questions (e.g., “How about in January of 

this year? The second week, which is January 7th through January 13th. The same number of 

people you think?”). In some instances, interviewers left off all important elements of this 

question and instead verified that the respondent had the same workers during the second 

reference period, as demonstrated in the following exchange. 

I: Did you have any workers on- in January too?

R: Yes.

I: The week of- it would have been the week of- let's see, well I don't know. Do you have time? I 

just need to put the wages in real quick. 

R: I mean you can go ahead and put something in there. It doesn't matter. We had people 

working. Whether they was working that week or it was snowing that week and they was taking 

off, I couldn't tell you. 

I: Okay. You couldn'tR: What week that was. I couldn't tell you what week it was. 

I: Yeah. 

R: We were supposed to have workers. The same two were supposed to be working that week. 

I: Okay.

R: Unless, if it was the first week, unless it was the first week of January, we only had one of 

them. The other one didn't start until like the second week of January. 

I: Oh, I see. Okay and this one, wait- I'm not to January yet. (typing). Oh it's January 7th through 

the 13th. 

R: Okay.

I: That would be the second week. 

R: I only had one worker January 7th through the 13th. 

I: Oh alright. Let’s see.

Not only did this interviewer leave off several important elements of the survey question and 

asked the question in a leading way, it took eight exchanges to reach to a final response. 

3.2 Worker Type Introduction

The worker type introduction was worded the same in both versions, with the exception of the 

addition of base and incentive wages in the revised version (Figure 3). The placement of this 

screen varied across version. On the original labor CATI instrument, this introductory text was 

placed directly after the screener question. On the revised instrument, this introductory text was 

placed after the 150-day question. 
7

 Figure 3. Worker Type Introduction Revised Version

Interviewers also did not administer the worker type introduction as worded very often for the 

first reference period and never read this text as worded for the second reference period,

regardless of version (Table 3). When administering the worker type introduction for the first 

reference period (April), interviewers made major changes to the text (original: 45%, revised: 

37%) and shortcutted the text (did not read at all) at high rates (original: 25%, revised: 58%)

across instruments. For the second reference period (January), interviewers shortcutted the 

introductory text on both instruments at very high rates (original: 92%, revised: 100%). 

Table 3. Worker Type Introduction Interviewer Behavior

Code April January

Original Revised Original Revised

ES 30% 5% 0% 0%

VER 0% 0% 0% 0%

NOVER 0% 0% 0% 0%

MC 45% 37% 8% 0%

SC 25% 58% 92% 100%

Total 20 19 13 17

When interviewers made major changes to this question, they often only read some of the 

information on the screen. For example, several interviewers only read the first sentence on the 

screen. Others read the first three sentences only. In some instances, the interviewers paraphrased 

the information on the screen, leaving off the instructions and/or the worker types (e.g., “Okay 

we're going to categorize them”). Finally, in some interviews, the interviewers turned the 

introductory text into a question (e.g., “How would you categorize those workers? There are 

different worker groups we got. We've got fieldworkers - which is like a jack of all trades, 

machinery operators, vegetable pickers, et cetera. I don't think you got livestock workers there.”). 

3.3 Worker Type
8

The worker type questions were the same in both versions of the questionnaire (Figure 4), and 

interviewers administered this question in the same manner across versions (Table 4). When this 

question was administered for the first reference period, interviewers often asked the question as 

worded (original: 24%, revised: 30%), verified the response (original: 16%, revised 10%) or 

failed to verify the response (original: 20%, revised 23%). The high rates of verification and 

failure to verify are due to respondents often providing their response after the interviewers read 

the introductory text. More concerning is the high rate of major changes made to this question 

for the first reference period (original: 33%, revised: 37%). 

Figure 4. Worker Type Question Example

Interviewers administered this question as worded less often for the second reference period

when using both versions. No interviewers read this question as worded when using the original

version, and only 4 percent read this question as worded when using the revised version. 

Interviewers continued to verify the response (original: 18%, revised: 11%). Several interviewers

failed to verify the response for the second reference period (original: 44%, revised 33%). 

Interviewers made fewer major changes to this question for the second reference period

(original: 6%, revised 19%); however, they often shortcutted it (original: 32%, revised: 33%). 

Table 4. Worker Type Interviewer Behavior

Code April January

Original Revised Original Revised

ES 24% 30% 0% 4%

VER 16% 10% 18% 11%

NOVER 20% 23% 44% 33%

MC 33% 37% 6% 19%

SC 6% 0% 32% 33%

N 49 30 34 27

In almost all instances where interviewers made major changes to this question, they did not read 

the part of the question that asked “type of work hired to do.” In some interviews, the interviewer 

changed the question to ask what type of work the workers were doing or what type of workers 

they were. This change is problematic because respondents may answer based on what type of 

work was needed to be done during the reference week rather than what the worker was hired to 

do:
9

I: And what kind of work do they do for you sir?

R: They’re setting tobacco right now.

Since workers often perform more than one job but are typically compensated for the job that 

requires the highest skill level, this could possibly skew the data. Another similar change 

interviewers made to this question was asking what types of workers they were: 

I: What kind of workers were they? Field workers, livestock workers, supervisor/manager, or 

other? 

R: Kind of all three. Or, all four, really. 

In some interviews, interviewers asked this question in a leading way (e.g., “Were all 8 of them 

hired to do- were they hired to do the same thing pretty much? Field work, livestock work, 

supervision management, or other work.”). When asking this question for the second reference 

period, the interviewers sometimes asked whether the respondent would list them under the same 

worker type as they did for the previous reference period (e.g., “Would they still be under that 

same classification?”). This is problematic because it is making an assumption that the 

respondent had the same workers in both reference periods. Additionally, if the question was not 

asked correctly for the first reference period, the error is carried forward to the second reference 

period. 

3.4 Worker Subtypes

The worker subtype questions (Figure 5) were read as worded or verified a small percentage of 

the time across reference periods and versions of the survey (Table 5). This problem was 

particularly pronounced for the second reference period. When asking about the second reference 

period, these questions were never asked as worded using either version. Interviewers entered a 

response without verifying the information at high rates in all administrations of these questions,

but more so when administering the questions for the second reference period (original: 47%, 

revised 31%). Major changes to the subtype questions were often made in the administration of 

the first reference period questions using both versions (original: 41%, revised 33%), but major 

changes occurred less often in the administration of the second reference period subtype 

questions (original: 0%, revised: 7%). Finally, interviewers entered a response to these questions 

without asking the survey questions at a high rate, particularly for the second reference period

using the revised questionnaire. For the first reference period, interviewers shortcutted these 

questions 22 percent of the time using the original instrument and 37 percent of the time using 

the revised instrument. For the second reference period, this increased to 38 percent of the time 

using the original version and 62 percent using the revised version. 
10

Figure 5. Worker Subtype Questions

Table 5. Worker Subtypes Interviewer Behavior

Code April January

Original Revised Original Revised

ES 6% 3% 0% 0%

VER 8% 7% 15% 0%

NOVER 22% 20% 47% 31%

MC 41% 33% 0% 7%

SC 22% 37% 38% 62%

Total 49 30 34 29

When interviewers made major changes to the worker subtype questions, they most often did not 

read the question as worded, did not read the instruction and only offered some of the response 

options (e.g., “And in the livestock were they like did they operate equipment or were they more 

farmworkers where they worked farm animals and ranch animals?”). In addition to these 

changes, some interviewers asked this question in a leading way (e.g., “And that would be, I 

guess, first-line supervisor, of farmworkers, basically?”). Similarly, a few interviewers tried to 

select a category for the respondent instead of reading the question and the categories, and 

allowing the respondent to select a response:

I: I'm trying to guess at some of these categories.

R: okay.

I: They're not breeders, are they?

R: No, ma'am. Not breeders. 

I: Okay. I'm going to put them under any other worker not listed.

3.5 Total Hours

Interviewers were more likely to administer the hours questions (Figure 6) when asking about 

workers during the first reference period compared to the second reference period, particularly 

when using the original questionnaire (Table 6). During the administration of the first reference 

period questions, interviewers asked the hours questions as worded 40 percent of the time when 

using the original questionnaire and 43 percent of the time when using the revised questionnaire. 

However, interviewers also made major changes to the hours questions at a high rate when 
11

asking about the first reference period (original: 48%, revised: 33%). When asking about the 

second reference period, interviewers verified the response (original: 24%, revised 25%), failed 

to verify the response (original: 44%, revised 18%), and shortcutted the question at high rates 

(original: 21%, revised: 32%). 

Figure 6. Total Hours Question

Table 6. Hours Worked Interviewer Behavior

Code April January

Original Revised Original Revised

ES 40% 43% 0% 4%

VER 8% 7% 24% 25%

NOVER 0% 7% 44% 18%

MC 48% 33% 12% 21%

SC 4% 10% 21% 32%

Total 50 30 34 28

When interviewers made major changes to this question, the majority did not read “total” or the 

number of workers in the category (e.g., “How many hours did they each work would you say? 

April the 8th through the 14th? That would be 6 days.”). Respondents often provided responses 

to this type of question that could be interpreted as total hours per worker, such as 40 hours. 

Sometimes the interviewer then followed up with a question like “per person?” other times they 

entered the response without following up. 

Interviewers also asked this question in a leading way: 

“And how many hours did he work that week? Was it 20 hours?”

“Were they working 40 hours a week?”

A few interviewers asked for averages or estimates instead of totals (e.g., “And then what were 

the total hours that those workers worked that week if you had to take a good average?”) A few 

interviewers also asked the respondent whether they knew how many hours their workers 

worked (e.g., “Do you know the total hours that they worked?”).

3.6 Gross Wages
12

The administration of the gross wages question (Figure 7) was similar to the administration of 

the hours questions. Interviewers asked the gross wages questions most frequently during the 

administration of the first reference period questions independent of the version of the 

questionnaire used. However, they made major changes to the question in most administrations 

(original: 70%, revised: 63%). Interviewers verified the answer, failed to verify the answer, and

shortcutted the question more often when administering the second reference period questions

(Table 7). 

Figure 7. Gross Wages Question

Table 7. Gross Wages Interviewer Behavior

Code April January

Original Revised Original Revised

ES 22% 23% 0% 11%

VER 6% 0% 26% 21%

NOVER 2% 7% 44% 21%

MC 70% 63% 3% 11%

SC 0% 7% 26% 36%

Total 50 30 34 28

When interviewers made major changes to this question, they almost never read the sentence that 

preceded the question. Additionally, in the majority of major changes, they did not ask for total 

gross wages and often did not read the number of workers. Instead, interviewers asked for 

workers’ wages (e.g., “And then their wages?”), workers’ salary (e.g., “Okay... and it says what 

were the average salary per worker?”), rate of pay (e.g., “And what were their rate of pay?”), 

average weekly salary (“What is her average weekly salary?”), income (e.g., “What was your 

income?”), how much the workers made (e.g., “Ok. let's see, and how much did they make?”), or 

were paid (“And what was their pay?”). 
13

Often, when making these types of changes, interviewers assumed that workers were all paid the 

same amount. Sometimes respondents corrected them. In these cases, numerous exchanges were 

needed to come to a final response. 

In some interviews, the interviewers did not read “during the week” when asking the question. 

In some cases, they changed the question to ask about how much the workers were paid per week 

(e.g., “How much were gross wages was this individual paid per week?”).

In a few interviews, the interviewer changed the wording of the question and asked the question 

in a leading way (e.g., “And what was their wages, like $10 an hour, five to eight dollars an hour, 

12?”). 

Finally, in some interviews, the interviewer asked the respondent whether they knew or were 

able to provide workers’ wages. These types of questions sometimes resulted in nonresponse 

(e.g., I: Can you tell me approximately what your hourly wage is or total amount paid for that 

week? R: Off the top of my head – no) or don’t know responses (e.g., i: Do you know the total 

gross wages for the 8 workers that week? R: I don't know.). 

3.7 Base Wages

The base wage question was one of the new questions added to the revised instrument (Figure 8). 

The interviewers failed to administer the base wage question the majority of the time (Table 8). 

During the administration of this question for the first reference period, interviewers only asked 

this question as worded 7 percent of the time, and the response was verified 7 percent of the 

time. Thirty-three percent of the time, the question was asked with major changes and in 53 

percent of the administrations the question was shortcutted. This question was administered even 

less for the second reference period. During the administration of the second reference period 

questions, the question was never asked as worded and never verified. In 21 percent of the 

administrations, interviewers failed to verify the response. In 75 percent of the interviews, the 

interviewers shortcutted the question. 

Figure 8. Base Wages Question
14

Table 8. Base Wages Interviewer Behavior

Code April January

ES 7% 0%

VER 7% 0%

NOVER 0% 21%

MC 33% 4%

SC 53% 75%

Total 30 28

When interviewers made major changes to this question, they never read the definition of base 

wages that follows the survey question. They also often reworded this question in a leading way 

(e.g., “And those were all base wages?”). Sometimes interviewers asked if the gross wages 

included overtime or bonus pay as well (e.g., “Do they also get any overtime, incentive or that 

was the base pay?”). 

3.8 Incentive/Overtime

The incentive/overtime question was also a new question added to the revised questionnaire 

(Figure 9). As with the base wage question, the incentive/overtime question was not asked as 

worded by the interviewers very often for either reference period (Table 9). When administering 

this question for the first reference period, interviewers only administered this question as 

worded 3 percent of the time, and they never administered it as worded for the second reference 

period. For the first reference period, interviewers administered this question with major changes 

at a high rate (40%). Additionally, 43 percent of time they shortcutted the question. For the 

second reference period, interviewers failed to verify the answer 21 percent of the time and 

shortcutted the question 75 percent of the time. 

Figure 9. Incentive/Overtime Question
15

Table 9. Incentive/Overtime Wages Interviewer Behavior

Code April January

ES 3% 0%

VER 3% 0%

NOVER 10% 21%

MC 40% 4%

SC 43% 75%

Total 30 28

When interviewers made major changes to this question, they never asked how much of the gross 

wages paid that week were incentive or overtime pay. Instead, they often asked whether the 

workers ever received incentives and overtime pay (e.g., “Did they ever get incentive or 

overtime or anything like that?”). Sometimes they only asked about overtime (e.g., “Did they 

have any overtime wages added to that?”). At other times, they only asked about incentives (e.g., 

“And does he get paid incentive for anything like that?”). Moreover, some interviewers made 

these changes and asked the question in a leading way (e.g., “And you don't pay overtime?”).

3.9 150 & 149 Day Questions

As mentioned above, the question that asked whether workers worked 149 days or less was 

removed from the revised instrument. In the original instrument, respondents were asked how 

many of the workers worked 150 days or more (Figure 10). If all workers were not accounted for 

in that question, they were then asked how many of the workers worked 149 days or less (Figure 

11). These questions were administered after respondents reported all detailed information for 

workers on the payroll during the reference period. On the revised instrument, the 150-day 

question was moved to after the question that asked about the number of workers on the payroll 

during the reference week and before detailed information was collected on those workers.

Figure 10. 150-Day Question

 Figure 11. 149-Day Question

When administrating the 150-day question using the original instrument, interviewers asked the 

question as worded 46 percent of the time (Table 10). However, when administering the 149-day 

question, interviewers failed to verify the response 75 percent of the time. When administering 
16

the 150-day question using the revised instrument, interviewers administered the question with 

major changes just over half the time (53%). 

Table 10. 149 and 150 Day Question Interviewer 

Behavior

Code Original Revised

150 149 150 149

ES 46% 0% 11% -

VER 11% 0% 6% -

NOVER 17% 75% 17% -

MC 20% 13% 53% -

SC 6% 13% 14% -

Total 35 16 36 -

When major changes were made to the 150-day question, interviewers often left off several of 

the important elements in the question, such as “in 2018,” “of these [insert number], “total paid 

workers,” “will be,” and “paid” (e.g., “How many of those workers will be paid for more than 

150 days of work?”). Often they changed “paid” to “work” (e.g., “Will they work 150 days or 

more?”). In a few interviews, the interviewer asked whether the workers typically worked more 

than 150 days (e.g., “Okay and do they usually those 2 workers work 150 days or more?”). 

Sometimes they paraphrased the question and asked whether the workers were full-time or parttime (e.g., “Are they full-time or part-time workers.) Finally, as with other questions, a few 

interviewers asked this question in a leading way (e.g., “In 2018, how many of these 4 total paid 

workers for that week will be paid by this operation for 150 days or more? So out of those 9 

workers, all of them?”). 

3.10 Peak Number of Workers

The peak number of workers question differed across versions. In the original version, the peak 

number question was asked at the very end of the questionnaire, after the value of sales and 

farm-type questions. Additionally, this question did not have any include or exclude statements 

(Figure 12). On the revised version of the questionnaire, this question was asked after all 

questions for the January reference period had been completed. This question had a list of 

include and exclude statements that interviewers were required to read (Figure 13). 

Figure 12. Peak Number of Workers Original Version
17

Figure 13. Peak Number of Worker Revised Version

The original version of this question was asked as worded more often (59%) than the revised 

version (5%). However, interviewers still made major changes to the original version 32 percent 

of the time (Table 11). Interviewers made major changes to the revised question 74 percent of 

the time and shortcutted the question 16 percent of the time. 

There were two common major changes made to the original version of this question. First,

interviewers often did not read the reference year when asking the question. The reference year is 

particularly important in this question since this question asks about the previous calendar year, 

whereas the majority of other questions on the survey ask about the current calendar year. 

Second, interviewers asked this question in a leading way, implying to respondents that they 

should report the same number of workers reported in previous sections of the survey (e.g., 

“During 2017, what was the largest number of hired workers, including paid family members, on 

the payroll on any one day. Was it all 9?”). When asked in a leading way, respondent sometimes

corrected the interviewers and other times they did not. 

When major changes were made to the revised version of this question, the most common 

change made was not reading the include/exclude statements. In fact, in all but one of the 

interviews where a major change was made, the interviewers did not read the include/exclude 

statements. In the one interview where the interviewer read the include/exclude statements, she

did not read any includes and read only one exclude. As with the original version of this 

question, interviewers administering the revised version also did not read the reference year and 

asked this question in a leading way (e.g., “And so would you say 19 workers would be about the 

largest number of workers you would have on any one day?”). 

3.11 H-2A

Table 11. Peak Number of Workers Interviewer 

Behavior

Code Original Revised

ES 59% 5%

VER 0% 5%

NOVER 0% 0%

MC 32% 74%

SC 9% 16%

Total 22 19
18

The H-2A question also appeared in different locations on the two versions. In the original 

version, this question was asked at the end of the questionnaire, following the peak number 

question (Figure 14). In the revised version, this question was asked after the January questions, 

following the peak number question. Additionally, on the revised version a definition for H-2A 

workers followed the questions and interviewers were required to read it (Figure 15). 

Figure 14. Original Version H-2A

Figure 15. Revised Version H-2A

Substantial differences were found in the administration of this question across versions (Table 

12). When administering the original version of this question, interviewers read this question as 

worded 59 percent of the time, compared to 5 percent of the time when administering the revised 

version. Interviewers made major changes to both versions of this question at high rates. 

However, major changes were made at a particularly high rate when administering the revised 

version (original: 32%, revised 74%). 

In the majority of interviews where a major change was made to the original version of this 

question, the interviewers did not read the reference year (e.g., “And did you have any H-2A

temporary agriculture workers on the payroll?”). Again, this is problematic since the reference 

year for this question is the previous calendar year, whereas other sections of the survey ask 

about the current calendar year. 

Although no definition was provided in the original version, some interviewers provided 

inaccurate definitions to respondents: 

Table 12. H-2A Workers Interviewer Behavior

Code Original Revised

ES 59% 5%

VER 0% 5%

NOVER 0% 0%

MC 32% 74%

SC 9% 16%

Total 22 19
19

“During 2017, did this operation have any H-2A temporary agricultural workers on the payroll? 

That would be we like migrant workers or any contract work.”

“Did you hire any H-2A temporary agricultural workers on the payroll... They would have come 

in and did temporary stuff on the payroll.”

Two interviewers did not reference the H-2A temporary agricultural worker program when 

asking the question and instead asked about migrant and temporary workers:

“Did you have any migrant workers?”

“Okay and so that means you didn't have any temporary workers on the payroll.”

For the revised version of this question, in all the interviews where a major change was made the 

interviewers did not read the required definition of H-2A workers. Additionally, interviewers 

often did not read the reference year. 

In one interview, the interviewer did not read “H-2A” when administering the question (e.g, 

“And did you have any temporary workers other than just the one?”). This changed the construct 

being asked about to “temporary workers” as opposed to the intended construct of workers in the 

federal H-2A temporary agricultural worker program. 

Finally, one interviewer asked this question in a leading way (e.g., “None of them are H-2A 

temporary agricultural workers?”). 

4. CONCLUSIONS AND RECOMMENDATIONS

This research report compares the administration of questions from two Agricultural Labor 

Survey CATI instruments. Both instruments were redesigned to improve flow and usability of 

the instruments with the goal of increasing interviewers’ ability to ask the questions in a 

standardized manner. In addition, the revised script contained two new wage questions that were 

proposed to be added to the Agricultural Labor Survey. The goal of the behavior coding was to 

compare the administration of the original Agricultural Labor CATI instrument and the revised 

instrument, with special attention given to the new wage questions. 

Similar to results found in the behavior coding of the October 2017 Agricultural Labor Survey 

(Ridolfo et al. 2020), neither of the instruments were administered appropriately at an acceptable 

rate. Interviewers failed to verify responses, made major changes to survey questions, and 

shortcutted survey questions (did not ask them at all) at high rates. Interviewers’ lack of 

adherence to the CATI instrument was particularly pronounced in questions that asked about the 

second reference period. Furthermore, the new wage questions were often not asked as worded 

or not asked at all. This made it difficult to evaluate these new survey questions. 

In addition, when interviewers made major changes to survey questions and responded to 

respondents’ requests for clarification, it was observed that they lacked an adequate 
20

understanding of the constructs being measured. For example, interviewers were observed 

providing inaccurate definitions and inclusion criteria, and selecting inappropriate worker types. 

This led to workers being inappropriately screened into and out of the survey, and being 

classified as the incorrect worker types, which could have larger implications for published wage 

rates. 

As with the behavior coding of the October 2017 Agricultural Labor Survey (Ridolfo et al. 

2020), a limitation to this research was that images of the interviewers’ screen were not available 

during behavior coding. This made it difficult to determine which screen interviewers were on, 

particularly when they deviated from the script. This slowed down the coding process and 

analysis as coding often had to be double checked and edited. 

Despite this limitation, substantial evidence from this and the past behavior coding study 

(Ridolfo et al. 2020) indicates further work is needed to improve the usability of the Agricultural 

Labor Survey CATI instruments. Additionally, interviewers would benefit from additional 

training on the content of the Agricultural Labor Survey and standardized interviewing. We 

recommend the following:

1) Conduct enumerator training that focuses on the importance of reading survey questions 

as worded and discourages the use of conversational interviewing to reduce interviewer 

variation. 

2) Conduct enumerator training on the content of the Agricultural Labor Survey.

3) Continue to conduct behavior coding on future cycles of the Agricultural Labor Survey.

5. REFERENCES

Biagas, D., Abayomi, E.J., Rodhouse, J., and Ridolfo, H. (2019), “Examining Interviewer Effects 

on the Agricultural Labor Survey: A Mixed Methods Approach,” Poster presented at the 

Interviewers and Their Effects from a Total Survey Error Perspective Workshop, Lincoln, NE.

Cohen, J. (1960), “Coefficient of Agreement in Nominal Scales,” Educational and Psychological 

Measurement, 20, 37-46.

Landis, J. R. and Koch, G. G. (1977), “The Measurement of Observer Agreement for Categorical 

Data,” Biometrics, 33, 159-174. 

Oksenberg, L., Cannell, C., and Kalton, G., (1991), “New Strategies for Pretesting Survey 

Questions,” Journal of Official Statistics, 7, 349-365.

Ongena, Y. P., and Dijkstra, W., (2006), “Methods of Behavior Coding of Survey Instruments,” 

Journal of Official Statistics, 22, 419-451.

Reist, B., Wilson, T., Ridolfo, H., Young, and L.J. (2018), “Findings for the 2018 Agricultural 

Labor Base Wage Question Experiments,” Washington DC: National Agricultural Statistics 

Service. 
21

Ridolfo, H., Biagas, D., Abayomi, E., and Rodhouse, J., (2020), “Behavior Coding of the 

October 2017 Agricultural Labor Survey,” Washington, DC: National Agricultural Statistics 

Service.

Sloan, R., (2017), “Agricultural Labor Survey Cognitive Interview Report,” Washington, DC: 

National Agricultural Statistics Service. 

Sloan, R., (2018), “2017 Agricultural Labor Survey Cognitive Interview Report,” Washington, 

DC: National Agricultural Statistics Service.
22

APPENDIX A: Original CATI Script

Ag Labor – Original Questionnaire – April 2018
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
55

APPENDIX B: Revised CATI Script

Ag Labor – Revised Questionnaire – April 2018
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103