Using Respondent Requests for Help to Develop Quality Data

Collection Instruments: 

The 2000 Census of Agriculture Content Test

Jaki Stanley McCarthy

__________________________________________________________________

Responsibility for the United States Census of Agriculture was transferred from the U. S. Census Bureau to the

U.S. Department of Agriculture's National Agricultural Statistics Service (NASS) for the 1997 Census.

Substantial changes are planned for the report forms and instructions to be used in the following 2002 Census

of Agriculture. Changes include content, wording, and format of the report forms.

Due to the magnitude of the changes to the 2002 report form, a multi-phase test was conducted to evaluate and

improve the quality of the data collected with this form. This Census Content Test was conducted in three

phases: 1) cognitive pretesting, 2) split panel pilot test with 3 versions of a new draft report form, and 3) follow

up interviews. 

A toll free telephone number respondents could call for assistance was printed on all of the report forms mailed

out in the second phase of the Census Content Test. This number was staffed by NASS enumerators throughout

the data collection period. Calls made by respondents to the help line were evaluated as part of the Census

Content Test. The number and type of problems that respondents reported in calls to the toll free telephone line

was compared across the different versions of the forms and for different types of respondents. Examples of

problems which indicated areas where forms or instructions might be improved are provided. How this evaluation

was used to support other types of evaluations in the content test is also discussed.

KEY WORDS: data quality, questionnaire design, questionnaire evaluation

__________________________________________________________________

BACKGROUND AND

INTRODUCTION1

 Responsibility for the United States Census

of Agriculture was transferred from the U. S.

Census Bureau to the U.S. Department of

Agriculture's National Agricultural Statistics

Service (NASS) for the 1997 Census. The

quinquennial Census of Agriculture collects

information from all agricultural operations in

the United States using a mailed self

administered report form. Information is

collected on land acreage and use, crops,

livestock, sales of agricultural products,

income, production expenses, assets, and

characteristics of the farm operators. 

 Data collection instruments and instructions

for the 1997 census had already been

developed by the Census Bureau and were

based on report forms used in past censuses.

Due to the timing of the transfer, these report

forms were used with only minor

modifications for the 1997 Census. 

 However, substantial changes were planned

for the report forms and instructions to be

used in the following 2002 Census of

Agriculture. Traditionally, changes to the

information collected on the Census of

Agriculture report forms are made due to

requests from data users, emerging issues in

1This paper was presented at the 2001 Annual Conference of

the American Association for Public Opinion Research,

Montreal Canada, May 2001. Jaki Stanley McCarthy is with

the National Agricultural Statistics Service, Research and

Development Division. She is the head of the Data Quality

Research Section.
agriculture and problems identified in the

previous Census. For the 2002 Census, NASS

was also interested in collecting data in as

much the same way as possible as they are

currently being collected in ongoing survey

programs. In addition, the report form was

being formatted to allow data to be scanned

and captured from the paper forms using

Optical Character Recognition software in lieu

of being key entered as in the past. 

 Due to the magnitude of the changes to the

report form for the 2002 Census, a multiphase test was done in 2000-2001 to evaluate

and improve the quality of the data collected

on this form. This Census Content Test was

conducted in three phases: 1) cognitive

pretesting, 2) split panel pilot test with 3

versions of the new report form mailed to

respondents, and 3) follow up cognitive

interviews. 

 During the second phase of the Content Test,

a toll free telephone number that respondents

could call for assistance was printed on all of

the report forms. This help line was set up

and answered by trained enumerators

throughout the data collection period. An

evaluation of the calls made by respondents to

the toll free telephone help line was used in

conjunction with analysis of report form data

and the follow up interviews to determine

areas where forms or instructions might be

improved. This paper discusses how

information from the toll free help line was

gathered and used.

2 Methods

2.1 Mailing 

 Report forms were sent out to 14,799 farm

operators during December 2000. All states

except Alaska and Hawaii were included. A

postcard reminder/ thank you was sent a week

following the first mailing. Nonrespondents

were also mailed a second report form on

February 5, 2001. 

2.2 Sample

 The mailout sample was selected in two

parts. A general sample from NASS’s list

frame of farm and ranch operators was

selected and another special sample of

operations with specific characteristics was

also selected. The general sample was

stratified by value of annual sales for the

operation (descriptive information which is

carried on the list frame). Operations were

selected from all states except Alaska and

Hawaii. There were 13,568 operations in the

general sample.

 The special sample consisted of operations

hypothesized to have potential reporting

problems. These were of several types,

including operations that might not qualify as

farms (per USDA definition), operations that

raised unusual or specialty commodities,

operations that had reported questionable data

in the past, or those with atypical operating

arrangements. There were 35 types of special

cases2 with a total of 1,231 operations selected

for the special sample.

2.3 Data Collection Instruments 

 During the second phase of the Content Test,

respondents were mailed Test report forms.

Three different report forms (or panels) were

used, and within each panel there was a long

and a short form. All forms carried a due date

of February 5, 2001. The panels differed in a

2

Special cases were: nonfarms, institutional farms,

respondents involved in multiple operations, partnerships,

managed operations, tenants, operations with migrant workers,

cross state operations, operations with direct sales to consumers,

small farms, large farms, contractors, contractees, feedlots, sheep,

nursery and greenhouse, citrus grove owners, single rare

commodities (16 types), operations with grazing land and

operations who reported questionable yield and acreage data in the

past. 
number of ways both in format and content.

(A list of specific differences and copies of the

report forms can be obtained from the author.)

Respondents from the general sample were

equally assigned to each of the panels (and

divided across the long and short forms).

Respondents in the special sample were all

mailed long forms.

 The report forms collected comprehensive

information about the respondent’s operation

including land utilization, crop and livestock

inventory and production, sales, operating

expenses, characteristics and demographic

information about the farm operator.

2.4 Follow up Interviews 

 A subsample of the general sample

respondents and all of the special case

respondents who returned report forms were

selected for follow up interviews. During this

interview, the respondent was shown their

completed phase 2 report form and asked

specific questions about the data they had

provided. Most of these questions were

designed to determine whether the respondent

had understood and answered the questions

accurately. Additional questions were asked

of all special case respondents related to their

type of operation and particular suspected

reporting problems. A total of 3471 operations

were selected for possible follow up. Of

these, 1075 completed a report form and were

eligible for follow up and 657 phase 3 follow

up interviews were completed. 

2.5 Toll Free Help Line 

 A toll free telephone number was established

to provide assistance to respondents during

data collection of the mail out. The telephone

number was printed on both the front and back

pages of each of the panel report forms. For

panel #3, the toll free number was also printed

on the top of each page of the report form.

Calls were taken beginning on December 15

and ending on March 2. 

 When respondents called the toll free help

line, they were asked for their identification

number or name. This was to enable each call

to be matched back to other information about

that respondent, including the type of form

they had been mailed, the type of operation

they were, whether they ultimately mailed

their forms in, and the data they reported. For

each call, the enumerator determined the

caller’s stated reason for calling. This was

categorized into one of the following reasons:

C Needs help completing the form

C Form does not apply to respondent

(i.e. respondent is not a farm)

C Operation has gone out of business

during or prior to 2000

C Questions about reporting for

partnerships

C Questions about reporting land in

Conservation Reserve Program or

Wetlands Reserve Program 

C Questions about receipt of multiple

forms

C Request for time extension

C Request for additional or replacement

form

C Respondent reports already having

sent form in

C Questions about mandatory reporting

C Request from Congressional or

Senatorial office, questions about

Freedom of Information Act or request

to speak to NASS manager

C Question about confidentiality of data

(Title 7)

C General Complaint and other

situations

 If the caller wanted help in completing a

particular section or sections of the report

form, the enumerator also recorded the section
and item the caller asked about and the

specific problem they reported. Finally, the

enumerator recorded whether they resolved

the call, whether additional follow up was

needed from them after the call, whether the

call was referred to someone else, or some

other action was taken. 

3 RESULTS

 From December 15 through March 2, 814

calls were received and logged on the toll free

line. (A small but unknown number of calls

were not recorded due to technical difficulties

during the first week. Calls received after

March 2 were not recorded.) The table below

shows the number of forms mailed and calls

received by version.

 There was a significantly higher percentage

of calls from respondents mailed the long

form than the short form. However, there did

not seem to be any difference between the

three panel versions of the forms. 

 Contrary to expectations, printing the toll

free number on every page of the form

(Version 3), did not generate more calls than

from the forms with the toll free number only

on the form cover and back page. This

indicates that the placement of the toll free

number did not affect the likelihood that

respondents would call. 

 Special sample respondents did not seem to

call at a higher rate than the general sample

respondents who received the same versions

of the report form. P

2

 (5, N=14799) = 5.05,

p=.41
Form Version

Version 1,

Short Form

Version 2, 

Short Form

Version 3,

Short Form

Version 1,

Long Form

Version 2, 

Long Form

Version 3,

Long Form

General

Sample

Number

of Forms

Mailed

2271 2233 2230 2281 2281 2272

Number

(%) of

Calls

Received

113

4.98%

86 

3.85%

100

4.48%

153

6.71%

132

5.79%

138

6.07%

Special

Case

Sample

Number

of Forms

Mailed

0 0 0 1116 60 55

Number

(%) of

Calls

Received

85

7.62%

4

6.67%

3

5.45%

 

While overall the special case sample did not

make more calls to the toll free line than the

general sample, a few types of special case

respondents did call at a much higher rate. 

The number of calls by reason for the call for

the entire sample and for a few of these

special cases is shown below:

Overall, the number one reason that people

called the toll free line (327 calls) was to

report that they had already mailed in their

form, followed by calls for people

requesting help completing their forms.

Of the 75 people requesting additional forms,

at least 62 of these calls were generated by

people who received a reminder post card but

had not yet received the actual report form.
Number (%) of Calls

Reason for the Call All Aquaculture Horses Non-Farm

Claimed Form Already Filed 327 (40.2%) 2 (22.2%)

Needs Help Completing the Form 164 (20.1%) 4 (33%) 1 (25%) 1 (11.1%)

Requests an Additional Form 75 (9.2%) 2 (16.7%)

Claims to be Out of Business 66 (8.1%) 1 (25%) 1 (11.1%)

Thinks the Form Does not Apply 66 (8.1%) 6 (50%) 2 (50%) 4 (44.4%)

Wants to Know if Reporting is

Mandatory

30 (4.4%)

Wants Time Extension 25 (3.1%)

Questions about Reporting for

Partnerships

9 (1%)

Questions about Receipt of

Multiple Forms

8 (1%)

Questions about Reporting Land

in Federal Programs

7 (.9%)

Other Reasons 37 (4.5%) 1 (11.1%)

TOTAL (percent of entire

mailout)

814 (5.5%) 12 (42.9%) 4 (13.3%) 9 (11.25%)

 Calls for Help in Completing the Form.

Overall, the second most common reason for

calling was 164 calls from respondents

requesting help in completing their forms.

Fifty three of these calls were for general help

or help on the entire form. 

 For specific sections of the form, most of the

calls (59) were for help with the first section of

the form which defines the acreage for which

the respondent is to report. In this section the

respondent reports (a) the number of acres they

own, (b) the number they lease from others,

and (c) the number of acres they lease to others.

For the remainder of the report, they are to

report agricultural activity on the acres derived

from (a) + (b) - (c). If this number equals zero,

respondents are asked if they operated during

the year. If they did not, they should not

complete most of the rest of the form. The

routing and skip patterns for these questions

were complex and had not previously been

used on the form. This series of questions was

problematic in the Phase 1 pretests of the form
and nine of these calls referred specifically to

confusion with these questions.

 In the Phase 3 follow up interviews, problems

indicated by these phone calls were also

confirmed with 28% of 597 follow up

interview respondents reporting this as not

easy to answer. Since this problem was

identified both by calls to the toll free line and

in the follow up interviews, recommendations

were made to significantly alter or eliminate

this series of questions. 

 Many of the calls (11) about the first section

were about whether or not to report agriculture

on land that was leased to others. This also

appeared as a problem in the Phase 3 follow up

interviews, where 44% of the 126 people

reporting land rented out incorrectly reported

activity on these acres (which should have

been reported by the tenant operating the land.)

 In addition, in the Phase 3 follow up

interviews 3 out of four “potential non-farms”

(operations with agricultural activity under

$1000, landlords only, retired fromagriculture,

or otherwise not believed to be currently

operating) stated that they (incorrectly)

reported information for land which they did

not operate. This information led to

recommendations to add clear and specific

instructions to the form, the instruction sheet,

or both, regarding owned agricultural land that

is not operated by the respondent.

 While overall, the special sample cases did

not make more calls to the toll free line, some

specific types of special case operations did.

As shown in the table, operations classified as

aquaculture operations, horse operations, or

non-farms called at a much higher rate than the

sample as a whole. Calls from special sample

cases also revealed some of their specific

problems. 

 Calls from aquaculture operations were

primarily from State or Federal fish hatcheries.

These operations were unsure whether they

should report or the State/Federal parent

agency should report. They were also unsure if

they should report their activity if the fish were

never sold (i.e. used for restocking). Problems

were also reported because practices dealing

with fish differ from traditional livestock and

the form did not accommodate this (i.e. units

were acres, not gallons; salt was used to treat

water but is not traditionally considered

fertilizer, etc.) These calls led to a

recommendation to provide specific

supplementary instructions to these types of

operations with their forms.

 Two of the four calls from operations

classified as having horses were from

respondents who thought the form did not

apply to them. This was not unexpected, since

these respondents may not consider themselves

a farm, even if they are classified that way by

the USDA. These types of special operations

may also need individualized special

instructions, while other types of special

operations did not appear to have reporting

problems and may not require special

treatment.

4 DISCUSSION AND CONCLUSION

 During the 2000 Census Content Test we

attempted to use information about the calls

made by respondents to support other

information from analyses of data collected on

the forms mailed out and in follow up

interviews. A relatively small percentage of

the respondents who we mailed forms to called

the toll free number. Since a respondent must

take action and initiate this telephone call, we

felt that any problems reported on the toll free

number were probably relatively serious.

Information gained from other sources in the

Content Test was evaluated in conjunctionwith
the information about the telephone calls.

Problems that were indicated by multiple

sources got thorough review and almost

universally lead to recommendations for

changes to the report forms, instructions or

both. 

 Types of operations which called the toll free

number at a higher rate than the sample as a

whole were also examined closely.

Recommendations for these operations

included changes to the general reporting

instructions and providing specific targeted

instructions for these types of operations. 

 The calls received on the toll free line also

led to recommendations for things that should

be considered in development of procedures,

edit and analysis systems, instructions, and

information that should be made available for

the toll free operators during the 2002 Census.

For example, the high number of respondents

calling claiming that they had already returned

their forms indicates that the mail list for the

second mailing probably did not include a

large number of returns received just prior to

the generation of the mail list. This lag

between the generation of the list for the

second mailing and the date of the second

mailout was clearly too long in the Content

Test. For the 2002 Census of Agriculture, this

will be considerably shortened.

 Another example was the high number of

calls from respondents who had received the

follow up post card but had not yet received

their form. This was likely due to local postal

employeesdelivering the small postcard before

the large and bulky Census of Agriculture

packages during the busy December mail.

This prompted a recommendation to change

the date of the postcard mailing to two weeks

following the initial mailing instead of one.

 Overall, information about calls to the toll

free line provided both unique and

corroborative information used to improve the

2002 Census of Agriculture. While normally

viewed as an aid to respondents, the toll free

line can and should be used as an aid to survey

designers as well.