Survey Burden and its Impact on Attitudes Toward the Survey

Sponsor

Jaki Stanley McCarthy and Daniel G. Beckler

_______________________________________________________________________________________

The organization sponsoring a survey data collection may affect respondents’ willingness to respond. USDA’s

National Agricultural Statistics Service (NASS) has begun asking agricultural establishments questions during

ongoing surveys about respondents’ knowledge and attitudes about NASS. These questions were asked of both

respondents and non-respondents to the surveys, and clearly showed a correlation between respondents’

knowledge and attitudes toward NASS and their willingness to cooperate when contacted by NASS. 

Past burden (number, length, complexity, frequency of contacts, etc.) imposed by NASS was also measured and

found to have little correlation with survey cooperation. It was hypothesized that increased contacts may provide

opportunities for public relations and help foster more positive attitudes towards the survey sponsor. If increases

in burden create more positive opinions of the survey sponsor, this may offset expected declines in

cooperativeness as burden increases. However, little relationship was found between respondents’ attitudes

toward NASS and the past burden imposed on them.

KEY WORDS: Respondent burden, accumulated burden, agricultural survey

______________________________________________________________________________________

1 INTRODUCTION1

 Survey methodologists have long speculated

about what factors affect survey respondents’

willingness to cooperate when contacted.

Attributes of the interviewer, the respondent,

the survey process or the external environment

may all impact survey cooperation. (For an

extensive review of these factors in household

surveys, see Groves and Couper, 1998.) In

interview surveys, the interviewer -

respondent interaction is critical to gaining

cooperation. Interviewers are often free to

introduce an interview in whatever way they

feel is suitable. The survey introduction may

include any number of different appeals

intended to increase cooperation. Groves,

Cialdini, and Couper (1992) have argued that

many fall into one of six principles of

compliance: reciprocation, consistency,social

validation, authority, scarcity and liking. One

method interviewers report using to gain

cooperation is to tailor the interaction (and

consequently which compliance principles are

used) according to the particular respondent.

Snijkers, Hox, and de Leeuw (1999) studied

the tactics that high performing survey

interviewers use to gain cooperation. Similar

to Groves, et al. (1992) they found that

tailoring the interaction was important.

However, they also found that mentioning

Statistics Netherlands as the survey sponsor

was rated as a highly effective means of

securing cooperation. Successful interviewers

also felt that the agency should pay more

attention to public relations and thus, “the

image of the agency is seen as a tool to work

with and attain a better response rate.”

 One of the critical components to a survey

introduction is the identification of the

sponsoring organization and explanation of

the survey purpose. It has generally been

assumed that government or university

1

Paper presented at the International Conference

on Establishment Surveys II, June 2000. Both authors are

with the National Agricultural Statistics Service, Research

and Development Division. Jaki Stanley McCarthy is the

Head of the Data Quality Research Section. Daniel Beckler

works in that section. 
sponsorship of a survey increases cooperation.

However, to date, there is little empirical

evidence to show what the relationship

between survey sponsorship and cooperation

is. The Census Bureau found there was not a

significant correlation between reported

knowledge of the Census and likelihood of

returning a Census mail form (Bates and

Buckley, 1999). However, in an evaluation of

response to the 1990 Census, respondents’

attitudes toward the Census Bureau’s handling

of data confidentiality and privacy were

somewhat correlated to both self reported

Census returns (Fay, Bates and Moore, 1991)

and actual Census returns(Singer,Mathiowetz

and Couper, 1993). Harris-Kojetin and

Tucker (1999) also found that in times of

more positive public opinion regarding the

government and government leaders,

cooperation rates on the Current Population

Survey, a major government survey, were

higher.

 While there may be little research to show

positive effects of changing respondents’

attitudes about a survey sponsor, Federal

agencies do feel that public relations and

publicity are important. Some agencies

devote more to this than others, but the

importance placed on this is evident in the

Census Bureau’s budget of $167 million in

advertising designed to promote the 2000

Census.

 Survey respondents representing

establishments may be somewhat different

from respondents representing themselves or

their households. These differences may

make attitudes toward the survey sponsor

more important than in general household

surveys. For example, establishment survey

respondents, particularly large or unique ones,

may be contacted much more frequently by an

organization than household respondents.

They may also use or be more directly affected

by the survey results (see Interagency Group

on Establishment Nonresponse, 1999 for an

extensive discussion of issues relevant to

nonresponse in establishment surveys).

 Within the agricultural establishment

population, research done in the late 1970's

indicated that farmers’ self reported

participation in USDA surveys was related to

how well they felt the survey purpose had

been explained to them, whether they

themselves used USDA reports, and their

opinion of whether or not they thought

governmentreportswere generally worthwhile

(Jones, Sheatsley and Stinchcombe, 1979).

This is similar to findings in household

surveys, where survey cooperation has been

linked to respondents’ general attitudes

toward the usefulness of surveys (Frankel and

Sharp, 1981).

 Aside from speculation about the influence

of survey sponsorship on survey response, a

commonly held belief is that increased burden

is negatively correlated with survey

cooperation. Federal Statistical Agencies are

currently striving to lessen the reporting

burden placed on respondents. The U.S.

Office of Management and Budget has set a

government-wide goal of five percent yearly

reduction in information collection burdens

(Paperwork Reduction Act of 1995). Burden

may be defined in a number of ways – length

of the interview or questionnaire, number of

contacts, difficulty in reporting the requested

data, etc. The length of the survey

questionnaire is often assumed to be positively

correlated with survey non-response.

However, a literature review by Bogen (1996)

found that while this claim was supported in

some studies, other studies showed exactly the

opposite and still others showed no

relationship. Frankel and Sharp (1981) also

found the length of a single completed survey

interview was related to expressedwillingness
to participate in later interviews. Respondents

who participated in a 25 minute interview

were more likely to agree to participate in a

future interviewthan respondents participating

in a 75 minute interview. However, there was

little difference in actual cooperation between

those who had the long or short initial

interview when later contacted for the second

interview. 

 USDA’s National Agricultural Statistics

Service (NASS) publishes official statistics

based on data collected from farmers, ranchers

and other agribusiness operators in voluntary

surveys. For example, the Quarterly

Agricultural Survey collects data on inventory

and production; the Farm Labor Survey

collects information on hours worked and

wages; the Agricultural Resource

Management Study collects information on

production practices, chemical and pesticide

use, and farm economics. A particular

agricultural establishment may be selected for

any or all of these surveys, both within a

single year and over multiple years. 

 In NASS surveys, there is little evidence that

accumulated burden contributes to later nonresponse. Reasons given for refusing to

participate in NASS surveys are more often

“too busy/lack of time,” or privacy concerns

than frequency or number of survey requests

(Jones, Sheatsley and Stinchcombe, 1979

O’Connor, 1991, 1992). Our own research

has shown almost no correlation between

traditional measures of burden (for a set of

contacts made by NASS with farm and ranch

operations over a two and one half year

period) and response on a subsequent survey

(McCarthy and Beckler, 2000).

 However, increased burden in either

increased number of contacts or time spent

with the respondent in the interview setting,

may also provide an opportunity for the survey

sponsoring agency to promote the survey or

agency. This may in turn change respondents’

attitudes for the better with increased burden,

instead of simply making respondents less

likely to cooperate. If this is the case, we

might expect declines in cooperation due to

increased burden to be offset with more

cooperative attitudes due to a more positive

view of the survey agency and its mission.

Perhaps the public relations work that may be

part of our survey contacts can mitigate

negative effects of potential increases in

survey burden. 

 This paper looks first at the accumulated

burden imposed on agricultural operations

selected as respondents in our surveys and at

the relationship between respondents’

attitudes about NASS as the survey sponsor

and their survey cooperation. Based on prior

NASS research, we expect little relationship

between burden and cooperation, but a strong

correlation between respondents’ attitudes and

cooperation. In addition, this paper looks at

the relationship between the burden imposed

by NASS and the attitudes held by

respondents (and non-respondents). Our

hypothesis is that since burden does not

appear to be negatively correlated with

cooperation, as burden increases, respondents’

attitudes become more positive to offset

expected declines in cooperation.

2 METHODOLOGY

 Interview disposition (not sampled,

completed,refused, non-contact) was recorded

for each South Dakota agricultural operation

selected for the Quarterly Agricultural

Surveys, the Agricultural Labor Surveys, the

Hog Surveys, the Cattle and Sheep Surveys,

the Monthly Cattle on Feed Surveys, the

Agricultural Resource Management Studies

(ARMS), and the Yield surveys conducted by

NASS between January 1997 and December
1999. South Dakota was selected because they

have historically had a relatively high rate of

survey refusals. Details about these surveys

appear are available from the authors, but they

collect data on agricultural inventory,

production and economics. There were 118

total possible surveys in this set. 

 Samples were stratified based on surveyrelated control data (usually size and type of

operation) maintained on the list frame. Mode

of contact varied, but involved primarily

telephone (CATI), with limited face-to-face

and mail for most surveys. The exception is

the ARMS, which are all face-to-face

interviews. The sensitivity and difficulty of

the surveys also varied. Except for the ARMS,

all data collected are generally readily

available to farmers and ranchers. The ARMS

collects extremely detailed income, expense

and debt information, in addition to other

potentially sensitive questions on pesticide

and chemical use. The samples were not all

independent with some surveys samples

overlapping. However, respondents are not

notified of potential additional later survey

contacts.

 Questions about respondents’ knowledge and

opinions of NASS were added to the South

Dakota Quarterly Agricultural Survey (QAS)

for hogs and crops, the January and July Cattle

Survey, and the Sheep Survey, all of which

collect data on agricultural production and

inventory. We selected these surveys because

they include a broad cross section of

agricultural operations by commodity and

size. Beginning with the June 1998 QAS

Crops/Stocks Survey, we added a series of

questions designed to measure operations’

familiarity with NASS and what we do, and

their opinion of our work. Questions were

asked of all sampled operations, both those

who did and did not provide survey data.

Many operators who did not provide survey

data answered these questions. For this

analysis, we considered only those opinions

given in the time period June 1999 to

December 1999 (at the end of our period of

study) since this would reflect attitudes of

respondents with the most possible

accumulated burden.

 In order to benefit from the largest sample

size possible, attitudinal data were utilized

from multiple surveys, each with distinct but

overlapping populations. All analyses used

unweighted data and thus does not permit

expanding our results to our entire population

of farm operations. However, the data set used

for the analyses is representative of a set of

farm operations sampled for surveys NASS

conducts. In this respect, our results reflect the

interactions of attitudes, respondent burden

and response on NASS’ surveys.

3 RESULTS

3.1 Assessment of Accumulated NASS

Survey Burden

 Table 1 shows the number of agricultural

operations who were contacted one or more

times by NASS in South Dakota between

January 1997 and December 1999. As shown

in the table, of the 14,728 operations

contacted, the highest percentage of

operations (34.3%) were contacted only once

during this time period. The maximum

number of times an operation was contacted

was 57 but 67% of the operations were

contacted three times or less. Only 6.6% of the

operations were contacted nine times or more.
Table 1: Counts of Number of Surveys Operations Were Contacted For

Surveys Count % of

Total Surveys Count % ofTotal Surveys Count % ofTotal

1 5,056 34.3 11-12 201 1.4 31-32 8 0.1

2 2,559 17.4 13-14 105 0.7 33-34 6 0.0

3 2,233 15.2 15-16 57 0.4 35-36 2 0.0

4 1,467 10.0 17-18 32 0.2 37-38 10 0.1

5 1,003 6.8 19-20 26 0.2 39-40 9 0.1

6 673 4.6 21-22 13 0.1 41-42 28 0.2

7 438 3.0 23-24 9 0.1 43-44 14 0.1

8 328 2.2 25-26 6 0.0 45-46 8 0.1

9 258 1.8 27-28 10 0.1 47-57 13 0.1

10 151 1.0 29-30 5 0.0 Total 14,728 100

3.2 Confirmation of Prior Research

We began our analyses by replicating previous

research which addressed accumulated burden

and respondents’ awareness of, and attitudes

towards NASS. For the last survey in our

selected time period (December 1999 QAS

Crops/Stocks), respondents and refusals did

not differ in NASS imposed accumulated

burden over the period from 

January 1997 to December 1999 (measured in

number of contacts, total amount of time or

total number of surveys selected for). These

findings confirm earlier work (McCarthy and

Beckler, 2000). Also as shown in prior

research (McCarthy, Ott, and Johnson, 2000),

respondents’ attitudes and knowledge of the

survey sponsor (in this case, NASS) are

correlated with their willingness to respond. 

3.3 Relationship between Respondent

Attitudes and Burden

We then examined the relationship between

burden and attitudes toward the survey

sponsor. Respondents were classified into

three groups based on the number of times

they had been contacted. Low burden

respondents were defined as having been

contacted one to three times in the three year

period. Medium burden respondents had been

contacted four to eight times, and high burden

respondents were those contacted nine or

more times. 

Respondents were asked if they recalled ever

being contacted by us. As you would expect,

those respondents who had been contacted

most often (nine or more times) had higher

recall of having been previously contacted

(Table 2). This held true for all respondents.

Respondents who recalled being contacted,

were asked about the type of contact they had

in the past (Table 3). The two main types of

contact were telephone and face-to-face

contacts. Those who had highest burden had

higher recall of face-to-face contact. This also

held true for all respondents. This is not

surprising, since face-to-face interviews are

the least common mode of data collection and

those with fewer contacts are less likely to

have ever been contacted in person. 

When asked about their opinions of the

accuracy of NASS reports, there was no

relationship between respondents’ attitudes

and the amount of burden imposed on them

(Table 4). Again, this was true for all groups

of respondents, regardless of their cooperation

rate.
Table 2: Recall of Prior Contact With SDASS

 NASS 1/

Burden

Yes No Don’t Know Total Count Row % Count Row % Count Row %

Low 410 67.43 84 13.82 114 18.75 608

Medium 494 72.33 52 7.61 137 20.06 683

High 279 83.53 15 4.49 40 11.98 334

Overall 1,183 72.80 151 9.29 291 17.91 1,625

1/ Measured by NASS contacts made between January 1997 and December 1999.

Table 3: Type of Contact with SDASS 2/

NASS 1/

Burden

Telephone Personal Visit Producer Group

Meeting State Fair Other

Total

Count Row% Count Row% Count Row% Count Row% Count Row

%

Low 379 82.75 67 14.63 4 0.87 3 0.66 5 1.09 458

Medium 446 78.38 105 18.45 7 1.23 6 1.05 5 0.88 569

High 182 52.30 157 45.11 0 0.00 5 1.44 4 1.15 348

Overall 1,007 73.24 329 23.93 11 0.80 14 1.02 14 1.02 1,375

1/ Measured by NASS contacts made between January 1997 and December 1999.

2/ Respondents were permitted to answer with more than one type of contact.

Table 4: Opinion of Accuracy of NASS Reports

 NASS 1/

Burden

Almost Always

Accurate Mostly Accurate

Accurate as

Often as

Inaccurate

Mostly 

Inaccurate

Almost Always

Inaccurate Total

Count Row

%

Count Row% Count Row% Count Row% Count Row

%

Low 39 8.23 225 47.47 161 33.97 36 7.59 13 2.74 474

Medium 46 5.60 396 48.18 300 36.50 55 6.69 25 3.04 822

High 22 4.75 229 49.46 165 35.64 32 6.91 15 3.24 463

Overall 107 6.08 850 48.32 626 35.59 123 6.99 53 3.01 1,759

1/ Measured by NASS contacts made between January 1997 and December 1999.

Table 5: Opinion of Importance to Respond to NASS Survey Requests

NASS 1/

Burden

Very Important

to Respond

Sometimes

Important to

Respond

Response Does

Not Matter

Somewhat

Important Not to

Respond

Very Important

Not to Respond Total

Count Row% Count Row% Count Row% Count Row% Count Row

%

Low 192 14.92 478 37.23 380 29.60 69 5.37 165 12.85 1,284

Medium 121 12.91 322 34.36 288 30.74 59 6.30 147 15.69 937

High 81 18.12 189 42.28 107 23.94 22 4.92 48 10.74 447

Overall 394 14.77 989 37.07 775 29.05 150 5.62 360 13.49 2,668

1/ Measured by NASS contacts made between January 1997 and December 1999.

Table 6: Opinion of Whether NASS Reports Help or Hurt Farmers

NASS 1/

Burden

Helps Farmers Hurts Farmers Sometimes Helps,

Sometimes Hurts

Has No Effect on

Farmers Don’t Know

Total

Count Row

%

Count Row% Count Row% Count Row% Count Row

%

Low 83 13.70 113 18.65 183 30.20 33 5.45 194 32.01 606

Medium 77 11.42 116 17.21 230 34.12 40 5.93 211 31.31 674

High 43 12.91 51 15.32 139 41.74 19 5.71 81 24.32 333

Overall 203 12.59 280 17.36 552 34.22 92 5.70 486 30.13 1,613

1/ Measured by NASS contacts made between January 1997 and December 1999.
 Respondents were also asked whether it was

important that they respond when contacted,

that it didn’t matter, or that it was important

that they did NOT respond when contacted.

The only relationship between burden and this

attitude was found in the highest burdened

group. Those with the highest burden (again,

nine plus contacts) felt it was somewhat or

very important to respond more often than

those with less burden. See Table 5. 

 Finally, respondents were also asked if they

thought what NASS does helps farmers, hurts

farmers, does both or don’t know. There was

no apparent relationship between answers to

this question and burden (Table 6).

4 DISCUSSION AND CONCLUSION

 In this paper we were able to confirm prior

NASS research relating burden and attitudes

toward the sponsor to survey response. Again,

we found there was no apparent correlation

between the prior burden imposed on

respondents and their likelihood ofresponding

on a subsequent survey. We also confirmed

the hypothesis that respondents with more

positive attitudes toward NASS were more

likely to respond to survey requests. 

 We hypothesized that increased contacts

might serve to change potential respondents’

attitudes toward us, making them more

favorable over time. This, in fact, might be

the reason why we did not find the

relationship between increased burden and

decreased cooperation. While we did find that

respondents are more likely to recall being

contacted by us if they have been contacted

more often, we did not find that many

attitudes toward us as the survey sponsor

changed as the number of times we contacted

them increased. While we hypothesized this

may have masked the effect of increased

burden on cooperation, our results indicate

this is not the case. 

 What this does suggest to us, is that we are

not effectively using our contacts with

respondents as opportunities for public

relations and promotion. NASS does not

typically design standard materials that can be

delivered to respondents in pre-survey

mailings or during interviews. However,

since many respondents are contacted

repeatedly over time, effective promotional

materials might be a way to increase

cooperation with them on later contacts. The

respondents remember that we contact them,

but they are not being persuaded to have

different opinions of us. If respondents’

attitudes are truly related to their willingness

to participate in our surveys, we need to seize

the opportunities available to us to change

them for the better.

5 REFERENCES

Bates, N. and S. Buckley (1999), “Reported

Exposure to Paid Advertising and

Likelihood of Returning a Census

Form,” Proceedings of the 1999

Annual Conference of the American

Association for Public Opinion

Research, American Statistical

Association.

Bogen, K. (1996), “The Effect of

Questionnaire Length on Response

Rates – A Review of the Literature,”

Proceedings of the Section on Survey

Research Methods, American

Statistical Association, pp. 1020-1025.

Fay, R., N. Bates, and J. Moore (1991),

“Lower Mail Response in the 1990 Census: A

Preliminary Interpretation,” Proceedings of

the Bureau of the Census Annual Research

Conference, U.S. Bureau of the Census, pp. 3-

32.
Frankel, J. and L. Sharp (1981),

“Measurement ofRespondent Burden:

Summary of Study Design and Early

Findings,” Statistical Reporter, 81(4),

pp. 105-111.

Groves, R., R. Cialdini, and M. Couper

(1992), “Understanding the Decision

to Participate in a Survey,” Public

Opinion Quarterly, 56(4), pp. 475-

495.

Groves, R. and M. Couper (1998),

Nonresponse in Household Interview

Surveys, New York: J. Wiley and

Sons, Inc. 

Harris-Kojetin, B. and C. Tucker (1999),

“Exploring the Relation of Economic

and Political Conditions with Refusal

Rates to a Government Survey,”

Journal of Official Statistics, 15(2),

pp. 167-184.

Heberlein, T. and R. Baumgartner (1978),

“Factors Affecting Response Rates to

Mailed Questionnaires: A Qualitative

Analysis of the Published Literature,”

American Sociological Review, 43(4),

pp. 447-461.

Interagency Group on Establishment

Nonresponse (1999), Establishment

Nonresponse: Revisiting the Issues

and Looking into the Future.

Statistical Policy Working Paper 28,

NTIS Document PB99-132029,

Seminar on Interagency Coordination

and Cooperation, Washington, DC:

U.S. Office of Management and

Budget.

Jones, C., P. Sheatsley, and A. Stinchcombe

(1979), Dakota FarmersandRanchers

Evaluate Crop and Livestock Surveys,

National Opinion Research Center

Report No. 128, Chicago: National

Opinion Research Center.

McCarthy, J. and D. Beckler (2000). An

Analysis of the Relationship Between

Survey Burden and Nonresponse: If

we bother them more are they less

cooperative? RDD Research Report

RDD-00-02, Washington, DC:

National Agricultural Statistics

Service, U.S. Department of

Agriculture.

McCarthy, J., J. Johnson, and K. Ott (2000,

forthcoming), Survey Participation

and Survey Sponsorship: What do Our

Respondents Think of Us and Should

We Care? Washington, DC: National

Agricultural Statistics Service, U.S.

Department of Agriculture.

O’Connor, T. (1991), Identifying and

Classifying Reasons for Nonresponse

on the 1990 Farm Costs and Returns

Survey, SRB Research Report SRB91-11, Washington, DC: National

Agricultural Statistics Service, U.S.

Department of Agriculture.

O’Connor, T. (1992), Identifying and

Classifying Reasons for Nonresponse

on the 1991 Farm Costs and Returns

Survey, SRB Research Report SRB92-10, Washington, DC: National

Agricultural Statistics Service, U.S.

Department of Agriculture.

Paperwork Reduction Act of 1995, 44 U.S.C.

§3505(a)(1).

Singer, E., N. Mathiowetz, and M. Couper

(1993), “Privacy, Confidentiality, and

the 1990 U.S. Census,” Public

Opinion Quarterly, 57(4), pp. 465-

482.
Snijkers, G., J. Hox, and E. de Leeuw (1999),

“Interviewers’ Tactics for Fighting

Survey Nonresponse,” Journal of

Official Statistics, 15(2), pp. 185-198.