The Effect of Different 

Correspondence Methods 

on Response Rates 

in the 2010 Census of 

Agriculture Content Test

Nancy J. Dickey 

Zulma T. Riberas

HoaiNam N. Tran

United States 

Department of

Agriculture

National

Agricultural

Statistics

Service

Research and

Development Division

Washington DC 20250

RDD Research Report

Number RDD-11-09

April 2012

This document was prepared for limited distribution to the research community outside the United 

States Department of Agriculture. The views expressed herein are not necessarily those of the 

National Agricultural Statistics Service or of the United States Department of Agriculture. 

i

EXECUTIVE SUMMARY

The National Agricultural Statistics Service’s (NASS) primary purpose is to provide timely, 

accurate, and useful statistics on United States agriculture. The census of agriculture is 

conducted every five years to obtain agricultural statistics for the nation, every state, and every 

county or county equivalent. This report focuses on the 2010 Census of Agriculture Content 

Test which was conducted in preparation to the 2012 Census of Agriculture. Results from the 

2010 Census of Agriculture Content Test were analyzed to identify modifications to incorporate 

into the final design of the 2012 Census of Agriculture data collection tools and procedures.

The 2010 Census of Agriculture Content Test experimented with different correspondence 

methods to encourage respondents to return their report form via mail or Internet. These 

experiments utilized combinations of cover letters, postcards, and pre-recorded telephone 

messages for the initial and follow-up contacts to determine respondent correspondence 

preferences. 

The first experiment focused on ways to encourage respondents to report their data by the 

Internet rather than by mail. The second experiment tested the effectiveness of pre-survey 

recorded telephone message notification. The third experiment focused on the likely out of 

scope records for which the agricultural activity is unknown. Different follow-up methods were 

tested on the likely out of scope records to encourage respondents to return their report form. 

Finally, very similar to the third experiment, the fourth experiment tested the effectiveness of 

different follow-up methods on the probable farm sample. 

Overall, results showed that response rates were significantly different between the treatment and 

control groups. The groups receiving either the postcard or the pre-recorded telephone message

reminder stressing respondents may be personally visited or telephoned if they did not return 

their report form resulted in higher response rates than the groups receiving the standard 

message. Also, the groups receiving the postcard follow-up had higher response rates than the 

groups that received the pre-recorded telephone message follow-up. Generally, any form of 

courtesy reminders had some impact on improving response rates.

ii

RECOMMENDATIONS

1. Always include a paper report form in the initial mailing.

2. Use pre-survey recorded telephone messages.

3. Use the autodialer software for reminders, such as for subpopulations which reported 

online in previous surveys to report online for the current survey.

4. Use postcards or recorded telephone messages for pre-survey and follow-up notifications 

stressing they may receive a personal visit or telephone follow-up if they do not return 

the form.

5. Increase the number of attempts made when using recorded telephone messages.

1 Nancy J. Dickey and Zulma Riberas are Survey Methodologists; and HoaiNam N. Tran is a Mathematical 

Statistician with the National Agricultural Statistics Service, Research and Development Division, 3251 Old Lee 

Highway, Room 305, Fairfax, VA 22030. The authors would like to thank Orrin Musser, an Information 

Technology Specialist with NASS, for his assistance on this research project.

1

The Effect of Different Correspondence Methods on Response Rates in the 

2010 Census of Agriculture Content Test

Nancy J. Dickey, Zulma T. Riberas, HoaiNam N. Tran1

Abstract

The 2010 Census of Agriculture Content Test experimented with different 

correspondence methods to encourage respondents to return their report form 

via mail or Internet. These experiments utilized combinations of cover letters, 

postcards, and pre-recorded telephone messages for the initial and follow-up 

contacts to determine the correspondence preferences of respondents. 

The first experiment focused on ways to encourage respondents to report their 

data by the Internet rather than by mail. The second experiment tested the 

effectiveness of pre-survey pre-recorded telephone message notification. The 

third experiment focused on different follow-up methods to encourage likely 

out of scope respondents to return their report form. This likely out of scope 

group generally has low response rates as most do not qualify as agricultural 

operations, but they are still required to return the form indicating their status. 

The fourth experiment focused on probable farm operations with variations of 

follow-up correspondence to encourage them to return their report form. 

Key Words: cover letter, Web surveys, electronic data reporting (EDR), 

postcard, pre-recorded telephone message 
2

1. INTRODUCTION 

In an effort to increase response rates for the 2012 Census of Agriculture, the National 

Agricultural Statistics Service (NASS) conducted four different experiments during the 2010 

Census of Agriculture Content Test. These four experiments utilized different versions of cover 

letters, postcards, and pre-recorded telephone messages for the initial and follow-up mailings. 

The objective was to determine which communication strategy most effectively persuaded 

respondents to return their report forms. 

The first experiment focused on electronic data reporting (EDR). If respondents are not provided 

a paper report form and respondents are given detailed instructions on how to report their data 

via the Internet, then there may be an increased Web response rate. This experiment took place 

in November 2010 prior to the general December mail out of the report form. NASS tested the 

effectiveness of mailing a letter and a postcard without a questionnaire to two different groups of 

respondents to encourage reporting via the Internet. 

Research regarding mixed-mode designs is extensive but also conflicting. An experiment using 

the 2000 Department of Defense’s Information Services Survey, which targeted members of the 

armed forces and related populations, found a slight increase in response rates in the group given 

a choice of a Web option or a paper survey (Quigley, Riemer, Cruzen, & Rosen, 2000). Other 

research, such as the Yost & Homer’s (1998) study, suggests that adding a Web option does not 

change the total response rate, just shifts the mode by which respondents complete the survey. 

Response rates stayed about the same with the introduction of a new mode option, and almost 

one-third of respondents given the choice used the Web to complete their survey. Still other 

research provides evidence that offering a choice of mode actually decreases the overall response 

rate relative to a mail-only-version. For example, adding an Internet response option for the 

American Community Survey lowered overall response rates (Griffin, Fischer, & Morgan, 

2001). Griffin et al. found that providing respondents with a choice yielded lower mail response 

rates for the group invited to respond online.

The second experiment tested whether using a pre-survey recorded telephone message prior to 

the survey encouraged respondents to return their report form. All records selected for this 

sample had to have valid telephone numbers. 

Several research studies, such as Xu, Bates & Schweitzer (1993), McCarthy (2008), and 

McCarthy & Tran (2010), found evidence supporting claims that leaving messages on telephone 

answering machines can have a positive impact on survey participation by potential respondents.

The third experiment targeted likely out of scope respondents. The objective of this experiment 

was to see if using stronger wording would increase cooperation from this population. Likely 

out of scope respondents tend to have a low response rate as most do not qualify or feel they do 

not qualify as agricultural operations. They were still required to return their report form 

indicating their status. Non-response from this group increases follow-up contacts cost and may 
3

increase non-response bias. All four groups in this experiment received the standard mail out 

package with a cover letter and a printed copy of the 2010 Census of Agriculture Content Test 

report form. The difference between the groups was in the follow-up treatments. For the 

treatment groups, one group received a postcard and the other group received a recorded 

telephone message. The message for both these groups was the same stating that if they did not 

reply they may receive a personal visit or telephone call. They were asked to return the report 

form even if they were no longer farming or had never farmed. The control groups received 

either a postcard or a recorded telephone message with the standard message.

In a study conducted by Nichols, Jurgenson & Norris (2009), they pretested a series of deadline 

messages proposed for the 2010 Census Program for Evaluations and Experiments (CEPEX). 

This study concluded that non-respondents were motivated to return their report form when the 

message mentioned that an enumerator would personally visit them to conduct an interview and 

collect the information.

The fourth experiment was for probable farm operations, the majority of the 2010 Census of 

Agriculture Content Test sample. Similar to Experiment Three, the objective of this experiment 

was to see if using stronger wording would increase response rates from this population. All 

four groups in this experiment received the standard mail out package with the cover letter and a 

printed copy of the 2010 Census of Agriculture Content Test report form. The difference 

between the groups was in the follow-up treatments. One group received a postcard and another 

group received a recorded telephone message stating that if they did not reply they may be 

personally visited or would receive a telephone call. The control groups received either a 

postcard or a recorded telephone message with the standard message.

These experiments were analyzed using a combination of the chi-square test of homogeneity, the 

Mantel-Haenszel chi-square test and Fisher’s exact test. The chi-square test is a family of tests 

commonly used in the statistical analysis of experimental data (Bolboaca, Jantschi, Sestras,

Sestras, & Pamfil, 2011). It was used to determine whether frequency counts were identically 

distributed or proportions in each category were similar across all treatments. 

Let r be the number of treatments (columns) and c the number of response categories (rows). 

The chi-square degrees of freedom were calculated by (r-1)*(c-1). The null hypothesis states that 

the response rates in each category were similar across all groups. The alternative hypothesis 

states otherwise (i.e. there were at least two different response rates in one response category, 

across all groups). Mathematically: 

 H0: Pi,j = Pi,k

 Ha: Pi,j ≠ Pi,k, 

The null hypothesis was rejected if the p-value was less than alpha, α=0.05.

The Mantel-Haenszel chi-square statistic tests the alternative hypothesis that there is a linear 

association between the row and column variables. 
4

Fisher’s exact test is the probability of observing a table value that gives at least as much 

evidence of association as the one actually observed, given that the null hypothesis is true. 

Fisher’s exact test is more accurate than the chi-square test of independence when the expected 

numbers are small.

 

2. EXPERIMENT 1: ELECTRONIC DATA REPORTING

2.1 Method

This experiment took place in November 2010 prior to the general mail out of the report form

which occurred on December 30, 2010. This experiment consisted of three groups with a sample 

of 1,000 records in each group. Two of the groups were treatment groups and one was a control

group. The samples were selected from respondents of the 2007 Census of Agriculture who

reported having high speed Internet access. The assumption was those who reported having high 

speed Internet access would be more likely to report online. Treatment group 1 received an 

initial letter (L2 – shown in Appendix) by mail without a report form. This letter encouraged 

respondents to report online and gave detailed instructions on how to complete their 2010 Census 

of Agriculture Content Test report form on the NASS Internet site. The letter was printed on 

attractive paper with an agriculture related picture. 

Treatment group 2 received a folded postcard (L3 – shown in Appendix) by mail without a report 

form. The message on this postcard was succinct and encouraged respondents to visit the NASS 

Internet site to complete their 2010 Census of Agriculture Content Test report form. The folded 

postcard contained their personal 17 digit survey code needed to access their report form on the 

NASS Internet site. 

Group 3 was the control group which received the standard mail-out package containing a cover 

letter (L1 – shown in Appendix) and a printed copy of the 2010 Census of Agriculture Content 

Test report form. This package was mailed on December 30, 2010. The cover letter used was 

similar in wording and style to the one used for the 2005 Census of Agriculture Content Test and 

included instructions on how to access the NASS Internet site to complete the report form. 

During the week of January 12, 2011 the entire sample received a thank you/reminder postcard 

(L4 – shown in Appendix). This postcard encouraged respondents once again to visit the NASS 

Internet site to complete the 2010 Census of Agriculture Content Test report form. 

All three groups were mailed the same follow-up package which consisted of a cover letter (L14

– shown in Appendix) and a report form on February 17, 2011.

Table 1 outlines the EDR experiment. This table shows the sample size, and correspondence 

materials used for each group.
5

Table 1: EDR Experiment 

Treatment Group 1

Letter and No Printed 

Report Form

Treatment Group 2

Postcard and No Printed 

Report Form

Control Group 3

Letter and Printed 

Report Form

Sample 

Size 1,000 1,000 1,000

First 

Mailing

Letter L22 with no 

printed report form

Folded postcard L32 with 

no printed report form

Standard letter L12 with 

printed report form

Follow-up Thank you/reminder postcard L42

Second 

Mailing Letter L142 with printed report form

2 See Appendix for the text of the messages.

2.2 Results 

Table 2 shows response rates by groups. The chi-square results showed that there were 

statistically significant differences between the expected frequencies and the observed 

frequencies across the three groups at an alpha level of 0.05. The control group 3, which 

received the mailing package with the cover letter and the printed report form, achieved the 

highest overall response rate. The response rate for the control group was 15.7 percentage points 

higher than the response rate for treatment group 1, which received a letter and no printed report 

form; and 14.5 percentage points higher than the response rate for treatment group 2, which 

received a postcard and no printed report form.

Table 2: EDR Experiment -- Response Rates 

Treatment Group 1

Letter and No 

Printed Report Form

Treatment Group 2

Postcard and No 

Printed Report Form

Control Group 3

Letter and Printed 

Report Form

No. % No. % No. %

Returned Records 364 36.4 376 37.6 521 52.1

Out of Scope (OS) 98 9.8 101 10.1 83 8.3

Non-response 538 53.8 522 52.2 395 39.5

Refusals 0 0.0 1 0.1 1 0.1

Total 1,000 100.0 1,000 100.0 1,000 100.0

χ

2 (4, N=3,000) = 63.64, p<.0001 (Refusals were eliminated for the analysis.)

Mantel-Haenszel χ

2 (1, N=3,000) = 19.1023, p<.0001

Fisher: p=7.201E-20

Table 3 shows response rates by data collection mode: EDR, paper, and Computer Assisted 

Telephone Interviews (CATI). The two treatment groups had slightly higher response rates for 

EDR and CATI than the control group. However, the control group, which received the printed 
6

report form, had the highest paper return rate approximately 20 percentage points higher than the 

two treatment groups.

Table 3: EDR Experiment -- Response Rates by Data Collection Mode

Response Code Treatment Group 1

Letter and No Printed 

Report Form

Treatment Group 2

Postcard and No

Printed Report Form

Control Group 3

Letter and Printed 

Report Form

No. % No. % No. %

EDR Returns 121 12.1 117 11.7 93 9.3

Paper Returns 229 22.9 244 24.4 428 42.8

CATI Returns 70 7.0 66 6.6 38 3.8

Other OS 42 4.2 50 5.0 45 4.5

Non-response 538 53.8 522 52.2 395 39.5

Refusal 0 0.0 1 0.1 1 0.1

Total 1,000 100.0 1,000 100.0 1,000 100.0

This experiment demonstrates the importance of including a paper report form in the initial 

mailing to the entire sample.

3. EXPERIMENT 2: PRE-SURVEY RECORDED TELEPHONE MESSAGE

3.1 Method:

There were two groups in this experiment each with 1,000 records. Treatment group 4 received 

a pre-survey recorded telephone message (L5 – shown in Appendix) during the week of 

December 27, 2010. This pre-survey recorded message informed respondents that they would 

receive the 2010 Census of Agriculture Content Test report form in the mail shortly and 

encouraged them to complete and return the form promptly. 

Group 5 was the control group for this experiment. They did not receive a pre-survey recorded 

telephone message. They received the standard mail out package with the cover letter (L1 –

shown in Appendix) and a printed copy of the 2010 Census of Agriculture Content Test report 

form. 

Two weeks later, both the treatment and the control group received the standard follow-up 

materials consisting of a thank you/reminder postcard (L6 – shown in Appendix) followed by a 

second mailing of a letter (L14 – shown in Appendix) and a report form.

Table 4 outlines the pre-survey recorded telephone message experiment.
7

Table 4: Pre-survey Recorded Telephone Message Experiment

Treatment Group 4

Pre-survey Recorded 

Telephone Message

Control Group 5 

No Pre-survey Recorded 

Telephone Message

Sample Size 1,000 1,000

Pre-notification Pre-survey recorded 

telephone message

No pre-survey recorded 

telephone message

Initial Mailing Letter L12 with printed report form

Follow-up Thank you/reminder postcard L62

Second Mailing Letter L142 with printed report form

2 See Appendix for the text of the messages.

3.2 Number of Automated Telephone Call Attempts

For the pre-survey recorded automated message, if the message was delivered to either a person 

or to an answering machine, the call was coded as complete. If the call had a busy signal, fax, or 

modem, or any other outcome such as the outgoing message is too long, up to three attempts 

were made to leave a message. After three automated attempts, if the message was not delivered 

to either a person or an answering machine, no more attempts were made.

3.3 Results 

The outcome of the recorded automated message attempts was analyzed. Respondents with 

invalid telephone numbers were excluded from all comparisons. As shown in Table 5, 71 

percent of the calls were delivered to either a person or an answering machine, of which 46 

percent were answered by a person who listened to the entire message. Around 11 percent did 

not receive the recorded automated telephone reminder message. 

 
8

Table 5: Pre-Survey Recorded Telephone Message Experiment -- Outcome

Outcome

Pre-survey Recorded 

Telephone Message (Group 4)

No. %

No Answer (maximum attempt 3 calls) 107 10.7

Answered by a Person 455 45.5

Answered by an Answering Machine 258 25.8

Hang Up/Partial Message Left 0 0.0

Busy 20 2.2

Not Called 42 4.2

Other 3 118 11.8

Total 1,000 100.0

3 No connection, outgoing message (OGM) too long, Fax or Modem, Telephone Co. message.

Table 6 shows the response rates from Experiment 2 by groups. Differences between groups 

were statistically significant at the .05 level. The response rate for treatment group 4, the group 

which received the pre-survey recorded telephone message, was 7.7 percentage points higher 

than the response rate for control group 5, the group which did not receive the pre-survey 

recorded telephone message. Using a pre-survey recorded telephone message helped increase 

response rates.

Table 6: Pre-survey Recorded Telephone Message Experiment -- Response Rates

Response

Treatment Group 4

Pre-survey Recorded 

Telephone Message

Control Group 5 

No Pre-survey Recorded 

Telephone Message

No. % No. %

Response 510 51.0 433 43.3

Out of Scope 96 9.6 108 10.8

Non-response 393 39.3 459 45.9

Refusal 1 0.1 0 0.0

Total 1,000 100.0 1,000 100.0

 χ

2 (2, N=2,000) = 12.11, p=.0024 (Refusals were eliminated for the analysis)

 Mantel-Haenszel: χ

2 (1, N=2,000) = 3.4041, p=.0650

 Fisher: p=5.229E-6

4. EXPERIMENT 3: LIKELY OUT OF SCOPE POPULATION

4.1 Method

This experiment focused on the likely out of scope population. This population consists of 

records for which agricultural activity is unknown. This group typically shows a low response 
9

rate as most do not qualify or feel they do not qualify as agricultural operations. They were still 

required to return the report form. 

There were four groups in this experiment each consisting of 500 records. All four groups 

received the standard mail-out package with the cover letter (L1 – shown in Appendix) and a 

printed copy of the 2010 Census of Agriculture Content Test report form. The experiment 

focused on each group having different follow-up treatments.

Treatment group 6 received a thank you/reminder postcard (L7 – shown in Appendix) with 

wording targeting likely out of scope respondents. It instructed respondents to write on their 

report form if they were no longer farming and return it. Later they received a second mailing 

consisting of a letter with screening questions (L13 – shown in Appendix), a cover letter (L14 –

shown in Appendix), and a report form.

Treatment group 7 received a recorded telephone message (L8 – shown in Appendix) instructing 

respondents to specify that they were no longer farming on their report form and return it. They 

also received a second mailing consisting of a letter with screening questions (L13 – shown in 

Appendix), a cover letter (L14 – shown in Appendix), and a report form.

Control group 8 received the standard follow-up materials by mail which consisted of a thank 

you/reminder postcard (L6 – shown in Appendix) followed by a second mailing of a letter (L14 –

shown in Appendix) and a report form. 

Control group 9 received a recorded telephone message (L9 – shown in Appendix) similar in 

content to the standard thank you/reminder postcard (L6 – shown in Appendix) followed by a 

second mailing of a letter (L14 – shown in Appendix) and a report form. Table 7 outlines the 

likely out of scope experiment.
10

Table 7: Likely Out of Scope Population Experiment

Treatment Treatment 

Group 6

Follow-up 

Postcard 

Modified 

Message

Treatment 

Group 7

Follow-up 

Recorded

Telephone 

Modified Message

Control 

Group 8 

Follow-up 

Postcard 

Standard 

Message

Control Group 9 

Follow-up 

Recorded 

Telephone 

Standard 

Message

Sample Size 500 500 500 500

Initial Contact Letter L12 and Report form

Follow-up Postcard

Modified 

Message (L7) 2

Automated 

Modified Message

(L8) 2

Postcard

Standard 

Message (L6)2

Autodial Standard 

Message (L9) 2

Second Mailing Letter with screening questions (L13) 2

Letter (L14) 2 and Report form

Letter (L14) 2 and Report form

2 See Appendix for the text of the messages.

4.2 Results 

Table 8 shows for both autodial groups about 40 percent of the sample received the message with 

either the call answered by a person or by an answering machine. Also, for both groups, about 

50 percent of the sample had three attempts made but no answer was achieved.

Table 8: Likely Out of Scope Population Experiment -- Outcome

Outcome Pre-recorded Telephone Message Reminder 

Calls

Group 7 Group 9

No. % No. %

No Answer (maximum attempt 3 calls) 268 53.6 245 49.0

Answered by a Person 88 17.6 70 14.0

Answered by an Answering Machine 116 23.2 120 24.0

Hang Up/Partial Message Left 2 0.4 1 0.2

Busy 2 0.4 46 9.2

Not Called 3 0.6 3 0.6

Other4 21 4.2 15 3.0

Total 500 100.0 500 100.0

4 No connection, outgoing message (OGM) too long, Fax or Modem, Telephone Co. message.

Table 9 displays the response by group of the likely out of scope experiment. Response rates 

were very similar across all four groups. Group 6, which received the postcard with the modified 

message, had the highest response rate at 11.2 percent and also the highest out of scope rate at 

11.4 percent. Group 8, which received the postcard with the standard message, had the lowest 

out of scope returns at 6.4 percent. In addition, it was also clear from this experiment that this 
11

group did indeed have very low overall response rates (about half of the response rate for groups 

included in the other experiments.)

Table 9: Likely Out of Scope Population Experiment -- Response Rates

Response 

Code

Treatment 

Group 6

Follow-up 

Postcard 

Modified 

Message

Treatment 

Group 7

Follow-up 

Recorded 

Telephone 

Modified 

Message

Control

Group 8

Follow-up 

Postcard 

Standard 

Message

Control

Group 9

Follow-up 

Recorded 

Telephone 

Standard 

Message

No. % No. % No. % No. %

Responses 56 11.2 54 10.8 50 10.0 46 9.2

Out of Scope 57 11.4 43 8.6 32 6.4 51 10.2

Non-response 386 77.2 403 80.6 418 83.6 403 80.6

Refusals 1 0.2 0 0.0 0 0.0 0 0.0

Total 500 100.0 500 100.0 500 100.0 500 100.0

Table 10 summarizes the response rates in a two by two factorial design, testing the model

(No.=a+b*Mode+c*Message+d*Mode*Message), to see whether the mode (postcard, autodial), 

and message (modified or standard message), and/or their interaction had an impact on response 

rates with the logistic procedure. Binary (response or non-response) are required for the logistic 

model. The “Responses” and “Out of Scope” categories were combined into Response. 

Similarly, “Non-response” and “Refusals” were combined into “Non-response”.

Table 10: Likely Out of Scope Population Experiment -- Responses by Mode and Message

Mode Response

Modified Message Standard Message Mode Main Effect

No. % No. % No. %

Response 113 22.6 82 16.4 195 19.5

Postcard Non-response 387 77.4 418 83.6 805 80.4

Response 97 19.4 97 19.4 194 19.4

Autodial Non-response 403 80.6 403 80.6 806 80.6

Message 

Main 

Effect

Response

Non-response

210

790

21.0

78.9

179

821

17.9

82.1

Effect P-value

Mode 0.9605

Message 0.0800

Mode*Message 0.0800

The result showed that the mode, message, and their interaction were not statistically significant 

at the alpha level of 0.05. However, the message and the interaction effects’ p-values are both 

0.08, suggesting the postcard with the modified message stressing personal visit or telephone call 

tends to increase response rates.
12

5. PROBABLE FARMS REMINDER MESSAGE EXPERIMENT

5.1 Method

The remaining 25,000 records from the content sample were split into four groups of 6,250 

records each. This sample consists of probable farms from the census mail list. All four groups 

received the standard mail out package with the cover letter (L1) and a printed copy of the 2010 

Census of Agriculture Content Test report form. Each group had a different follow-up treatment.

Treatment group 10 received a postcard (L11) with a message stressing that if the form was not 

returned by the due date they might be personally visited or telephoned by an enumerator. This 

was followed by a second mailing of a cover letter (L14) and a report form.

Treatment group 11 received a recorded telephone message (L10) stressing that if the form was 

not returned by the due date they might be personally visited or telephoned by an enumerator. 

This was followed by a second mailing of a cover letter (L14) and a report form. Control group 

12 received the standard follow-up materials by mail which consisted of a thank you/reminder 

postcard (L6) followed by a second mailing of a letter (L14) and a report form.

Control group 13 received a recorded telephone message (L9) similar in content to the control 

thank you/reminder postcard (L6). Later they received a second mailing of a letter (L14) and a 

report form.

Table 11 summarizes the probable farm reminder message experiment.

Table 11: Probable Farms Reminder Message Experiment

Treatment

Group 10

Follow-up 

Postcard 

Modified

Message

Treatment

Group 11

Follow-up 

Recorded 

Telephone 

Modified Message

Control

Group 12

Postcard 

Standard 

Message

Control

Group 13

Recorded 

Telephone 

Standard Message

Sample 

Size 6,250 6,250 6,250 6,250

Initial 

Contact

Letter (L1) 2

Report form

Followup

Postcard Modified 

Message (L11) 2

Automated Modified 

Message (L10) 2

Postcard Standard 

Message (L6) 2

Automated Standard 

Message (L9) 2

Second 

Mailing Letter (L14) 2 and Report form

2 See Appendix for the text of the messages.
13

5.2 Results 

Table 12 displays the outcome of the recorded telephone message attempts for treatment group 

11 and control group 13. Approximately 65 percent of the sample for both groups received the 

message either by person or on an answering machine. About 14 percent of the sample in each 

group had three attempts made but no answer was achieved.

Table 12: Probable Farms Reminder Message Experiment -- Outcome of Recorded 

Telephone Message Attempts

Outcome Recorded Telephone Message Reminder Calls

Treatment Group 11 Control Group 13

No. % No. %

No Answer (maximum attempt 3 calls) 870 13.9 853 13.7

Answered by a Person 2,323 37.2 2,324 37.2

Answered by an Answering Machine 1,801 28.8 1,916 30.6

Hang Up/Partial Message Left 9 0.2 10 0.2

Busy 277 4.4 221 3.5

Not Called 315 5.0 296 4.7

Other 5 655 10.5 630 10.1

Total 6,250 100.0 6,250 100.0

5 No connection, out-going message (OGM) too long, Fax or Modem, Telephone Co. message.

Table 13 shows response rates by treatment group. Treatment groups received the message 

either by postcard or recorded telephone message stressing personal visit or phone follow-up 

resulted in higher response rates than the control groups that received the standard message. 

Also, for the treatment groups, the postcard follow-up resulted in slightly higher response rates 

than the recorded telephone message follow-up.

Table 13: Probable Farms Reminder Message Experiment -- Response Rates 

Group 10

Follow-up 

Postcard 

Modified

Message

Group 11

Follow-up 

Recorded 

Telephone 

Modified 

Message

Control

Group 12

Postcard 

Standard 

Message

Control

Group 13

Recorded 

Telephone 

Standard 

Message

No. % No. % No. % No. %

Responses 3,111 49.8 3,034 48.6 2,992 47.9 2,991 47.9

Out of Scope 623 10.0 615 9.8 599 9.6 551 8.8

Non-response 2,515 40.2 2,600 41.6 2,656 42.5 2,706 43.3

Refusals 1 0.0 1 0.0 3 0.0 1 0.0

Total 6,250 100.0 6,250 100.0 6,250 100.0 6,249 100.0
14

Table 14 summarizes the response rates in a two by two factorial design, testing the model 

(No.=a+b*Mode+c*Message+d*Mode*Message) to see whether the mode (postcard, autodial), 

and message (modified or standard message), and/or their interaction had an impact on response 

rates with the logistic procedure. Binary (response or non-response) were required for the

logistic model. Therefore, “Responses” and “Out of Scope” categories were combined into 

Response. Similarly, “Non-response” and “Refusals” were combined into “Non-response”.

The result showed the message was statistically significant with a p-value equal to 0.0014. The 

mode and the interaction of mode and message were not statistically significant. 

Table 14: Probable Farms Reminder Message Experiment -- Response Rates

Mode Response

Message Mode Main 

Modified Message Standard Message Effect

No. % No. % No. %

Postcard Response 3,734 59.8 3,591 57.5 7,325 58.6

Non-response 2,516 40.3 2,659 42.5 5,175 41.4

Autodial Response 3,649 58.4 3,542 56.7 7,191 57.5

Non-response 2,601 41.6 2,707 43.3 5,308 42.4

Message

Main 

Effect

Response

Non-response

7,383

5,117

59.1

40.1

7,133

5,366

57.1

42.9

Effect P-value

Mode 0.0866

Message 0.0014

Mode*Message 0.6311

6. CONCLUSION 

Results from the EDR experiment strongly suggest that including a paper report form with the 

initial mailing would help to increase the likelihood of response. Although, the groups that did 

not receive a paper report form had higher EDR response rates, their overall response rates were

much lower than the group receiving the paper form. These findings were consistent with the 

study conducted by Quigley, Riemer, Cruzen, & Rosen (2000). This study found a slight 

increase in response rates in the group given a choice of a Web option or a paper survey.

All the experiments showed response rates increased by adding automated telephone 

notification/reminders. These findings were consistent with previous research studies such as Xu, 

Bates & Schweitzer (1993), McCarthy (2008), and McCarthy & Tran (2010), which found that 

leaving messages on telephone answering machines can have a positive impact on survey 

participation. The experiment suggested it may be beneficial to increase the number of autodial 

attempts as the no answer category ranges from about 11 to 54 percent. Since the price per call 

is minimal, response rates might be further increased if more call attempts were made with 

automated messages, and consequently more messages were delivered. Studies conducted by 
15

McCarthy (2007, 2008) suggest that five automated telephone attempts may increase the number 

of messages received by sample units.

In addition, using the modified message stressing that non-response would result in telephone or 

in person contacts helped to increase response rates. The groups receiving the postcard followup had higher response rates than the groups that received the recorded telephone message

follow-up. 

Generally, any form of courtesy reminders had some impact on improving response rates. 

However, not providing a paper report form and providing only instructions for completing the 

form on-line resulted in significantly lower overall response rates.

Further studies are needed utilizing combinations of postcards, letters, and autodial messages. 

Presently, a research study is being conducted by NASS on the National Agricultural 

Classification Survey (NACS). The NACS survey is conducted in preparation of the Census of 

Agriculture to determine whether or not the records should be included on the Census mailing 

list. The current study utilizes a standard cover letter and a modified cover letter with stronger 

wording, stating that if the respondent does not reply, they may be personally visited or receive a 

telephone call. 

Federal mandates seek to minimize the paperwork burden on individuals and businesses. The 

Government Paperwork Elimination Act of 1998 requires Federal agencies to allow individuals 

or entities the option to submit information electronically. NASS should continue to investigate 

contact strategies that will not only encourage response to surveys, but encourage respondents to 

use the Web as the preferred response mode. 

7. RECOMMENDATIONS 

1. Always include a paper report form in the initial mailing.

2. Use pre-survey recorded telephone messages.

3. Use the autodialer software for reminders, such as for subpopulations which reported 

online in previous surveys to report online for the current survey.

4. Use postcards or recorded telephone messages for pre-survey and follow-up notifications 

stressing they may receive a personal visit or telephone follow-up if they do not return 

the form.

5. Increase the number of attempts made when using recorded telephone messages.
16

8. REFERENCES

Bolboaca S., Jantschi L., Sestras A., Sestras R., and Pamfil D. (2011). Pearson-Fisher ChiSquare Statistic Revisited. Basel, Switzerland: MDPI.

Griffin, D., Fischer, D., and Morgan, M. (2001). Testing an Internet Response Option for the 

American Community Survey. Presented at the Annual Conference of The American 

Association for Public Opinion Research. Retrieved at

http://www.census.gov/acs/www/Downloads/library/2001/2001_Griffin_01.pff

McCarthy, J. (2007). Pre-recorded Telephone Messages As An Alternative To Follow-Up 

Survey Postcards: The Effectiveness Of Autodialer Software Used During The Pennsylvania 

County Estimates Survey. Research and Development Division Report #RDD-07-04, U.S. 

Department of Agriculture: National Agricultural Statistics Service.

McCarthy, J. (2008). Pre-recorded Telephone Messages As An Alternative To Follow-Up 

Survey Postcards II: The Effectiveness Of Automated Telephone Reminder Messages Used 

During The 2007 Pennsylvania County Estimates Survey. Research and Development Division 

Report #RDD-08-04, U.S. Department of Agriculture: National Agricultural Statistics Service.

McCarthy, J. and Tran, H. (2010). Pre-recorded Telephone Messages As An Alternative To 

Follow-Up Survey Postcards III: The Effectiveness Of Automated Telephone Reminder 

Messages Used During The 2007 Census of Agriculture. Research and Development Division, 

U.S. Department of Agriculture: National Agricultural Statistics Service.

Nichols, E., Jurgenson, N., & Norris, D. (2009). Report on Cognitive Pretesting of the 2010 

Census Program for Evaluations and Experiments Panels Mailing Package: Deadline Messages. 

Statistical Research Division, U.S. Census Bureau.

Quigley, B., Riemer, R.A., Cruzen, D.E., and S. Rosen. (2000). Internet Versus Paper Survey 

Administration: Preliminary Findings and Response Rates. Presented at the 42nd Annual 

Conference of the International Military Testing Association, Edinburgh, Scotland.

Xu, M., Bates, B., and Schweitzer, J. (1993). The Impact of Messages on Survey Participation in 

Answering Machine Households. Public Opinion Quarterly 57(2): 232-237. 

Yost, P.R., and Homer, L.E. (1998). Electronic Versus Paper Surveys: Does the Medium Affect 

the Response? Presented at the 13th Annual Conference of the Society for Industrial and 

Organizational Psychology.
17

APPENDIX : COVER LETTERS, POSTCARDS, AND PRE-RECORDED TELEPHONE 

MESSAGES
18

L1
19

L2
20

L3
21

L3
22

L4
23

L4
24

L5

Pre-notification Pre-recorded telephone message

Hello, this is the United States Department of Agriculture’s National Agricultural Statistics 

Service. In a few days you will be receiving a 2010 Census of Agriculture Test report form. 

This is a critical trial run before the 2012 Census of Agriculture. Please, provide your 

information promptly by Internet or mail. The information you provide is confidential and 

protected by law. 

If you have questions or require assistance please visit www.agcensus.nass.usda.gov or call tollfree, 1-888-424-7828. Thank you for your cooperation.
25

L6
26

L6
27

L7
28

L7
29

L8

Hello, this is the United States Department of Agriculture National Agriculture Statistics Service, 

reminding you to return your 2010 Census of Agriculture Test report form. THANK YOU...if 

you already returned your report form. If not, February 4, 2011 is the due date.

You can report online at www.agcounts.usda.gov using the 17-digit survey code on the label of 

your report form. If you do not reply, you may be personally visited or receive a telephone call 

from one of our enumerators. 

Even if you think this form does not apply to you because you have never farmed, are no longer 

farming, are a small operation, etc.: You SHOULD REPLY so we can update our records. The 

information you provide is confidential and protected by law. If you have questions please call 

toll-free, 1-888-424-7828. Thank you for your cooperation.
30

L9

Pre-recorded telephone message

Hello, this is the United States Department of Agriculture’s National Agricultural Statistics 

Service reminding you to return your 2010 Census of Agriculture Test report form. THANK 

YOU...if you already returned your report form.

If you have not returned your report form, please take a few minutes to complete and return it by 

February 4, 2011. Your prompt reply will help determine the best questions and design for the 

2012 Census of Agriculture. The information you provide is confidential and protected by law. 

If you have questions please call toll-free, 1-888-424-7828. Thank you for your cooperation.
31

L10

Pre-recorded telephone message

Hello this is the United States Department of Agriculture’s National Agricultural Statistics 

Service reminding you of the due date for returning your 2010 Census of Agriculture Test report 

form. REPLY via mail or online by February 4, 2011 to avoid a personal or telephone followup contact. The information you provide is confidential and protected by law. If you have 

questions please call toll-free, 1-888-424-7828. Thank you for your cooperation. 
32

L11
33

L11
34

L12
35

L13
36

L14