The Effectiveness of Automated 

Reminder Messages Used on the 2007 

Census of Agriculture and on the 

National Animal Health Monitoring 

System 2011 Small Producer Study

HoaiNam N. Tran

Michael W. Gerling

Jaki S. McCarthy

Terry P. O’Connor

United States 

Department of

Agriculture

National

Agricultural

Statistics

Service

Research and

Development Division

Washington DC 20250

RDD Research Report

Number RDD-11-08

May 2012

This report has been prepared for limited distribution to the research community outside the U.S. Department of 

Agriculture (USDA). The views expressed herein are not necessarily those of NASS or USDA.

i

EXECUTIVE SUMMARY

This report researches the impact of utilizing automated telephone reminders to increase 

response rates of the 2007 Census of Agriculture and the National Animal Health 

Monitoring System (NAHMS) 2011 Small Producer Study. 

The first study was comprised of three experiments and was conducted on the 2007 

Census of Agriculture. The target population for the experiments consisted of records for 

which agricultural activity was unknown. These records are historically the toughest 

group to contact and obtain a completed report. The standard data collection process 

consisted of an initial mail out form, followed by a thank you/reminder postcard and, if 

necessary, a first and second follow-up mail out form. 

For the first experiment, the control group received the standard data collection process. 

The treatment group received an automated telephone reminder after the first follow-up 

form in addition to the standard data collection method.

For the second experiment, the control group received the standard data collection 

process. The treatment group received an automated telephone reminder after the first 

follow-up form and an extra postcard reminder after the second follow-up form in 

addition to the standard data collection method.

For the third experiment, in addition to the standard data collection method, the control 

group received an automated telephone reminder after the first follow-up form. The 

treatment group received an automated telephone reminder after the first follow-up form 

and a postcard reminder after the second follow-up form, in addition to the standard data 

collection method.

The second study consisted of a sample of known agricultural operations from the 

NAHMS 2011 Small Producer Study. The NAHMS standard data collection process 

consisted of a pre-survey package, a mail out form, and, if necessary, a telephone followup. The control group for this study received the NAHMS standard data collection 

package. The two treatment groups received automated telephone reminders after the 

initial mail out but before the telephone follow up. These telephone reminder messages

varied in their wording.

Results from these studies showed that using automated telephone reminders significantly 

increased response rates for the 2007 Census of Agriculture but only showed a slight 

increase in response rates for NAHMS. Mailing extra postcard reminders to operators 

who had already received the automated telephone reminders did not have a significant 

effect on the overall response rates.

ii

RECOMMENDATIONS

1. Continue to use automated telephone reminders and conduct additional testing to 

determine the most effective use of reminder messages. This includes how to use 

automated reminders most effectively within standard NASS data collection 

procedures, trained or professional speakers, or a NASS data user to record the 

message. In addition, different content should also be tested; for example 

stressing that if respondents do not respond they will receive additional contacts.

2. Investigate additional uses of the automated telephone reminders for pre-survey 

notification messages.

3. Increase the number of call attempts to five when using automated telephone 

reminders.

1

The Effectiveness of Reminder Messages Used on the 

2007 Census of Agriculture and National Animal Health 

Monitoring System 2011 Small Producer Study

HoaiNam N. Tran, Michael W. Gerling,

Jaki S. McCarthy, and Terry O’Connor1

Abstract

The National Agricultural Statistics Service (NASS) researches the use

of automated telephone reminder software to increase survey response 

rates. The first study consisted of three experiments conducted on a 

sample of the 2007 Census of Agriculture. The first experiment 

evaluated the effectiveness of using automated telephone reminders as 

an addition to the Census of Agriculture’s standard data collection 

procedures. The second experiment tested whether sending a second 

postcard reminder to respondents who already received automated 

telephone and standard postcard reminders increased response rates. 

The third experiment identified whether sending extra postcard 

reminders to respondents who received automated telephone reminders 

increased response rates. The second study was conducted on the 

NAHMS 2011 Small Producer Study, tested whether using different 

automated telephone reminder messages improve response rates. 

Overall, the use of automated telephone reminder software is a 

promising addition to the tools NASS can use to increase survey 

response rates. However, further examination is needed to determine 

the most efficient and effective use of this technology.

Key Words: automated message, follow-up reminder, postcard 

reminder, response rate.

1 HoaiNam Tran and Michael W. Gerling are Mathematical Statisticians, Jaki McCarthy is the Branch 

Chief and Terry O’Connor is the Deputy Chief of the Survey Methodology and Technology Branch with 

the National Agricultural Statistics Service (NASS) - Research & Development Division, located at Room 

305, 3251 Old Lee Highway, Fairfax, VA 22030. A special thanks to Orrin Musser and Peter Quan of 

NASS for their dedication and commitment in making this study a success.
2

I. INTRODUCTION AND BACKGROUND

Postcards mailed as survey follow-up reminders have been used in the past to increase 

response rates and are a generally accepted best practice in survey data collection. Today, 

voice messages left with respondents via the telephone are also being used. Automated 

telephone reminder software can be used to schedule and call lists of phone numbers and 

leave messages, either with a person or an answering machine. This technology has been 

used for many types of telephone messages, such as medical appointment reminders, 

broadcasts by groups to their membership, and emergency notifications. Research in 

some of these situations has shown that these reminders are effective and are received 

positively by their recipients (Dini, Linkins, Sigafoos, 2000; Franzini, Rosenthal, Spears, 

Martin, Balderas, Brown, Milne, Drutz, Evans, Kozinetz, Oettgen and Hanson, 2000; 

Krishna, Balas, Boren and Maglaveras, 2002). 

Overall, there is a limited amount of documentation on the use of automated telephone 

reminder messages in survey research; however, automated telephone reminders do 

appear to increase response rates. The U.S. Census Bureau found that response rates were 

higher for households who had been left a reminder message than for those which 

received either no reminder or a postcard reminder (Bouffard, Brady, Stapleton and Imel, 

2003). A test by the National Science Foundation also showed that telephone reminder 

messages prompted slightly higher survey response rates (Fecso, Ferraiuolo, and 

Finamore, 2006). Earlier tests by the National Agricultural Statistics Service (NASS) 

Pennsylvania Field Office showed that groups receiving automated telephone reminders 

resulted in higher response rates compared to groups receiving a postcard or no reminders

(McCarthy, 2007 and 2008). 

Mailing postcards is relatively inexpensive (24.05 cents each). Automated telephone 

reminder software can be even cheaper depending on the size of the sample (7 cents per 

completed call). The greater the sample size and/or the number of times used make this 

technology a cost effective solution. Also, telephone message can be easily modified and 

are not affected by mail delays.

II. RULES AND DEFINITIONS

1. Rules of Using Automated Software

The automated telephone reminder software was programmed to begin at 8:00 a.m. and 

end at 9:00 p.m., calling up to three times for unanswered numbers. The software was set 

to hang-up after six rings or if it reached an automated answering machine with an 

outgoing message longer than 25 seconds (these were assumed to be problem numbers or 

message machines unable to take recordings). 
3

2. Rules of Automated Number of Attempts

The call was considered complete if:

a. The message was received by either a person or an answering machine.

b. The call had a busy signal or any other outcome such as outgoing message too 

long, fax or modem, or any other telephone company message, or

c. Up to three attempts were made to leave a message. 

3. Definitions

Farm: NASS defines a farm as any agricultural operation with any combination of sales, 

potential sales, and government payments totaling at least $1,000.

Criteria Record: A record that has potential to be considered a farm by NASS’ definition.

In Scope (IS): An operating arrangement on which there were or could be sales of at least 

$1,000 worth of crops, livestock, poultry, or other agricultural products during the 

year. This includes Conservation Reserve Program only operations that received 

government payments of at least $1,000.

Out of Scope (OS) include the following situations:

a. An agricultural operation that is now out of business.

b. An operation that was never involved in agriculture

c. An agricultural operation having less than $1,000 in agricultural sales.

4. Chi Square Test of Homogeneity

The Chi Square test is a family of tests and is commonly used in statistical analyses to 

answer questions about association between categorical variables (Bolboaca, Jantschi,

Sestra, Sestras, and Pamfil, 2011). The Chi Square Test of Homogeneity was used in the 

analyses throughout this paper, to determine whether frequency counts were identically 

distributed or proportions in each category were similar across all treatments.

Let r be the number of treatments (columns) and c the number of response categories 

(rows). The Chi Square degree of freedom was calculated by (r-1)*(c-1). The null 

hypothesis states that the response rates in each category were similar across all 

groups. The alternative hypothesis states otherwise (i.e. there were at least two different 

response rates in one response category across all groups). Mathematically: 

H0: Pi,j = Pi,k

Ha: Pi,j ≠ Pi,k, 

The null hypothesis was rejected if the p-value was less than an alpha of 0.05.
4

III. EXPERIMENTS

This report describes the methods used and the results of the four automated telephone 

reminder experiments conducted during the 2007 Census of Agriculture and the NAHMS 

2011 Small Producer Study.

1. 2007 CENSUS OF AGRICULTURE EXPERIMENTS

The 2007 Census of Agriculture enumerated all known potential agricultural operations. 

The primary mode of data collection was a self-administered mail form, which collected

information on crop and livestock inventory and production, land use, agricultural 

economics, and demographics of agricultural operators. The Census of Agriculture 

standard data collection process consisted of an initial mail out form. This was followed 

by a thank you/reminder postcard, and, if necessary, first and second follow-up mail out 

forms.

 

Response to the 2007 Census of Agriculture, unlike most NASS surveys, was mandatory 

and required by law. Report forms were mailed to known and questionable agricultural 

operations on the NASS’ list frame. This listing was compiled from many sources, which 

contained a mix of agricultural and non-agricultural records. Some of these sources had

minimal information, perhaps only a name and address or phone number. Such

questionable records were first screened for agricultural activity before the Census of 

Agriculture by a separate survey, the Agricultural Identification Survey (AIS). 

The target population for this study was operations who did not respond to the AIS prior 

to the census or who were added to the mail list too late to be included in the AIS from 

nine states (California, Colorado, Florida, Georgia, Illinois, Louisiana, Nebraska, South 

Dakota, and Washington). For both of these groups, their agricultural activity was 

unknown. In addition, historical response rates for this particular group have been 

substantially lower than other groups. 

This target population was chosen for additional reminders (automated telephone or extra 

postcard reminders, or both) since it was hypothesized that many of them might have 

thought that the form did not apply to them and therefore did not realize they had to 

return it. Only records having telephone numbers were chosen for automated telephone 

reminders. Both automated and extra postcard reminders stated that returning the form 

was required by law, even if they were not farming, and that failing to return the form 

would result in additional follow-up contacts. 

The 2007 Census of Agriculture standard data collection process consisted of an initial 

mail out form. This was followed by a thank you/reminder postcard, and, if necessary,

first and second follow-up mail out forms. 

The first experiment was conducted on sampled records from Illinois, Louisiana, South 

Dakota, and Washington. In this study, the control group received the standard data 
5

collection method. The treatment group received an automated telephone reminder after 

the first but before the second mailing of follow-up forms in addition to the standard data 

collection method.

The second experiment was conducted on sampled records from California and Georgia. 

The control group received the same standard data collection package as the control 

group in the first experiment. The treatment group received an automated telephone 

reminder after the first but before the second mailing of follow-up forms. The treatment 

group also received an extra postcard reminder after the second follow-up form in 

addition to the standard data collection method.

The third experiment was conducted on sampled records from Colorado, Florida, and 

Nebraska. The control group received the standard data collection package and an 

automated telephone reminder after the first but before the second mailing of follow-up 

forms. In addition to the standard data collection method, the treatment group also 

received an automated telephone reminder and an extra postcard reminder after the 

second mailing of follow-up form. 

Since U.S. Postal Service standard mail was used to send out the forms, exactly how long 

it would take the forms to be delivered was indeterminable. Therefore, a delivery time 

of one to three weeks was estimated. To be sure the reminders were received after the 

forms, reminders did not begin until several weeks after the first mailing.

The National Processing Center in Jeffersonville, Indiana, conducted the automated 

telephone reminder calls. Reminders were scheduled to begin 5 days after a second 

mailing of the Census of Agriculture form. The messages were written with the most 

critical information first, so in the event of a recipient hanging up early, they would hear 

the purpose of the call. Due to the volume of calls, the maximum number of attempts was 

three. Also, records with invalid telephone numbers were excluded from the comparisons.

NASS’ North Carolina Field Office mailed the extra postcard reminders to selected 

records two weeks after the third mailing of the report form. 

1.1 Census of Agriculture - Experiment 1: Automated Telephone Reminders

(Illinois, Louisiana, South Dakota, and Washington)

1.1.1 Method (Census of Agriculture - Experiment 1)

In experiment 1, records were selected from Illinois, Louisiana, South Dakota, and 

Washington for which agricultural activity was unknown. The control group 

received the standard data collection method. The treatment group received an 

automated telephone reminder after the first but before the second mailing of follow-up 

forms in addition to the standard data collection method (Appendix A). Table 1

summarizes the experiment.
6

Table 1: Automated Telephone Reminders

Treatment Group:

Automated Telephone 

Reminders

Control Group:

No Reminder Call

Sample Size 18,462 2,063

Initial mail out form Mailed on December 28, 2007

Thank you/reminder postcard1/ Mailed on January 14, 2008

First follow-up form Began February 11 and ended on February 28, 2008

Automated telephone 

reminder2/

Began February 16, 

2008 None

Second follow-up form Began March 24 and ended on March 28, 2008

1/ 2/Appendix A

1.1.2 Results (Census of Agriculture - Experiment 1)

Table 2 shows the outcome of the automated telephone reminder calls. Forty five percent 

of the treatment group was reached and listened to the entire message. The message was 

delivered to an answering machine 13 percent of the time. Approximately 25 percent of 

the cases did not receive the reminder message due to message calling rules (limit of 

three attempts). Thus, 58 percent of the calls were delivered to either a person or an 

answering machine.

Table 2: Automated Telephone Reminder Results

Outcome

Automated Telephone 

Reminder calls

Number %

No answer (maximum attempt 3 calls) 4,642 25.1

Answered by a person 8,210 44.5

Answered by a machine 2,463 13.3

Hung up/Partial message left 59 0.3

Other1/ 3,088 16.7

Total number of cases2/ 18,462 99.9

 1/ No connection, out-going message (OGM) too long, fax or modem, telephone company message, 

busy after voice. 

 2/ Percent Total may be greater or less than 100 percent due to rounding.

Table 3 displays the response rates by treatment group. The results showed that the 

completion rates were higher for the treatment groups for both in scope (1.2 percent) and 

out of scope (2.6 percent). The inaccessible rates were 3.7 percentage points lower for 
7

the group that received automated telephone reminders compared to the group that did 

not receive the automated telephone reminders. 

The Chi Square test results determined that frequency counts in each response code were 

not identically distributed across all groups. This signified that the group which received 

an automated telephone reminder had a significantly higher response rate than the group 

that did not receive a reminder call. 

Table 3: Response by Treatment Group

χ

2 (3, N=20,525) = 12.1356, p=0.0069

1.2 Census of Agriculture - Experiment 2: Automated Telephone and Extra

Postcard Reminders (California and Georgia)

1.2.1 Method (Census of Agriculture - Experiment 2)

In experiment 2, records were selected from California and Georgia for which 

agricultural activity was unknown. This experiment studied whether sending a second 

postcard reminder to respondents who already received an automated reminder message 

and a standard postcard reminder helped to increase response rates. Specifically, the 

control group received the standard data collection package. The treatment group 

received an automated telephone reminder after the first but before the second mailing of 

follow-up forms. The treatment group also received an extra postcard reminder after the 

second follow-up form in addition to the standard data collection method (Appendix A).

Table 4 summarizes the method used. 

Response Code

Treatment Group:

Automated Telephone 

Reminders

Control Group:

No Reminder Call

Number % Number %

Contacted in 

follow-up 18,462 2,063

IS Complete 1,659 9.0 160 7.8

OS Complete 5,454 29.5 555 26.9

Inaccessible 11,083 60.0 1,315 63.7

Refusal 266 1.5 33 1.6
8

Table 4: Automated Telephone and Extra Postcard Reminders

Treatment Group:

Automated Telephone and 

Extra Postcard Reminders

Control Group:

No Automated Telephone

or Extra Postcard

Reminders

Sample Size 25,951 2,862

Initial mail out form Mailed on December 28, 2007

Thank you/reminder 

postcard Mailed on January 14, 2008

First follow-up form Began February 11 and ended on February 28, 2008

Automated telephone 

reminder Began February 16, 2008 None

Second follow-up form Began March 24 and ended on March 28, 2008

Extra postcard reminder1/ Mailed between April 9th

and 16th

None

1/ Thank you/reminder postcard was different from the extra postcard reminder (Appendix A).

1.2.2 Results (Census of Agriculture - Experiment 1)

Table 5 displays the outcome of automated telephone reminder calls. Fifty-five percent

of the calls were delivered to either a person (44 percent) or an answering machine (11

percent). Thirty two percent of the cases did not receive the reminder message which 

may be due to limiting the number of attempts to three calls. 

Table 5: Outcome of Automated Telephone Reminder Attempts

Outcome

Automated Telephone Reminder 

calls

Number %

No answer (maximum attempt 3 calls) 8,355 32.2

Answered by a person 11,300 43.5

Answered by a machine 2,829 10.9

Hung up/Partial message left 87 0.3

Other1/ 3,380 13.0

Total number of cases2/25,951 99.92/

1/ No connection, OGM too long, fax or modem, telephone company message, and busy after 

voice. 

2/ Percent Total number of cases may be greater or less than 100 percent due to rounding.
9

Table 6 displays the response by group. For in scope and out of scope categories, the 

completion rates of the treatment group were higher than the control group by 1.1 and 2.0 

percentage points, respectively. The inaccessible rate was 2.6 percentage points lower 

for the group that received both automated telephone and extra postcard reminders, 

compared to the group that received the regular thank you/reminder postcard only. 

The Chi Square test results showed that frequency counts in each response category code 

were not identically distributed across all groups. This signified that automated telephone 

and extra postcard reminders significantly increased response rates. However, the result 

did not show whether the automated telephone or extra postcard reminders helped to 

increase the response rate. 

Table 6: Response by Treatment Group

χ

2 (3, N=28,813) = 15.3280, p =0.0016. 

1.3 Census of Agriculture - Experiment 3: Extra Postcard Reminder (Colorado, 

Florida, Nebraska)

1.3.1 Method (Census of Agriculture - Experiment 3)

In experiment 3, records were sampled from Colorado, Florida, and Nebraska for which

agricultural activity was unknown. This experiment was to determine whether sending 

extra postcard reminders to respondents who received automated telephone reminders, 

helped to increase response rates. This experiment was similar to the second experiment 

except that the control group received an automated telephone reminder call. Table 7

summarizes the experiment.

Treatment Group:

Automated Telephone and

Extra Postcard Reminders

Control Group:

No Automated Telephone or 

Extra Postcard Reminders

Number % Number %

Contacted in 

follow-up

25,951 2,862

IS Complete 1,851 7.1 173 6.0

OS Complete 5,681 21.9 570 19.9

Inaccessible 18,112 69.8 2,073 72.4

Refusal 307 1.18 46 1.61
10

 Table 7: Extra Postcard Reminder Treatment

Treatment Group:

Automated telephone and

Extra Postcard Reminders

Control Group:

No Extra Postcard

Reminder

Sample Size 11,525 1,305

Initial mail out form Mailed on December 28, 2007

Thank you/reminder 

postcard Mailed on January 14, 2008

First follow-up form Began February 11 and ended on February 28, 2008

Automated telephone 

reminder Began February 16, 2008

Second follow-up form Began March 24 and ended on March 28, 2008

Extra postcard reminder1/ Mailed between April 9th

and 16th

None

1/ Thank you/reminder postcard was different from the extra postcard reminder (Appendix A).

1.3.2 Results (Census of Agriculture - Experiment 3)

Table 8 displays the outcome of automated telephone reminder calls. Fifty-eight percent

of the calls were delivered to either a person (43 percent) or an answering machine (15

percent). Also, 25 percent of the cases did not receive the automated reminder message,

which may be due to the limited number of attempts.

Table 8: Outcome of Automated Telephone Reminder Attempts

Outcome

Automated Telephone

reminder calls

Number %

No answer (maximum attempt 3 calls) 3,265 25.4

Answered by a person 5,531 43.1

Answered by a machine 1,971 15.4

Hung up/Partial message left 31 0.2

Other1/ 2,032 15.8

Total number of cases2/ 12,830 99.9

1/ No connection, OGM too long, fax or modem, telephone company message, and busy after 

voice.

2/ Percent Total number of cases may be greater or less than 100 percent due to rounding.
11

Table 9 displays the response by group. For in scope and out scope categories, the

completion rates of the treatment group were slightly higher than the control group by 0.6 

and 0.3 percentage points, respectively. The inaccessible rate was 0.6 percentage points 

lower for the group that received both automated telephone and extra postcard reminders 

(in addition to the standard data collection method), compared to the group that received 

the standard data collection method. 

The Chi Square test results showed that distribution of counts in each response code were 

not significantly different across the two groups. In the other words, response rates were 

similar between the treatment and control groups. This suggested mailing an extra 

postcard reminder to respondents who received an automated telephone message 

reminder did not have a significant effect on response rates.

Table 9: Response by Treatment 

 χ

2 (3, N=12,830) = 0.8796, p =0.8303

1.4 Discussion (Census of Agriculture - Experiments)

Both Experiments 1 and 2 showed that automated telephone reminders were effective in 

increasing response rates. In addition, their effectiveness might be enhanced if more 

messages are delivered. Future uses of automated reminders should consider increasing 

the messaging capacity or the rules for the number of call attempts. However, mailing 

extra postcard reminders to respondents who already received the automated telephone 

reminders did not have a significant effect on the overall response rates as shown in 

Experiment 3. 

These experiments’ samples consisted of records for which agricultural activity was 

unknown, and response rates for this particular group have been historically lower than 

those with known agriculture. Thus, even a fractional improvement in response rates is 

deemed beneficial.

 

Treatment Group:

Automated Telephone and Extra 

Postcard Reminders

Control Group:

No Extra Postcard 

Reminder

 Number % Number %

Number contacted 

in follow-up

11,525 1,305

IS Complete 993 8.6 105 8.0

OS Complete 3,259 28.3 365 28.0

Inaccessible 7,052 61. 2 807 61.8

Refusal 221 1.9 28 2.1
12

2. NATIONAL ANIMAL HEALTH MONITORING SYSTEM 2011 SMALL 

PRODUCER STUDY

The NAHMS 2011 Small Producer Study (NAHMS) collects information on animal health, 

and food safety issues in 48 states. The Animal and Plant Health Inspection Service (APHIS), 

an agency of the Unites States Department of Agriculture, funds the survey and disseminates 

the findings.

The target population was livestock operations that meet the current USDA definition of small 

farms, having less than $250,000 value of sales. Also, included was a small portion of livestock 

operations between $250,000 and $500,000 value of sales to compare small farms against. Any 

in-business operation with a livestock farm type under $500,000 value of sales was eligible. 

The sample was stratified based on Commodity Groupings (Appendix C).

2.1 Method (NAHMS - Experiment)

NAHMS’ standard data collection process consisted of a pre-survey package, an initial 

mail out form, and, if necessary, a telephone follow-up. 

This experiment evaluated the effectiveness of different automated reminder messages. 

The control group received the standard data collection method. The two treatment 

groups received automated telephone reminders after the initial mail out but before the 

telephone follow up, in addition to the standard data collection method. Treatment group 

1 received an automated telephone message that included the statement; “Reply via mail 

by April 11, 2011 to avoid a telephone follow up contact”. Treatment group 2 received a

regular reminder/thank you message (Appendix B). 

Similar to the 2007 Census of Agriculture study, automated telephone reminder software 

was programmed to begin at 8:00 a.m. and end at 9:00 p.m., calling up to three times for 

unanswered numbers. Automated software was also set to hang-up after six rings, or if it 

reached an automated answering machine with an out-going message longer than 25 

seconds (these are assumed to be problem numbers or answering machines that were not

able to take recordings). Table 10 summarizes the experiment.
13

Table 10: Automated Telephone Reminder Treatment

2.2 Results (NAHMS - Experiment)

Respondents with invalid telephone numbers were excluded from all comparisons. Table 

11 shows the results. A person was reached and listened to the entire message for 

slightly over 40 percent, for both groups. Approximately 70 percent of the calls were 

delivered to either a person or an answering machine. Around 10 percent of the cases did 

not receive the automated telephone reminder, which may be due to the limited number 

of call attempts.

Table 11: Outcome of Automated Telephone Reminder Attempts

Outcome

Automated Telephone Reminder Calls

Message 1 Group

Number %

Message 2 Group

Number %

No answer (maximum attempt 3 

calls) 694 9.9 721 10.3

Answered by a person 2,874 41.1 2,869 40.1

Answered by a machine 1,943 27.8 1,935 27.6

Hung up/Partial message left 10 0.1 9 0.1

Telephone company message 472 6.7 446 6.4

Busy after Voice 85 1.2 92 1.3

No Call (not on called files). 852 12.2 855 12.2

Other 70 1.0 73 1.0

Total1/7,000 100.0 7,000 99.0

 

1/ Total Percent may not equal 100 percent due to rounding.

Treatment Group 1:

Automated 

Telephone Reminder 

Message 1

Treatment Group 2:

Automated 

Telephone Reminder 

Message 2

Group 3:

Control – No 

Automated Telephone 

Reminder

Sample Size 7,000 7,000 2,000

Pre-survey 

Package Mailed on March 14, 2011

Initial mail out 

form Began March 21 and ended April 8.

Automated 

telephone 

Reminder

Began March 28 None

Telephone 

follow-up

Began April 11 and ended April 29
14

Table 12 displays the response by treatment group and shows that there were minimal 

variations in the response rates across the groups. The difference in completion rates 

between Messages 1 and 2 was only by 0.2 percentage points. The completion rates of 

the treatment groups using Messages 1 and 2 were 1.0 and 0.8 percentage points higher, 

respectively, than the control group.

The Chi Square test results showed that the differences in response rates between groups

were not statistically significant. Thus, there was not enough evidence to conclude that 

the wording used in Message 1 or in Message 2 had an impact on response rates.

Table 12: Response by Treatment Group 

 

1/ Total Percent in each group may not equal 100 percent due to rounding.

 χ

2 (6, N=16,000) = 4.3074, p=0.6352

2.3 Discussion (NAHMS - Experiment)

Response rates for the groups which received automated reminders were higher than the 

group that did not, although in this case the difference was not statistically significant. 

There was also not a statistically significant difference between the messages that stated, 

“REPLY via mail by April 11, 2011 to avoid a telephone follow up contact,” the 

generic reminder message, and no reminder message. Testing on different or stronger 

worded messages that stress what kinds of follow-up contacts would happen if the form 

was not returned might produce different results. Also, using a trained or professional 

speaker or a NASS data user to record the message may also improve the effectiveness of 

this technology.

However, unlike the 2007 Census of Agriculture study, all of the non-responding 

NAHMS cases also received a subsequent computer assisted telephone interview (CATI). 

This may have diluted the measurable effectiveness of the automated telephone reminder 

message technology. Using automated telephone reminders earlier in the survey data 

collection process might increase the mail response so that additional data collection 

methods are not needed.

Treatment

Group 1

Automated Telephone

Reminder Message 1

Group 2

Automated Telephone

Reminder Message 2

Group 3

No Automated 

Telephone Reminder

Number % Number % Number %

Complete 4,176 59.7 4,166 59.5 1,173 58.7

Refusal 668 9.5 713 10.2 199 10.0

Inaccessible 1,346 19.2 1,305 18.6 405 20.3

Office Hold 810 11.6 816 11.7 223 11.2

Total1/7,000 100.0 7,000 100.0 2,000 100.2
15

IV. COST ANALYSIS

The National Processing Center in Jeffersonville, Indiana was contracted to provide these 

automated reminder calls for both 2007 Census of Agriculture and NAHMS studies. The 

estimated cost per completed call was 7 cents. A completed call means that an automated 

reminder message was delivered to a person or an answering machine. There was no 

charge if a call could not be made due to no connection, out-going message too long, fax 

or modem, telephone company message, or busy. In contrast, the total cost for a postcard 

reminder was 24.05 cents (1.25 cents for printing and 22.8 cents for mailing). Therefore, 

using automated telephone reminders can be a cost effective strategy to increase response 

rates.

V. CONCLUSION

The overall results showed that automated telephone reminders help increase response 

rates in some situations. However, mailing extra postcard reminders to operations who 

already received automated telephone reminders did not affect the overall response rates, 

nor did adding reminders in advance of CATI telephone interview calls. 

Additional research on the speaker and/or wording used in the automated telephone 

reminder messages could also be tested to obtain the most effective deployment of this 

technology.

Overall, the results showed that NASS should continue to use automated telephone 

reminders when applicable.

VI. RECOMMENDATIONS

1. Continue to use automated telephone reminders and conduct additional testing to 

determine the most effective use of reminder messages. This includes how to use 

automated reminders most effectively within standard NASS data collection 

procedures, trained or professional speakers, or a NASS data user to record the 

message. In addition, different content should also be tested; for example 

stressing that if respondents do not respond they will receive additional contacts.

2. Investigate additional uses of the automated telephone reminders for pre-survey 

notification messages.

3. Increase the number of call attempts to five when using automated telephone 

reminders.
16

VII. REFERENCES

Bolboaca S., Jantschi L., Sestra A., Sestras R., and Pamfil D. (2011). Pearson-Fisher 

Chi-Square Statistic Revisited. Basel, Switzerland: MDPI.

Bouffard, J., Brady, S., Stapleton, C. and Imel, J. (2003). Effect of New and Additional 

Contact Strategies on Cooperation Rates and Data Quality. Paper published in the 

Proceeding of the 2003 Joint Statistical Meeting.

Dini, E., Linkins, R., and Sigafoos, J. (2000). The Impact of Computer-Generated 

Messages on Childhood Immunization Coverage. American Journal of Preventative 

Medicine, 18(2), 132-139.

Fecso, R., Ferraiuolo, N. and Finamore, J. (2006). Using Autodialer Technology in 

Telephone Follow-up. Paper presented at the Second International Conference on 

Telephone Survey Methodology.

Franzini, L., Ronsenthal, J., Spears, W., Martin, H., Balderas, L., Brown, M., Milne, G., 

Drutz, J., Evans, D., Kozinetz, C., Oettgen B., and Hanson, C. (2000). Cost-Effectivess 

of Childhood Immunization Reminder/Recall System in Urban Private Practices. 

Pediatrics, 106(1), 177-183.

Krishna, S., Balas, E.A., Boren, S.A., and Maglaveras, N. (2002). Patient Acceptance of 

Educational Voice Messages: A Review of Controlled Clinical Studies. Methods of 

Information in Medicine, 41(5), 360-369.

McCarthy, J. (2007). Pre-Recorded Telephone Messages As An Alternative To FollowUp Survey Reminder Postcards: The Effectiveness Of Autodialer Software Used During 

The Pennsylvania County Estimates Survey. Research and Development Division Report 

#RDD-07-04, U.S. Department of Agriculture: National Agricultural Statistics Service.

McCarthy, J. (2008). Pre-Recorded Telephone Messages As An Alternative To FollowUp Survey Reminder Postcards II: The Effectiveness Of Autodialer Software Used 

During The 2007 Pennsylvania County Estimates Survey. Research and Development 

Division Report #RDD-08-04, U.S. Department of Agriculture: National Agricultural 

Statistics Service.
17

Appendix A

2007 Census of Agriculture

Post Card Reminder, Reminder Message, and Extra Postcard Reminder

2007 Census of Agriculture - Postcard Reminder
18

2007 Census of Agriculture - Reminder Message

This is a courtesy call from the U.S. Department of Agriculture reminding you to return 

your 2007 Census of Agriculture form. Everyone who received a census form is required 

to respond. Even if you are not farming, please indicate this on the form. Either way, your 

prompt response will save us from having to contact you again. If you already responded, 

thank you. The information you provide is confidential and protected by law. If you have 

questions, please visit www.agcensus.usda.gov or call, toll free, 888-424-7828.
19

2007 Census of Agriculture - Extra Postcard Reminder
20

Appendix B 

NAHMS - Reminder Messages 1 and 2

NAHMS - Reminder Message 1

Hello this is the United States Department of Agriculture’s National Agricultural 

Statistics Service reminding you of the due date for returning your Livestock Operations 

report form. REPLY via mail by April 11, 2011 to avoid a telephone follow up contact. 

THANK YOU if you already returned your report form. The study is designed to provide 

producers such as you with valuable information about the health, marketing, 

management, and biosecurity practices of small-scale livestock operations in the United 

States. If you have questions please call toll-free, 1-888-424-7828. Thank you for your 

cooperation.
21

NAHMS – Reminder Message 2

Hello this is the United States Department of Agriculture’s National Agricultural 

Statistics Service reminding you of the due date for returning your Livestock Operations 

report form. REPLY via mail by April 11, 2011. THANK YOU if you already 

returned your report form. The study is designed to provide producers such as you with 

valuable information about the health, marketing, management, and biosecurity practices 

of small-scale livestock operations in the United States. If you have questions please call 

toll-free, 1-888-424-7828. Thank you for your cooperation.
22

Appendix C

Commodity Groupings