Pre-Recorded Telephone

Messages As An Alternative

To Follow-Up Survey 

Reminder Postcards:

The Effectiveness Of Autodialer Software

Used During The Pennsylvania County

Estimates Survey

Jaki S. McCarthy

United States

Department of

Agriculture

National

Agricultural

Statistics

Service

Research and

Development Division

Washington DC 20250

RDD Research Report

Number RDD-07-04

June 2007

This report was prepared for limited distribution to the research community outside the US Department

of Agriculture. The views expressed herein are not necessarily those of NASS or USDA.

1

EXECUTIVE SUMMARY

Software which dials and delivers automated telephone messages was tested as an alternative to

postcard reminders during the Pennsylvania County Estimates Survey. The recorded telephone

messages were automatically dialed and delivered to two subsamples of telephone numbers of

operations selected for the survey that had not already responded. One subsample received the

message following the first mailing of the questionnaire. A second subsample received the

message immediately preceding the second mailing of the questionnaire. An additional third 

unique subsample received the same message as a traditional postcard reminder delivered via US

Postal Service mail. Response rates for each of these groups were compared to a control group

which received no reminder. All three reminder groups had higher response rates than the

control group, with the postcard reminder increasing response the most.

The use of autodialer software appears to be effective and is relatively inexpensive. While not

quite as effective in the present experiment as mailed reminder postcards, autodialer calls can be

easily implemented by a field office and messages can be tailored to specific groups. The use of

autodialer software appears to be a promising addition to the tools field offices can use to

increase response rates in mail surveys. Methods to use this technology should continue to be

tested and evaluated.
2

RECOMMENDATIONS

Based on the positive results of this initial experiment, the following recommendations are made:

1. Continue to use and evaluate the autodialer in Pennsylvania for follow-up reminders

in mail surveys.

2. Test alternatives to the methods employed, such as using alternative messages or a

non-NASS data user to record the message.

3. Consider additional uses of the autodialer software for reminders, such as for

subpopulations in the census of agriculture.
3

Pre-Recorded Telephone Messages As An Alternative To Follow-Up Survey 

Reminder Postcards: The Effectiveness Of Autodialer Software Used During 

The Pennsylvania County Estimates Survey

Jaki S. McCarthy 1

 1 Jaki S. McCarthy is Chief Cognitive Research Methodologist in the Research and Development Division of NASS,

Room 305, 3251 Old Lee Highway, Fairfax, VA 22030. The author would like to acknowledge the efforts of the

NASS Pennsylvania Field Office, who collaborated to develop this experiment and implemented and collected the

data. In particular, thanks go to Jillayne Weaber, Christa Eisenhuth and Marc Tosiano.

Abstract

Software which dials specified telephone numbers and delivers an automated 

telephone message was tested as an alternative to postcard reminders in the

Pennsylvania county estimates survey. The recorded telephone message was

automatically dialed and delivered to two randomly selected subsamples of

operations selected for the survey. One subsample received the message

following the first mailing of the questionnaire. A second subsample received

the message immediately preceding the second mailing of the questionnaire. A

third random sample received the same reminder message on a traditional

postcard reminder delivered via US Postal Service mail. Response rates for each 

of these groups were compared to a control group which received no reminder.

All three reminder groups had higher response rates than the control group, with 

the postcard reminder increasing response the most.

Key Words: autodialer, automated message, follow-up reminder, postcard 

reminder, response rate

.
4

1. BACKGROUND

Postcards mailed as survey follow-up reminders have been used in the past to increase response

rates and are a generally accepted best practice in survey data collection. An alternative to

postcards delivered through the mail are messages left with respondents via the telephone.

“Autodialer” software can be used to schedule and call lists of phone numbers and leave

messages, either with a person or an answering machine. This technology has been used for

many types of telephone messages, such as medical appointment reminders, broadcasts by 

groups to their membership, and emergency notifications. Research in some of these situations

has shown that these reminders are effective and received positively by their recipients (Dini,

Linkins, Sigafoos, 2000, Franzini, Rosenthal, Spears, Martin, Balderas, Brown, Milne, Drutz,

Evans, Kozinetz, Oettgen and Hanson, 2000, Krishna, Balas, Boren and Maglaveras, 2002).

There is very limited documented use in survey research. However, using automated telephone

messaging does appear to increase response rates. The U.S. Census Bureau found that response

rates were higher for households who had been left a reminder message than for either no

reminder or a postcard reminder (Bouffard, Brady, Stapleton and Imel, 2003)2. A test by the

National Science Foundation (NSF) also showed that telephone reminder messages prompted 

slightly higher survey response rates (Fecso, 2006).

Mailing postcards is relatively inexpensive, but autodialer systems are also inexpensive and over

time may be cheaper than postcards. Long term operating costs of using telephone reminders

would approach zero, if individual calls do not result in extra charges. Telephone calls can also 

be more tightly controlled by the office, are not affected by mail delays, and are easily tailored as

appropriate.

The present experiment compares use of the autodialer messages and postcard reminders to no 

reminders. Also, interest lies in whether respondents who receive the message will view them

positively or negatively.

In this test, two different uses of the autodialer messaging, and postcard reminders (the currently

planned operational procedure) were compared to a control group which did not receive any 

reminder follow-ups. This allowed a comparison of the two methods of using reminders and also 

to see the impact over doing no reminders. This was also a test of use of the specific system that

we have selected (PhoneTree) and how best to implement its particular features, although this

was of secondary importance.

2. METHODS

The Pennsylvania Field Office of the National Agricultural Statistics Service (NASS) mailed

questionnaires for the county estimates program to approximately 16,000 potential agricultural

 2 However, this Census Bureau study had several methodological limitations, notably they did not use comparable

samples, since only the small number of households where a telephone number look up was successful were

included in the subsample receiving an automated reminder message.
5

operations beginning in October 2006. A 4 page survey questionnaire was used that collected 

basic information on land operated, crop production and livestock inventory. A second mailing

of the questionnaire was made to initial non-respondents. This was followed with telephone

follow-up in another round of data collection. This was the standard data collection 

methodology used in previous county estimate surveys.

For this experiment, an additional follow-up reminder was added to data collection for three

randomly assigned experimental groups. For the first two groups, an autodialer telephone

reminder was added between the first and second questionnaire mailings. For a third group, a

follow-up reminder postcard was added. The control group used the standard data collection 

procedures used in the past and outlined above. The follow-up telephone interviewers were not

aware of which operations were in the experimental groups.

The table below summarizes the experiment, with details following:

Table 1. Experimental Group Treatments

All respondents without phone numbers were excluded from all comparisons. The remaining

experimental sample was randomly assigned to subsample replicates of slightly less than 2000

respondents each.

Because standard mail was used, we did not know precisely how long it would take the

questionnaires to be delivered and estimated a delivery time of at least one week, but possibly up 

to three weeks. However, we did not want to have the postcards or messages delivered before

the questionnaires. For this reason, we did not begin reminders until 3 weeks after the first

Group 1 Group 2 Group 3 Group 4

Treatment Autodialer reminder

calls with early

message #1

Autodialer reminder

calls with late

message #2

Postcard reminder

follow-up

Control

(no reminder call or

postcard)

Conducted by

Pennsylvania FO

Conducted by

Pennsylvania FO

Mailed by print mail

center

Date of initial

mailing

October 16 (via standard class mailing)

Date of reminder

follow-up

Began Monday,

November 6 ended on

Wednesday,

November 8

Begin Monday,

November 20 ending

on Wednesday,

November 22

Mail on November 6 N/A

Date of second 

mailing

November 20 (via standard class mailing)

Phone Data

collection Followup

Began December 11, continuing through January for all groups

Autodialer

Evaluation 

Follow-up Calls

50 operations contacted to find out their

opinion about autodialer messages (25 with a

message left with a person; 25 with a message

left on a machine)
6

mailing. The autodialer message and the content of the reminder postcard were comparable and 

are shown in Appendix A. They were written so that the most critical information was first, so in 

the event of a recipient hanging up early, they would hear the purpose of the call.

The two autodialer reminder call groups differed in timing. Research on reminder postcards has

indicated that they are most effective when delivered fairly close in time to the questionnaire.

This is usually done by mailing postcards to be received immediately after the first questionnaire

mailing. However, since we cannot tightly control the questionnaire delivery, the second 

autodialer group was scheduled to deliver the reminder immediately preceding the second 

mailing of the questionnaire. Since sampled operations will have received their first mailing, the

reminder can be delivered without worrying about getting to them before they have received any 

questionnaire (even if it gets delivered before the second mailing.) Autodialer calls were made

to all sampled operations that were non-respondents as of the Friday immediately preceding the

calls for that group, which all began on Monday.

The autodialer software was programmed to begin at 8:00 am and end at 9:00 pm, calling up to 5 

times for unanswered numbers. The software was also set to hang-up after 6 rings with no 

answer. The software also hung up if it reached an automated answering machine with an 

outgoing message longer than 25 seconds (these are assumed to be problem numbers or

messages that do not take recordings.) Our pre-recorded message was left regardless of whether

a person or answering machine answered.

The date of receipt for each questionnaire was recorded. In addition, mode of data collection

(mail or telephone) was tracked to determine whether questionnaires were returned by mail or

were completed during telephone follow-up.

In order to assess any adverse impact on respondent’s attitudes toward NASS or the

Pennsylvania Field Office, 50 respondents who received the autodialer message were recontacted 

(25 who were answered in person and 25 who had messages left on an answering machine.)

Recontacts were made after all data collection for the county estimates was complete in January.

They were asked whether they recalled receiving the message and what they thought about it

with the following questions:

• “Do you or anyone else recall receiving a telephone message from our office asking you 

to return your survey form?”

• “What did you think about that message? Did you think it was: helpful in prompting you 

to return the form; made no difference to whether you returned the form or not; a waste

of time/annoying; or anything else?”

• “Do you think these types of reminder messages will help us prompt other people to 

return their survey forms?”

These questions are similar to follow-up evaluations that have been done by organizations using 

autodialers for appointment reminders. The idea with these follow-up contacts was not to make
7

any statistical estimates of the proportion of people with positive or negative attitudes, but to get

a rough idea of whether a large proportion of respondents viewed these reminders as negative.

3

3. RESULTS AND DISCUSSION

A. Outcome of Autodialer Message Attempts

Although calls to a large number of operations in our experimental groups were made, not all

were successful in either reaching a person or answering machine. As shown in the following

table, a person was reached and listened to the entire message in a little less than 50% of the

cases (45% for Group 1 and 43% for Group 2). In addition, 13% of each group resulted in a

person listening to only part of the recorded message. Finally, messages left on answering 

machines were the remainder of cases where the message was delivered to an operation (31% for

Group 1 and 28% for Group 2).

Table 2. Outcome of Autodialer Attempts

B. Follow-Up Reminder Effects on Response Rates

Overall, both postcard reminders and automated telephone reminders significantly increased 

response rates as shown in the next table, χ

2 (6, N=13004) = 30.05, p<.0001. There was little

difference between the early and late telephone reminders with both increasing response rates

almost 4% over the control group. The postcard reminder increased response rates even more,

with a 5% increase in response over the control group. The follow-up reminders appear to 

reduce some data collection costs as well, as more of the completions are received by mail (see

Table 5 in Appendix B).

 3 Potential alternatives to making a separate contact were considered but have significant problems. For example,

we considered asking about the telephone message in any later telephone follow-ups, or of any of the county

estimates sample which happened to be in other closely following surveys. However, adding questions to the county

estimates telephone follow-ups would eliminate anyone who had already returned their questionnaire. The

remaining non-respondents who need a telephone follow-up may not have heard the message, or may generally have

more negative attitudes. So any conclusions drawn from contacting only these respondents might be negatively

biased. In addition, we discussed whether the Pennsylvania FO might be more interested in attitudes of larger

operations who are likely to be contacted more often. This is something we may want to look at more closely in the

future.

Disposition Group 1 

N (percent)

Group 2

N (percent)

Maximum attempts (5), no answer 65 (4.0%) 84 (5.9%)

Answered by a person 730 (45.1) 616 (43.4)

Answered by answering machine 508 (31.4) 396 (27.9)

Hang up/Partial message left 211 (13.0) 190 (13.4)

Telephone Company Message (bad #) 87 (5.4) 110 (7.8)

Fax or Modem 3 (.2) 9 (.6)

Duplicate number (not called) 2 ( .1) 0 (0)

Other 13 (.8) 13 (.9)

Total Cases called 1619 1418
8

Table 3. Response by Treatment Group

Treatment

Autodialer

reminder calls

with message #1

% (n)

Autodialer

reminder calls

with message #2

% (n)

Postcard 

reminder

follow-up

% (n)

Control

(no reminder

call or postcard)

% (n)

Number contacted in 

follow-up

(treatment not

administered for

operations who had 

already returned a

questionnaire)

(1619) (1418) (1858) (0)

Complete 56.6% (1052) 56.5% (1051) 57.8% (1073) 52.8% (3918)

Inaccessible 41.7 (775) 42.0 (781) 39.9 (742) 45.6 (3386)

Refusal 1.7 (31) 1.5 (28) 2.3 (43) 1.7 (124)

N 100 (1858) 100 (1860) 100 (1858) 100 (7428)

χ

2 (6, N=13004) = 30.05, p<.0001

The cumulative response rates by the date questionnaires were checked into the office is shown 

in Figure 1 in Appendix C. While we cannot tie any individual responses directly to receipt of

either messages or postcards, the chart shows that the early reminder calls and mailings give

response rates an immediate increase over the control and late message groups.

Postcards may be more effective than autodialer reminders due to the fact that they have a

physical presence that telephone messages do not. Postcards can be passed from one person to

another (if the initial recipient is not the respondent), and postcards also can serve as a constant

reminder if the respondent keeps them in view. Obviously, this is not the case for telephone

messages. In addition, not all members of our experimental groups even received an automated 

message with some numbers being out of service, and some which were never answered.

Autodialer calls which were answered by a person were more likely to result in a completion 

(53.0%) than calls where a message was left (47.4%) or where the person answering hung up 

before the message had finished (41.9%). One interesting, but not surprising, finding from the

autodialer outcomes is that certain types of autodialer outcomes are much more likely to remain 

ultimately inaccessible. Combining the two experimental autodialer groups, 71.9% of the

operations which had invalid numbers (i.e. disconnected, fax or modem, invalid number, etc.)

ultimately never returned a questionnaire and were ultimately coded as inaccessible. In contrast,

the percentage of cases which remained ultimately inaccessible was lower for both the group

which never answered (but were not obviously invalid) and the group where a message was

delivered. Details are shown in Table 4 below.
9

Table 4. Final Case Disposition for Telephone Reminder Groups by Message Outcome

One benefit to the use of the autodialer is that it helps to verify telephone numbers. Invalid and

disconnected numbers may indicate that these are not currently operating agricultural operations.

Indeed, more of these numbers ultimately are not reached than the group of working numbers.

However, some of these “invalid” numbers do return questionnaires. While we know nothing of

the cases which remained inaccessible, examination of the 49 complete returns in the group with

invalid numbers showed that 39 of the operations were classified as in scope records and 10 were

classified as out of business. So while a majority of invalid numbers may not be agricultural

operations, we cannot say that none of them are.

C. Respondent Reactions to Autodialer Messages

The follow-up contacts with respondents who had been called by the autodialer resulted in most

of the respondents being unable to recall receiving that call. This was true for both groups, with 

only 3 of 25 operations where a person had answered the telephone remembering the call and 

just one of 25 where a message was left on an answering machine remembering the call. None

of those 4 respondents was bothered by the call and none thought it would be harmful to use

autodialers to prompt response in the future.

D. Field Office Experience with the PhoneTree Hardware and Software

The staff in the Pennsylvania field office did not have any trouble either setting up or using the

autodialer system. The system was up and running within a few hours of its receipt. The

PhoneTree system purchased for this evaluation used only a single telephone line and cost just

over $1000. Additional ports can be purchased to allow dialing on multiple lines. It was

installed on a stand alone machine to minimize any security concerns. A file with the names and 

telephone numbers to be called had to be generated and copied to that machine. In addition, the

message used had to be recorded, but no special equipment was required for this.

 4 Invalid numbers included all cases where either the call was answered with a telephone company (Tri-Tone) S.I.T.

tone (disconnect/change); no signal was detected after dialing (indicating phone number is probably not valid); FAX

or modem answered; call was answered by a machine but the outgoing message was longer than 25 seconds; or any

other error was detected during the call.

Final Case Disposition

Answered 

by a

Machine

N (%)

Answered 

by a

Person

N (%)

Answered,

Hung up 

Before

Message

Completed

N (%)

Invalid 

Number

4

N (%)

No 

Answer

N (%)

Complete 429 (47.4%) 715 (53.0%) 168 (41.9%) 49 (27.5%) 68 (33.0%)

Inaccessible 466 (51.4%) 605 (44.9%) 223 (55.6%) 128 (71.9%) 133 (64.6%)

Refusal 11 (1.2%) 29 (2.2%) 10 (2.5%) 1 (.6%) 5 (2.4%)

Total 906 1349 401 178 206
10

4. FUTURE WORK

This initial test of the autodialer software has shown that it can be an effective tool to increase

response rates. While it did not increase response rates as much as a reminder postcard, over the

long run it will ultimately be cheaper to use than postcards. The set up and administration time

for autodialing is minimal and can easily be done by office staff. Messages can be tailored for

different subgroups or can be altered during data collection. Additional testing of the system

could be done to help optimize its effectiveness. For example:

• Using someone from outside NASS to record the message. For county estimates, the

state commissioner of agriculture or a state level farm group representative might be

appropriate. Their message could highlight why they think that survey’s data are

important.

• Modification of the content of the message to include more, less, or different information,

such as specific uses of the data (tailored to specific types of operations), whether we

intend to contact non-respondents by telephone, etc.

• Customization of messages for subgroups of respondents

• Changes to the timing of the reminders (including use as a pre-survey notification)

• Use of the automated message in combination with a postcard reminder

• Allowing the person called to connect to a live person in the office.

In addition, the autodialer software may also be helpful in other mail out/mail back data

collections such as the census of agriculture. It could be used with selected subgroups of

respondents, such as low response counties, to help boost response rates.

The software can also have uses beyond telephone reminders, any messages can be left for any 

groups of phone numbers. For example, information or reminders can be broadcast to office

staff, NASDA enumerators, or data users. Field offices may want to consider other uses for this

software.

5. RECOMMENDATIONS

Based on the positive results of this initial experiment, the following recommendations are made:

1. Continue to use and evaluate the autodialer in Pennsylvania for follow-up reminders

in mail surveys.

2. Test alternatives to the methods employed, such as using an alternative message or a

non-NASS data user to record the message.

3. Consider additional uses of the autodialer software for reminders, such as

subpopulations in the census of agriculture.
11

6. REFERENCES

Bouffard, J., Brady, S., Stapleton, C. and Imel, J. (2003). Effect of New and Additional Contact

Strategies on Cooperation Rates and Data Quality. Paper published in the Proceedings of the

2003 Joint Statistical Meeting.

Dini, E., Linkins, R., and Sigafoos, J. (2000). The Impact of Computer- Generated Messages

on Childhood Immunization Coverage. American Journal of Preventative Medicine, 18(2), 132-

139.

Krishna, S., Balas, E.A., Boren, S.A., and Maglaveras, N. (2002). Patient Acceptance of

Educational Voice Messages: A Review of Controlled Clinical Studies. Methods of Information 

in Medicine, 41(5), 360-369.

Fecso, R., Ferraiuolo, N. and Finamore, J. (2006). Using Autodialer Technology in Telephone

Follow-up. Paper presented at the Second International Conference on Telephone Survey

Methodology.

Franzini, L., Rosenthal, J., Spears, W., Martin, H., Balderas, L., Brown, M., Milne, G., Drutz, J.,

Evans, D., Kozinetz, C., Oettgen B., and Hanson, C. (2000). Cost-Effectiveness of Childhood 

Immunization Reminder/Recall Systems in Urban Private Practices. Pediatrics, 106(1), 177-183.
12

Appendix A. Reminder messages

Text of telephone reminder message:

Hi there! This is just a reminder to return the County Estimates Survey we recently 

sent from the National Ag Statistics Service in USDA. It was a green 4-page

survey that we use each year to create county statistics for the major commodities

in Pennsylvania.

If you already sent it back, “Thank you!” Otherwise, please send it in (even if you 

are not currently farming), so we don’t have to bother you with another mailing. If

you can’t find the green survey form, we will send another copy in a [couple of

weeks/few days]*.

County statistics are very important to many in agriculture such as Extension,

teachers, lenders, agencies that implement government programs and grower

organizations that work on behalf of farmers. We need your help to show that

'Agriculture Counts' in your county.

*alternate wording for treatments 1 and 2

Text of Postcard Reminder

2006 County Estimates Survey

November 6, 2006

Hi there! This is just a reminder to return the County Estimates Survey we recently 

sent from the National Ag Statistics Service in USDA. It was a green 4-page survey that

we use each year to create county statistics for the major commodities in Pennsylvania.

If you already sent it back, “Thank you!” Otherwise, please send it in (even if you are not

currently farming), so we don’t have to bother you with another mailing. If you can’t

find the green survey form, we will send another copy in a couple of weeks.

County statistics are very important to many in agriculture such as Extension, teachers,

lenders, agencies that implement government programs and grower organizations that

work on behalf of farmers. We need your help to show that 'Agriculture Counts' in your

county.

- - Marc Tosiano, Director, USDA, NASS-PA Field Office
13

Appendix B.

Table 5. Response by Mode.

Treatment Autodialer

reminder

calls with 

message

#1

Autodialer

reminder

calls with 

message

#2

Postcard 

reminder

follow-up

Control

(no 

reminder

call or

postcard)

N 1858 1860 1858 7428

Mail completes (%) 908 (48.9%) 920 (49.5%) 942 (50.7%) 3280 (44.2%)

Telephone completes (%) 144 (7.1%) 131 (7.8%) 131 (7.1%) 638 (8.6%)

Overall Response (%) 56.6 % 56.5% 57.8% 52.8%
14 0

10

20

30

40

50

60

10/23/200610/30/2006

11/6/2006

11/13/2006

11/20/200611/27/2006

12/4/2006

12/11/200612/18/200612/25/2006

1/1/20071/8/2007

1/15/20071/22/20071/29/2007

completion date

Response Rate

Control Tel 1 Tel 2 Postcard

Tel. followup

begins, all

groups Tel group #1

msgs begin;

Postcards

mailed Tel group #2

msgs begin;

Second Q.

mailing

Appendix C.

Figure 1 - Cumulative Response by Group