Pre-Recorded Telephone 

Messages As An Alternative 

To Follow-Up Survey 

Reminder Postcards II: 

The Effectiveness Of Automated Telephone 

Reminder Messages Used During The 2007 

Pennsylvania County Estimates Survey 

Jaki S. McCarthy

 United States 

 Department of 

 Agriculture

 National 

 Agricultural 

 Statistics 

 Service

 Research and 

 Development Division 

 Washington DC 20250 

 RDD Research Report 

 Number RDD-08-04 

 May 2008 

This report was prepared for limited distribution to the research community outside the US Department 

of Agriculture. The views expressed herein are not necessarily those of NASS or USDA. 

i

EXECUTIVE SUMMARY 

During the 2007 Pennsylvania County Estimates Survey, a test of software which dials and 

delivers automated telephone messages was conducted, comparing it to postcard reminders and a 

control group with no reminders. The software automatically dialed and delivered a prerecorded message to two random subsamples of operations selected for the survey. Following 

the first mailing of the questionnaire, one subsample received a generic message which did not 

identify the speaker. A second subsample received an equivalent message recorded by the 

Pennsylvania Secretary of Agriculture. An additional third unique subsample received the same 

message on a traditional postcard reminder delivered via US Postal Service mail. Response rates 

for each of these groups were compared to a control group which received no reminder. All 

three reminder groups had higher response rates than the control group, with the generic 

autodialer message increasing response the most. 

As in previous research (McCarthy, 2007), this experiment showed the use of autodialer software 

to be effective and relatively inexpensive. Autodialer software is a promising addition to the 

tools field offices can use to increase survey response rates. Use of a prominent non-NASS 

agricultural speaker to record the message did not increase the effectiveness of the message. 

Methods to use this technology should continue to be evaluated. 
ii

RECOMMENDATIONS 

Based on the positive results of this and previous experiments, the following recommendations 

are given: 

1. Continue to use and evaluate the autodialer in Pennsylvania for follow-up reminders 

in mail surveys. 

2. Test alternatives to the methods employed, such as using a different message, trained 

or professional speakers, or another non-NASS data user to record the message. 

3. Consider additional uses of the autodialer software for reminders, such as for 

subpopulations in the census of agriculture. 
1

Pre-Recorded Telephone Messages As An Alternative To Follow-Up Survey 

Reminder Postcards II: The Effectiveness Of Autodialer Software Used 

During The 2007 Pennsylvania County Estimates Survey 

Jaki S. McCarthy 1

Abstract 

Software which dials specified telephone numbers and delivers an automated 

telephone message was tested as an alternative to postcard reminders in the 

Pennsylvania County Estimates Survey. Similar to prior research (McCarthy, 

2007), software automatically dialed and delivered a pre-recorded message to two 

randomly selected subsamples of operations. In the present experiment, one 

subsample received a message recorded by an unidentified speaker (the field 

office director) following the first mailing of the questionnaire. A second 

subsample received the same message recorded by the Pennsylvania Secretary of 

Agriculture. A third random sample received the same reminder message from 

the field office director, on a traditional postcard reminder delivered via US Postal 

Service mail. Response rates for each of these groups were compared to a control 

group which received no reminder. All three reminder groups had higher 

response rates than the control group, but unlike a previous test, the highest 

response rate was produced by the generic automated telephone message. 

Key Words: autodialer, automated message, follow-up reminder, postcard reminder, response 

rate 

1. BACKGROUND 

Postcards mailed as survey follow-up reminders have been used in the past to increase response 

rates and are a generally accepted best practice in survey data collection. An alternative to 

postcards delivered through the mail are messages left with respondents via the telephone. 

“Autodialer” software can be used to schedule and call lists of phone numbers and leave 

messages, either with a person or an answering machine. This technology has been used for 

many types of telephone messages, such as medical appointment reminders, broadcasts by 

groups to their membership, and emergency notifications. Research in some of these situations 

has shown that these reminders are effective and received positively by their recipients (Dini, 

Linkins, Sigafoos, 2000; Franzini, Rosenthal, Spears, Martin, Balderas, Brown, Milne, Drutz, 

Evans, Kozinetz, Oettgen and Hanson, 2000; Krishna, Balas, Boren and Maglaveras, 2002). 

There is very little documented use of automated reminder messages in survey research. 

 

1

 Jaki S. McCarthy is Chief Cognitive Research Methodologist in the Research and Development Division of NASS, 

Room 305, 3251 Old Lee Highway, Fairfax, VA 22030. The author would like to thank and acknowledge the extra 

efforts of the NASS Pennsylvania Field Office, who collaborated to develop this experiment and collected the data. 

Their agreement to include the control group was vital for our experiment, even though they assumed this would 

lower the response rate for that group. In particular, thanks go to Jillayne Weaber, the leader of the County 

Estimates Survey team, Christa Eisenhuth and Marc Tosiano. 
2

However, using automated telephone messaging does appear to increase response rates. The US 

Census Bureau found that response rates were higher for households who had been left a 

reminder message than for those which received either no reminder or a postcard reminder 

(Bouffard, Brady, Stapleton and Imel, 2003)2

. A test by the National Science Foundation (NSF) 

also showed that telephone reminder messages prompted slightly higher survey response rates 

(Fecso, 2006). An earlier test by the National Agricultural Statistics Service (NASS) 

Pennsylvania Field Office resulted in an increase in response rate for the groups given telephone 

reminders, although not as great an increase as that shown with reminder postcards (McCarthy, 

2007). 

Mailing postcards is relatively inexpensive, but autodialer systems are also inexpensive and over 

time may be cheaper than postcards. Long term operating costs of using telephone reminders 

would approach zero, if individual calls do not result in extra charges. For our previous study of 

autodialer reminder messages, the total estimated cost to call (2,224 calls; 1,652 minutes) was 

$34.12 in telephone company charges.3

 After the initial investment for the system, in this case, 

approximately $1000, this is clearly cheaper than printing and mailing postcards. Telephone 

calls are more easily controlled by the office, are not affected by mail delays, and are easily 

tailored as appropriate. 

The present experiment again compares use of the autodialer messages and postcard reminders to 

no reminders. However, in the present study, we are comparing the standard message with the 

same message recorded by the Pennsylvania Secretary of Agriculture, Dennis Wolff. The 

hypothesis is that farmers in Pennsylvania will regard a message from their state secretary of 

agriculture favorably, prompting higher response rates than a generic message. 

In this test, two different autodialer messages and postcard reminders (the currently planned 

operational procedure) were compared to a control group which did not receive any reminder 

follow-ups. This allowed a comparison of the two messages and an opportunity to see the 

impact over doing no reminders. 

2. METHODS 

The NASS Pennsylvania Field Office mailed questionnaires for the county estimates program to 

potential agricultural operations beginning in October 2007 as shown in Table 1. Data collection 

methodology was implemented as in prior surveys. A four page survey questionnaire collecting 

basic information on land operated, crop production, and livestock inventory was mailed to 

sampled operations. This was followed by a second mailing of the questionnaire to initial nonrespondents. Finally, telephone follow-up was made in another round of data collection for 

remaining non-respondents. 

 

2

 However, this Census Bureau study had several methodological limitations. Notably they did not use comparable 

samples, since only the small number of households where a telephone number look up was successful were 

included in the subsample receiving an automated reminder message. 

3

 Estimate by the NASS Pennsylvania field office, email dated 8/24/07. 
3

For this experiment, an additional follow-up reminder was added to data collection for three 

randomly assigned experimental groups. For the first two groups, an autodialer telephone 

reminder (one using a generic message and another recorded by the State Secretary of 

Agriculture) was added between the first and second questionnaire mailings. For a third group, a 

follow-up reminder postcard was added with similar content to the autodialer messages. The 

content of the messages is shown in Appendix A. The control group used the standard data 

collection procedures used in the past. Interviewers conducting the follow-up telephone 

interviews were not aware of which operations were in the experimental groups. 

The table below summarizes the experiment, with details following: 

Table 1. Experimental Group Treatments 

 Group 1 Group 2 Group 3 Group 4 

Treatment Autodialer 

reminder calls 

with unidentified 

speaker 

Autodialer reminder 

calls with State 

Secretary of 

Agriculture 

Postcard reminder 

follow-up 

Control 

(no reminder call or 

postcard) 

Conducted by 

Pennsylvania FO 

Conducted by 

Pennsylvania FO 

Mailed by print 

mail center 

Sample size (n) 3855 3858 2521 2518 

Date of initial 

mailing 

October 15 (via standard class mailing) 

Date of 

reminder followup 

Began November 6 ended on November 

19 

(no calls on Sundays) 

Mail on November 

5 

N/A 

Date of second 

mailing 

November 19 (via standard class mailing) 

Phone Data 

collection 

Follow-up 

Began December 3, continuing through January for all groups 

Respondents without phone numbers were excluded from all comparisons. The remaining 

experimental sample was randomly assigned to subsample replicates. Based on the previous 

success of automated messaging, larger samples were allocated to these groups. 

Because standard mail was used, exactly how long it would take the questionnaires to be 

delivered was unknown. A delivery time of at least one week, but possibly up to three weeks 

was estimated. However, we did not want to have the postcards or messages delivered before the 

questionnaires. For this reason, reminders did not begin until several weeks after the first 

mailing. The autodialer message and the content of the reminder postcard were comparable and 

are shown in Appendix A. They were written with the most critical information first, so in the 

event of a recipient hanging up early, they would hear the purpose of the call. 

The two autodialer reminder call groups differed only with respect to the speaker. The first 

message did not identify the speaker. The second message was the same as the first, except that 
4

the message was prefaced with Secretary of Agriculture Dennis Wolff identifying himself4

. The 

message recorded by Secretary Wolff had a slower pace and less color and intonation than the 

generic message. 

The autodialer software was capable of handling both messages at once, so both messages were 

delivered to the appropriate respondents throughout the calling period. The autodialer software 

was programmed to begin at 8:00 am and end at 9:00 pm, calling up to five times for unanswered 

numbers. The software was also set to hang-up after six rings with no answer. The software also 

hung up if it reached an automated answering machine with an outgoing message longer than 25 

seconds (these are assumed to be problem numbers or messages that do not take recordings). 

The pre-recorded message was left regardless of whether a person or answering machine took the 

call. 

The date of receipt for each questionnaire was recorded. In addition, mode of data collection 

(mail or telephone) was tracked to determine whether questionnaires were returned by mail or 

were completed during telephone follow-up. 

3. RESULTS AND DISCUSSION 

3.1 Outcome of Autodialer Message Attempts 

Although calls to a large number of operations in our experimental groups were made, not all 

were successful in reaching either a person or answering machine. As shown in the following 

table, a person was reached and listened to the entire message in a little less than 50% of the 

cases5

. In addition, 9% of Group 1 and 17% of Group 2 resulted in a person listening to only 

part of the recorded message. Finally, messages left on answering machines comprised the 

remainder of cases where the message was delivered to an operation (31% for Group 1 and 23% 

for Group 2). The percentages for Group 1 are similar to that obtained in a previous experiment 

with a similar message (McCarthy, 2007). However, for Group 2, where the message was left by 

the Pennsylvania Secretary of Agriculture, it appears that more people picked up the phone to 

listen to the message and then hung up before the complete message was delivered rather than let 

the answering machine record the message. 

 

4

 The messages differed slightly from those used in our previous experiment, since the Secretary of Agriculture 

edited and approved the message. 

5

 Autodialer results for 12 records in Group 1 and 10 records in Group 2 were inadvertently lost due to technical 

problems. They were excluded from this table. 
5

Table 2. Outcome of Autodialer Attempts 

Disposition Group 1 

n (%) 

Group 2 

n (%) 

Maximum attempts (5), no answer 148 (3.9) 162 (4.2)

Answered by a person 1868 (48.6) 1891 (49.1)

Answered by answering machine 1191 (31.0) 899 (23.3)

Hang up/Partial message left 354 (9.2) 646 (16.8)

Telephone Company Message (bad #) 173 (4.5) 155 (4.0)

Fax or Modem 17 (0.4) 18 (0.4)

Other 92 (2.4) 77 (2.0)

Total Cases6 3843 3848 

3.2 Follow-Up Reminder Effects on Response Rates 

Overall, postcard reminders and automated telephone reminders significantly increased response 

rates as shown in the next table, χ

2 (6, n=12752) = 17.18, p<.01. In contrast to previous research 

(McCarthy, 2007), the postcard reminder did not produce the highest increase in response rate. 

The follow-up reminders also appear to reduce some data collection costs, as more of the 

completions are received by mail instead of with more costly telephone enumeration (see Table 5 

in Appendix B). 

Table 3. Response by Treatment Group 

Treatment 

Autodialer 

reminder calls 

with message #1 

% (n) 

Autodialer 

reminder calls 

with message #2 

% (n) 

Postcard 

reminder 

follow-up 

% (n) 

Control 

(no reminder 

call or postcard) 

% (n) 

Number contacted 

in follow-up 

3855 3858 2521 2518 

Complete 58.6 (2259) 55.83 (2154) 56.37 (1421) 54.65 (1376) 

Inaccessible 40.23 (1551) 43.11 (1663) 42.17 (1063) 43.57 (1097) 

Refusal 1.17 (45) 1.06 (41) 1.47 (37) 1.79 (45) 

χ

2 (6, N=12752) = 17.18, p<.01 

The cumulative response rates by the date questionnaires were checked into the office are shown 

in Figure 1 in Appendix C. While individual responses cannot be directly tied to receipt of either 

messages or postcards, the chart shows no real difference in response until the second mailing. 

At this point, the returns for Group 1 jump and remain higher than the other groups for the 

remainder of the data collection period. 

Previous research suggested that postcards may be more effective than autodialer reminders due 

to the fact that they have a physical presence that telephone messages do not. However, in the 

 

6

 Call outcomes for 12 cases in group one and 10 cases in group 2 were inadvertently lost. 
6

present experiment, postcards did not prompt a higher response rate than the generic autodialer 

message. Between the two autodialer message groups, it appears that the increased number of 

messages that were only partially listened to for Group 2 may have adversely impacted response 

rates for that group. Perhaps in these cases, the person listening to the message was someone 

other than the targeted respondent who may not have passed that message on to the respondent. 

Screened messages on answering machines are recorded and can be replayed by other members 

of the household. This is not the case for messages that are partially listened to. 

Autodialer calls which were answered by a person (and listened to completely) were more likely 

to result in a completion (53.0%) than were calls where a message was left (47.4%) or where the 

person answering the phone hung up before the message had finished (41.9%). One interesting, 

but not surprising, finding from the autodialer outcomes is that certain types of autodialer 

outcomes are much more likely to remain inaccessible. Combining the two experimental 

autodialer groups, 56% of the operations which had invalid numbers (i.e. disconnected, fax or 

modem, etc.) never returned a questionnaire and were ultimately coded as inaccessible. In 

contrast, the percentage of cases which remained ultimately inaccessible was lower for groups 

where a message was delivered. Details are shown in Table 4 below. 

Table 4. Final Case Disposition for Telephone Reminder Groups by Message Outcome 

Final Case 

Disposition 

Answered by 

a Machine 

n (%) 

Answered by a 

Person 

n (%) 

Answered, 

Hung up Before 

Message 

Completed 

n (%) 

Other7

n (%) 

Complete 1134 (54.26) 2369 (63.02) 539 (53.90) 365 (43.35) 

Inaccessible 936 (44.78) 1341 (35.67) 450 (45.00) 471 (55.94) 

Refusal 20 (0.96) 49 (1.30) 11 (1.10) 6 (0.71) 

Total 2090 (100) 3759 (100) 1000 (100) 842 (100) 

One benefit to the use of the autodialer is that it helps to verify telephone numbers. Invalid and 

disconnected numbers may indicate that these are not currently operating agricultural operations. 

Indeed, more of these numbers ultimately are not reached than the group of working numbers. 

However, some of these “invalid” numbers (i.e. faxes, answering machines which do not accept 

messages, non-working numbers) do return questionnaires. Of the 420 records with “invalid” 

numbers, 187 (44.5%) did return completed questionnaires. 

4. FUTURE WORK 

This study confirms that autodialer software can be an effective tool to increase response rates. 

However, the test did not show any increased effectiveness for using an outside speaker, in this 

 

7

 Invalid numbers included all cases where either the call was answered with a telephone company (Tri-Tone) S.I.T. 

tone (disconnect/change); no signal was detected after dialing (indicating phone number is probably not valid); FAX 

or modem answered; call was answered by a machine but the outgoing message was longer than 25 seconds; or any 

other error was detected during the call. 
7

case, the Pennsylvania Secretary of Agriculture, to deliver the message. Perhaps results would 

be different for a different “celebrity” speaker, a professionally trained speaker or a speaker with 

different vocal qualities. 

The set up and administration time for autodialing is minimal and can easily be done by office 

staff. Messages can be tailored for different subgroups or can be altered during data collection. 

There are additional tests of the system that could be done to help optimize its effectiveness. For 

example: 

• Using another speaker from outside NASS to record the message. While the State 

Secretary of Agriculture did not prompt a higher level of response, perhaps another 

speaker such as a “celebrity” or farm group representative might be effective. The 

individual speaker chosen should be one likely to be viewed positively by the operations 

in the sample. 

• Modification of the content of the message to include more, less, or different information, 

such as specific uses of the data (tailored to specific types of operations), whether we 

intend to contact non-respondents by telephone, etc. 

• Customization of messages for subgroups of respondents 

• Changes to the timing of the reminders (including use as a pre-survey notification) 

• Use of the automated message in combination with a postcard reminder 

• Allowing the person called to connect to a live person in the office. 

In addition, the autodialer software may also be helpful in other mail out/mail back data 

collections such as the census of agriculture. Use of the autodialer messaging for the census of 

agriculture prior to mailing the forms is currently being tested. It could also be used with 

selected subgroups of respondents, such as low response counties, to help boost response rates. 

The software can also have uses beyond telephone reminders -- any messages can be left for any 

groups of phone numbers. For example, information or reminders can be broadcast to office 

staff, NASDA enumerators, or data users. Field offices may want to consider these and other 

uses for this software. 

5. RECOMMENDATIONS 

Based on the positive results of this and previous experiments, the following recommendations 

are given: 

1. Continue to use and evaluate the autodialer in Pennsylvania for follow-up reminders 

in mail surveys. 

2. Test alternatives to the methods employed, such as using a different message, trained 

or professional speakers, or another non-NASS data user to record the message. 

3. Consider additional uses of the autodialer software for reminders, such as for 

subpopulations in the census of agriculture. 
8

6. REFERENCES 

Bouffard, J., Brady, S., Stapleton, C. and Imel, J. (2003). Effect of New and Additional Contact 

Strategies on Cooperation Rates and Data Quality. Paper published in the Proceedings of the 

2003 Joint Statistical Meeting. 

Dini, E., Linkins, R., and Sigafoos, J. (2000). The Impact of Computer- Generated Messages 

on Childhood Immunization Coverage. American Journal of Preventative Medicine, 18(2), 132-

139. 

Krishna, S., Balas, E.A., Boren, S.A., and Maglaveras, N. (2002). Patient Acceptance of 

Educational Voice Messages: A Review of Controlled Clinical Studies. Methods of Information 

in Medicine, 41(5), 360-369. 

Fecso, R., Ferraiuolo, N. and Finamore, J. (2006). Using Autodialer Technology in Telephone 

Follow-up. Paper presented at the Second International Conference on Telephone Survey 

Methodology. 

Franzini, L., Rosenthal, J., Spears, W., Martin, H., Balderas, L., Brown, M., Milne, G., Drutz, J., 

Evans, D., Kozinetz, C., Oettgen B., and Hanson, C. (2000). Cost-Effectiveness of Childhood 

Immunization Reminder/Recall Systems in Urban Private Practices. Pediatrics, 106(1), 177-183. 

McCarthy, J. (2007). Pre-Recorded Telephone Messages As An Alternative To Follow-Up 

Survey Reminder Postcards: The Effectiveness Of Autodialer Software Used During The 

Pennsylvania County Estimates Survey. Research and Development Division Report #RDD-07-

04, U.S. Department of Agriculture: National Agricultural Statistics Service.
9

Appendix A. Reminder messages 

Text of telephone reminder message: 

Hi, (*This is just a reminder /I’m Dennis Wolff, Pennsylvania Secretary of 

Agriculture, reminding you) to return the tan colored County Estimates Survey 

recently sent from the National Ag Statistics Service of the USDA. 

If you’ve already sent yours in, thank you. 

The results of these surveys are instrumental in planning and implementing the 

programs that support our agriculture industry. Thanks for helping us show that 

Agriculture Counts in your county. 

*alternate wording for treatments 1 and 2 

Text of Postcard Reminder 

2007 County Estimates Survey

November 6, 2007 

Hi there! 

This is just a reminder to return the tan-colored County Estimates Survey

we recently sent from the National Ag Statistics Service in USDA. 

If you already sent it back, “Thank you!” 

The results of these surveys are instrumental in planning and implementing 

the programs that support our agriculture industry. Thanks for helping us 

show that Agriculture Counts in your county. 

- - Marc Tosiano, Director, USDA, NASS-PA Field Office 
10

Appendix B.

Table 5. Response by Mode. 

χ

2 (3, N=7191) = 6.47, p=.09 

Treatment Autodialer 

reminder calls 

with message 

#1 

Autodialer 

reminder 

calls with 

message 

#2 

Postcard 

reminder 

follow-up 

Control 

(no 

reminder 

call or 

postcard) 

Sample Size n 3855 3858 2521 2518 

Mail completes n (%) 1993 (51.7) 1863 (48.3) 1240 (49.2) 1176 (46.7) 

Telephone completes n (%) 266 (6.9) 291 (7.5) 181 (7.2) 200 (7.9) 

Overall Response % 58.6% 55.83% 56.4% 54.7% 
11

Appendix C. 

Figure 1 - Cumulative Response by Group 

0

10

20

30

40

50

60

70

10/22/2007

10/29/2007

11/5/2007

11/11/2007

11/19/2007

11/26/2007

12/3/2007

12/10/2007

12/17/2007

12/24/2007

1/2/2008

1/9/2008

1/16/2008

Check in Date

Percent Response

Group 1 Group 2 Group 3 Group 4

11/5, Postcards mailed 

11/7, Autodialer begins 

11/19, 2nd Mailing 

12/3, Follow-up calls begin