1 

The Impact of Survey Communications on Response Rates 

Frances M. Barlas, 1 Amy E. Falcone, 1 Nikki D. Bellamy, 2 and Amy R. Mack1

1

ICF International 

2

Substance Abuse and Mental Health Services Administration 

Proceedings of the 2013 Federal Committee on Statistical Methodology (FCSM) Research Conference 

Response rates for all modes of survey administration have been declining in recent years and web-based surveys 

tend to have lower response rates than other administration modes. Past research has shown that the design and 

presentation of survey communications can affect survey response rates (De Leeuw et al., 2007; Dillman, Smyth, 

and Christian, 2009). However, much of this work has focused on mailed prenotifications and invitations; less 

research has focused on design decisions in email communications and their impact on response rates to web 

surveys. In this paper we test the impact of two features of survey communications on survey participation and 

response rates – email length and placement of the web survey URL within the email. 

This works builds on a past study, Falcone et al. (2012), which tested email invitation length and the location of the 

survey consent button on the landing page. That study found no difference in survey participation by email length 

or location of the survey consent button that initiates the web survey either at the top or bottom of the landing page; 

however, among those who viewed the landing page, those sent a long survey invitation with the survey consent 

button at the top of the landing page had the highest survey completion rate among the four conditions. 

Survey Communication Length 

Advice and preliminary research on email communication design for web surveys has been mixed. Guidance on 

length of email communications typically supports keeping survey communication emails short and to the point 

(Dillman, Smyth, and Christian, 2009). However, Kaplowitz et al. (2013) found that faculty and staff sent longer 

invitations had higher response rates than those sent shorter invitations, with no difference in response rates by 

invitation length among students. In addition, the results of the prior study by Falcone et al. (2012) suggest little 

impact of email length on survey initiation and associate longer emails with higher rates of survey completion. With 

this evidence that shorter survey communications may not necessarily lead to higher response rates, we were curious 

about the impact of moving some of the necessary language that we typically include on the web survey landing 

page to the email communication in order to eliminate the need for a landing page, therefore decreasing the overall 

length of the web survey and potentially minimizing perceived respondent burden. 

In conducting surveys sponsored by Federal agencies, the Office of Management and Budget (OMB) and the 

Protection of Human Subjects and Internal Review Boards typically require specific disclosures to respondents prior 

to their participation in the study. The Office of Management and Budget’s Standards and Guidelines for Statistical 

Surveys requires that researchers notify respondents of a number of key pieces of information: 

“Agencies must ensure that each collection of information instrument clearly states the reasons the 

information is planned to be collected; the way such information is planned to be used to further 

the proper performance of the functions of the agency; whether responses to the collection of 

information are voluntary or mandatory (citing authority); the nature and extent of confidentiality 

to be provided, if any, citing authority; an estimate of the average respondent burden together with 

a request that the public direct to the agency any comments concerning the accuracy of this burden 

estimate and any suggestions for reducing this burden; the OMB control number; and a statement 

that an agency may not conduct and a person is not required to respond to an information 

collection request unless it displays a currently valid OMB control number.” 

In addition, the Protection of Human Subjects and Institutional Review Board require that voluntary consent 

language be conveyed to all respondents. 

These OMB and voluntary consent statements are often included on the web survey landing page. Given the prior 

research which found no detrimental, or even positive, effects on response rates with longer email invitations, we 

were curious about the impact of moving the OMB statement and the voluntary consent language from the landing 
2 

page to the survey communications. Placing this information in the email communication would eliminate the need 

for a web survey landing page; as such respondents who received a longer communication could skip the landing 

page and immediately enter the survey when clicking on the link from the invitation. We hypothesized that skipping 

the landing page might diminish the impression of survey length and burden and result in more survey completes. 

In addition, eliminating the landing page would result in individuals only having to decide once to begin the survey 

– they click on the link in their email and immediately begin completing the survey. Those presented with a landing 

page have two decision points; respondents first click on the link in their email and then have to click on a second 

link on the landing page to initiate the survey. We hypothesized that eliminating the landing page would result in 

higher rates of survey initiation and completion. 

URL Location 

Couper (2008) recommends placing the web survey URL near the top of communication to avoid the need for 

respondents to scroll to view the link and access the survey. However, a recent study by Kaplowitz and colleagues 

(2013) found that two of their three respondent groups (faculty and students, but not staff) were more likely to 

respond to the invitation with the link at the bottom. We noted that, in their test, they placed the link either 

immediately following the greeting before any introductory text or at the very bottom of the communication, below 

the signature. To further explore the impact of URL placement, we were curious about the impact of a link

placement towards the top, after a very brief introduction of the purpose of the communication, the research aim, 

and the survey sponsor. Similarly, rather than placing the link at the very bottom of the email, we were curious 

about placing it toward the bottom after a longer introduction on the background, purpose of the survey, and reason 

for respondent selection. 

Method 

We conducted a two by two factorial experiment testing two elements of email survey communications – length and 

URL location. Respondents were randomly assigned to receive a short or long email with varying link location 

either toward the top or bottom of the email; Table 1 summarizes the experimental conditions. See Appendix A for 

examples of each of the four email invitations. Respondents received follow-up survey reminder emails consistent 

with their assigned condition; that is, each reminder had four versions consistent with the invitations. 

Table 1. Experimental Conditions 

Treatment Alternatives Description 

Length Short 174 words 

Long 386 words including OMB and voluntary consent 

language 

URL Location Top After brief introduction 

Bottom Toward bottom after long introduction 

All of the additional content contained in the long emails, including the OMB and voluntary consent statements, was 

presented on the web survey landing page for those who received the short emails. As such, those sent long email 

invitations and reminders skipped the web survey landing page and went directly to the first survey question when 

clicking the survey link. The page displaying the first survey question, with instructions at the top for navigating the 

survey, was identical for those sent the long and short emails. Images of the landing page and first survey page are 

included in Appendix B. 

Population and Sample 

This experiment was conducted with the 2013 Substance Abuse and Mental Health Services Administration Disaster 

Technical Assistance Center (SAMHSA DTAC) Customer Satisfaction Survey. The survey was administered to a 

census of coordinators and providers of disaster behavioral health services who had either requested training or 

technical assistance from SAMHSA DTAC in the past 12 months or were subscribed to one of their ecommunications.1

 There were a total of 345 customers invited to complete the survey, with 86 assigned to the 

Bottom Long, Bottom Short, and Top Short conditions, and 87 assigned to the Top Long condition. 

 

1

 Those who had participated in the 2012 administration of the SAMHSA DTAC Customer Satisfaction Survey were 

not included in the 2013 administration. 
3 

The Web Survey 

The web survey is administered annually, with minimal changes from year to year. Questions are about customers’ 

familiarity, experiences, and satisfaction with various aspects of SAMHSA DTAC and the services that they 

provide. The survey contained skip patterns such that respondents were only asked specific questions about the 

services or products that they indicated having used. The survey contained approximately 32 questions with 97 

response equivalents; with the skip patterns, nearly all respondents were asked fewer questions with far fewer 

response equivalents. 

Communications 

Respondents were sent a pre-notification, an invitation, and four reminder emails. All contacts were via email. The 

pre-notification was consistent across all four experimental groups. The communication schedule was as follows: 

Table 2. Survey Communication Schedule 

Communication Date Sent Day of the Week 

Pre-notification May 10, 2013 Friday 

Invitation May 14, 2013 Tuesday 

Reminder 1 May 29, 2013 Wednesday 

Reminder 2 June 10, 2013 Monday 

Reminder 3 June 19, 2013 Wednesday 

Reminder 4 June 27, 2013 Thursday 

The last reminder email was sent on Thursday, June 27, 2013 and the survey closed on July 3, 2013. In total, the 

survey was in the field for seven weeks. 

Results 

We first looked at the number of customers who began the survey. Of the 345 invited to participate in the study, 91 

customers or 26.4% began the survey. In looking at response rates, we see that a total of 64 completed the survey 

for an overall response rate of 18.6%. 

Email Length 

Though the difference was not statistically significant, as shown in Figure 1, a larger percentage of those who 

received the invitations with shorter text began taking the survey compared to those who received the longer email. 

Among those who received shorter email communications, 29.7% began taking the survey compared to 23.1% of 

those who received longer email communications. 

Figure 1. Percent who Began Survey by Email Length 

0.0% 

5.0% 

10.0% 

15.0% 

20.0% 

25.0% 

30.0% 

35.0% 

Long Email Short Email 

23.1% 

29.7% 

Difference = 6.6% pts
4 

We found no statistically significant difference in response rate by email length. As shown in Figure 2, among those 

who received shorter email communications, 20.9% completed the survey compared to 16.2% of those who received 

longer email communications, though the difference was not significant. 

Figure 2. Response Rates by Email Length 

0.0% 

5.0% 

10.0% 

15.0% 

20.0% 

25.0% 

30.0% 

35.0% 

Long Email Short Email 

16.2% 

20.9% 

Difference = 4.7% pts 

URL Location 

We also found no statistically significant differences by URL location. As shown in Figure 3, among those who 

received email communications with the survey link toward the top, 29.5% began taking the survey compared to 

23.3% of those who received email communications with the link toward the bottom. Though again the difference 

was not significant. 

Figure 3. Percent who Began Survey by Link Location 

0.0% 

5.0% 

10.0% 

15.0% 

20.0% 

25.0% 

30.0% 

35.0% 

Link at Top Link at Bottom 

29.5% 

23.3% 

Difference = 6.2% pts 

Figure 4 presents response rates by URL location. Again, this difference was not statistically significant. 
5 

Figure 4. Response Rate by Link Location 

0.0% 

5.0% 

10.0% 

15.0% 

20.0% 

25.0% 

30.0% 

35.0% 

Link at Top Link at Bottom 

19.7% 

17.4% 

Difference = 2.3% pts

All Conditions 

Across all four conditions, we hypothesized that those sent the long email might be more likely to begin the survey 

because they only had to agree to participate once. We found no significant difference in survey initiation rate 

across the four groups. Furthermore, the direction of the results did not support our initial hypothesis as those who 

received the long email with the link at the bottom were the least likely to begin the survey. 

Figure 5. Percent Beginning Survey Across All Conditions 

0.0% 

5.0% 

10.0% 

15.0% 

20.0% 

25.0% 

30.0% 

35.0% 

Bottom Long Bottom Short Top Long Top Short 

19.8% 

26.7% 26.4% 

32.6% 

We also hypothesized that those who received the long email would be more likely to complete the survey once 

initiated because they might have had a lower perception of burden after skipping the landing page. The results 

were not statistically significant, and as shown in Figure 6, the direction of the results did not support our 

hypothesis. There was a stronger relationship with URL location than email length. Among those who started the 

survey, those who received the email with the link at the bottom were more likely to complete the survey than those 

who received the email with the link at the top. In other words, there were fewer survey breakoffs in the group who 

received an email with the web survey URL at the bottom compared to those who received an email with the URL 

near the top. 
6 

Figure 6. Among those who Started, Percent Completing Survey 

0.0% 

10.0% 

20.0% 

30.0% 

40.0% 

50.0% 

60.0% 

70.0% 

80.0% 

Bottom Long Bottom Short Top Long Top Short 

76.5% 73.9% 

65.2% 67.9% 

However, the lower survey breakoff rate did not translate into higher response rates among those groups because 

fewer individuals began the survey in the first place. The groups with the highest response rates were those who 

received the short email. Those who received the short email that had the web survey URL toward the top of the 

email had the highest response rate among all four groups; however the difference was not statistically significant. 

Figure 7. Response Rates Across All Conditions 

0.0% 

5.0% 

10.0% 

15.0% 

20.0% 

25.0% 

30.0% 

35.0% 

Bottom Long Bottom Short Top Long Top Short 

15.1% 

19.8% 

17.2% 

22.1% 

Discussion 

Preliminary results show that the design of email communications can affect survey participation and response rates. 

A larger percentage of those who received the short email began and completed the survey, though this difference 

was not statistically significant. The direction of the difference failed to replicate prior research which supported the 

long email and did not support the strategy for including the OMB statement and voluntary consent language on the 

web survey landing page. Further research is needed to investigate the impact of survey length and the possibility of 

moving the OMB and consent language to the email communication and eliminating the survey landing page. 

The direction of the findings on locating web survey URL at top of the email was consistent with our prior research. 

However, it was not consistent with the Kaplowitz et al. 2012 study. We recommend continuing to explore ideal 

web survey URL location. 
7 

There are a number of limitations to this study. Most importantly, the study was conducted on a survey with a small 

sample size. Differences in completion and response rates were not statistically significant, likely in part due to the 

small sample size. Although not statistically significant, the direction of our findings failed to replicate past research 

and confirmation of these findings by conducting an experiment on a study with a larger sample size is warranted. 

In addition, the population of professional customers is unique; results should be replicated with a study of the 

general population in order to better generalize the findings to research conducted on the general population. 

In addition to replicating the study with a larger sample size, we also intend to look at the impact of boosting 

response rates on data quality. While higher response rates may translate into a more representative study, one of 

the consequences of increasing response rates may be attracting less serious respondents with less motivation to 

complete the study. Such respondents may be speedsters, or may be more likely to skip questions or select 

undifferentiated responses when completing a grid question. We intend to explore the relationship between higher 

response rates and data quality. 

 
8 

Appendix A. Email Invitations 

Long Email, Web Survey URL Link at Bottom 

You have been selected to participate in an important web-based survey on your experiences 

with the Substance Abuse and Mental Health Services Administration (SAMHSA) Disaster 

Technical Assistance Center (DTAC).

SAMHSA DTAC supports SAMHSA’s efforts to prepare states, territories, and local entities to 

deliver effective behavioral health (mental health and substance abuse) responses to disasters.

The purpose of this survey is to assess users’ satisfaction with SAMHSA DTAC to ensure high 

quality customer service and improve future services that SAMHSA DTAC may provide. This 

survey takes about 15 minutes to complete.

You have been asked to complete this survey because you have either requested technical 

assistance from SAMHSA DTAC in the past 12 months or you are subscribed to one of 

SAMHSA DTAC’s e-communications (The Bulletin or The Dialogue). The survey asks 

questions about your experiences and satisfaction with SAMHSA DTAC technical assistance, 

the SAMHSA DTAC website, and SAMHSA DTAC e-communications. 

Participation in this survey is completely voluntary. You can decide whether or not to take the 

survey and which questions to answer if you decide to take the survey. If you choose to 

participate in the survey, any information you provide will be treated as confidential and all 

information collected will be reported only at an aggregate level.

To complete the survey, please click on the following link or copy and paste it into your web 

browser:

https://spss.icfsurveys.com/mrIWeb/mrIWeb.dll?I.Project=SAMHSACSS&ID=100005BL

If you have any technical difficulties or questions about the survey, please call the survey 

helpline toll-free at 1-866-657-2798 or send an email to DTACSurvey@icfi.com. 

For any questions about SAMHSA’s research on this topic, please contact Nikki Bellamy, 

Ph.D., SAMHSA DTAC Project Officer, by phone at 240-276-2418 or by email at 

nikki.bellamy@samhsa.hhs.gov.

OMB No. 0930-0325

Expiration Date: 06/30/14

Public Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of 

information unless it displays a currently valid OMB control number. The OMB control number for this project is 0930-xxxx. 

Public reporting burden for this collection of information is estimated to average .25 hours per client per year, including the 

time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing 

and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 

collection of information, including suggestions for reducing this burden, to SAMHSA Reports Clearance Officer, 1 Choke 

Cherry Road, Room 7-1044, Rockville, Maryland, 20857

 
9 

Long Email, Web Survey URL Link at Top 

You have been selected to participate in an important web-based survey on your experiences 

with the Substance Abuse and Mental Health Services Administration (SAMHSA) Disaster 

Technical Assistance Center (DTAC).

Please read the information below. To complete the survey, please click on the following link or 

copy and paste it into your web browser:

https://spss.icfsurveys.com/mrIWeb/mrIWeb.dll?I.Project=SAMHSACSS&ID=100004TL

SAMHSA DTAC supports SAMHSA’s efforts to prepare states, territories, and local entities to 

deliver effective behavioral health (mental health and substance abuse) responses to 

disasters.

The purpose of this survey is to assess users’ satisfaction with SAMHSA DTAC to ensure high 

quality customer service and improve future services that SAMHSA DTAC may provide. This 

survey takes about 15 minutes to complete.

You have been asked to complete this survey because you have either requested technical 

assistance from SAMHSA DTAC in the past 12 months or you are subscribed to one of 

SAMHSA DTAC’s e-communications (The Bulletin or The Dialogue). The survey asks 

questions about your experiences and satisfaction with SAMHSA DTAC technical assistance, 

the SAMHSA DTAC website, and SAMHSA DTAC e-communications. 

Participation in this survey is completely voluntary. You can decide whether or not to take the 

survey and which questions to answer if you decide to take the survey. If you choose to 

participate in the survey, any information you provide will be treated as confidential and all 

information collected will be reported only at an aggregate level.

If you have any technical difficulties or questions about the survey, please call the survey 

helpline toll-free at 1-866-657-2798 or send an email to DTACSurvey@icfi.com. 

For any questions about SAMHSA’s research on this topic, please contact Nikki Bellamy, 

Ph.D., SAMHSA DTAC Project Officer, by phone at 240-276-2418 or by email at 

nikki.bellamy@samhsa.hhs.gov.

OMB No. 0930-0325

Expiration Date: 06/30/14

Public Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of 

information unless it displays a currently valid OMB control number. The OMB control number for this project is 0930-xxxx. 

Public reporting burden for this collection of information is estimated to average .25 hours per client per year, including the 

time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing 

and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 

collection of information, including suggestions for reducing this burden, to SAMHSA Reports Clearance Officer, 1 Choke 

Cherry Road, Room 7-1044, Rockville, Maryland, 20857.

.
10 

Short Email, Web Survey URL Link at Bottom 

You have been selected to participate in an important web-based survey on your 

experiences with the Substance Abuse and Mental Health Services Administration 

(SAMHSA) Disaster Technical Assistance Center (DTAC). SAMHSA DTAC wants to hear 

your thoughts about their services so that they can ensure they provide you with high 

quality customer service and improve their services in the future. 

We’re inviting you to participate in this SAMHSADTAC customer satisfaction survey 

because you have either requested technical assistance from SAMHSA DTAC in the past 

or you are subscribed to one of SAMHSA DTAC’s e-communications (the Bulletin or The 

Dialogue). This survey takes about 15 minutes to complete. 

To complete the survey, please click on the following link or copy and paste it into your web 

browser: 

https://spss.icfsurveys.com/mrIWeb/mrIWeb.dll?I.Project=SAMHSACSS&ID=100005BS

If you have any technical difficulties or questions about the web survey, please call the 

survey helpline toll-free at 1-866-657-2798 or send an email to DTACSurvey@icfi.com. 

For any questions about SAMHSA’s research on this topic, please contact Nikki Bellamy, 

Ph.D., SAMHSA DTAC Project Officer, by phone at 240-276-2418 or by email at 

nikki.bellamy@samhsa.hhs.gov. 

Please complete this important survey today! 

Thank you, 

SAMHSA DTAC 

 
11 

Short Email, Web Survey URL Link at Top 

You have been selected to participate in an important web-based survey on your experiences with 

the Substance Abuse and Mental Health Services Administration (SAMHSA) Disaster Technical 

Assistance Center (DTAC). To complete the survey, please click on the following link or copy and 

paste it into your web browser: 

https://spss.icfsurveys.com/mrIWeb/mrIWeb.dll?I.Project=SAMHSACSS&ID=100004TS

SAMHSA DTAC wants to hear your thoughts about their services so that they can ensure they 

provide you with high quality customer service and improve their services in the future. 

We’re inviting you to participate in this SAMHSADTAC customer satisfaction survey because you 

have either requested technical assistance from SAMHSA DTAC in the past or you are subscribed to 

one of SAMHSA DTAC’s e-communications (the Bulletin or The Dialogue). This survey takes about 

15 minutes to complete. 

If you have any technical difficulties or questions about the web survey, please call the survey 

helpline toll-free at 1-866-657-2798 or send an email to DTACSurvey@icfi.com. 

For any questions about SAMHSA’s research on this topic, please contact Nikki Bellamy, Ph.D., 

SAMHSA DTAC Project Officer, by phone at 240-276-2418 or by email at 

nikki.bellamy@samhsa.hhs.gov. 

Please complete this important survey today! 

Thank you, 

SAMHSA DTAC

 
12 

Appendix B. Web Survey Landing Page and First Survey Question 

Landing Page 
13 

First Survey Question 

 
14 

References: 

Couper, M. P. (2008). Designing effective web surveys. New York, NY: Cambridge University Press. 

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Internet, mail, and mixed-mode surveys: The tailored 

design method. New York, NY: Wiley. 

De Leeuw, E. D., Callegaro, M., Hox, J. J., Korendijk, E., & Lensvelt-Mulders, G. (2007). The influence of advance 

letters on response in telephone surveys: A meta-analysis. Public Opinion Quarterly, 71, 1-31.

Falcone, A.E., Thomas, R.K., Mack, A. (2012). Evaluating the impact of emails and landing page 

on web survey access, Paper presented at the 67th Annual Conference of the American Association for Public 

Opinion Research (AAPOR). Orlando, Florida. 

 

Kaplowitz, M.D, Lupi, F., Couper, M.P., & Thorp, L. (2013). The effect of invitation design on web survey response 

rates, Social Science Computer Review, 30, 339-349.

Office of Management and Budget (2006) Standards and Guidelines For Statistical Surveys, accessed October 15, 

2013, http://www.whitehouse.gov/sites/default/files/omb/assets/omb/inforeg/statpolicy/standards_stat_surveys.pdf