A Meta-Analysis of the Impact of Number of Contact Attempts on Response Rates and Web Completion Rates in Multimode Surveys 

Ting Yan

Westat

November 2, 2021

1

Multimode surveys

Multimode surveys are on the rise

Using multiple modes to contact and to collect data from potential respondents

Benefits (de Leeuw 2018; Tourangeau 2017):

Improving coverage

Increasing response rates

Reducing cost

Improving measurement

COVID-19 

2

This talk

Focuses on multimode surveys using mail and web to collect data from potential respondents

Using a meta-analytic approach to quantifying two benefits (de Leeuw 2018; Tourangeau 2017):

Improving coverage

Increasing response rates

Impact of design features on response rate

Reducing cost

Impact of design features on percent of web completes

Improving measurement

3

Types of web and mail multimode designs

Concurrent

Web and mail are offered to respondents at the same time

HINTS 5 Web Pilot (Westat, 2019): Web option condition

1st contact: web login + paper questionnaire

2nd contact: Postcard reminder

3rd contact: letter reminder with web login + paper questionnaire

4th contact: letter reminder with web login + paper questionnaire

4

Types of web and mail multimode designs

Sequential mail and web

Paper questionnaire is offered first and web is offered later

Lewiston and Clarkston Quality of Life Survey (Smyth et al. 2010)

1st contact: pre-notification letter

2nd contact: paper questionnaire

3rd contact: postcard reminder

4th contact: paper questionnaire + web login

5

Types of web and mail multimode designs

Sequential web and mail

Web is offered first and paper questionnaire offered later

2015 VA Survey of Veteran Enrollee’s Health and Use of Health Care (Westat, 2015): Hard-to-interview control condition

1st contact: pre-notification letter

2nd contact: invitation letter with web login

3rd contact: IVR reminder

4th contact: letter reminder with paper questionnaire

5th contact: Postcard reminder

6th contact: letter reminder with paper questionnaire

6

Impact of design features of multimode surveys

Type of design

Number of contacts

Was a pre-notification letter sent?

Was incentive offered? 

Was email used to reach sampled persons?

Timing of the second mode 

7

Meta-analysis

Searched for experimental studies that varied design features

Key words: multimode, mixed mode, multiple modes of data collection, web, mail, paper, sequential, concurrent, choice, push-to-web

Reviewed studies for eligibility

Has to be empirical studies

Has to include experiments

Has to include sufficient information on design features, sample size, response rate, number of web completes or percent of web completes

Retained 29 papers for coding

38 studies

113 estimates

8

Meta-analysis (2)

For each eligible study, 

Coded design features (e.g., type of multimode design etc)

Entered or manually calculated

Response rate

Percent of interviews completed on web

Calculated effect size (and variance) 

Logit transformation of response rate and web completion rate 

9

Meta-analysis (3)

Average effect size by type of multimode designs

Moderator analysis

design features

10

Results: Response rates by type of multimode designs

Concurrent mail/web designs:

28 effect sizes nested in 17 studies nested in 14 papers

Sequential mail/web:

15 effect sizes nested in 11 studies nested in 10 papers 

Sequential web/mail:

70 effect sizes nested in 34 studies nested in 26 papers

11

Results: Response rates by number of contacts

Ranging from 2 to 17

Mode number of contacts

Concurrent: 5

Mail/web: 4

Web/Mail: 4

Overall effect

0.12 (p<.0001)

12

Results: Response rates by pre-notification letter

% sent a pre-notification letter:

Concurrent: 61%

Mail/web: 73%

Web/Mail: 57%

Overall effect

0.29 (p<.05)

13

Difference 6.6% 8.2% 7.3% 

13

Results: Response rates by incentive

% offered incentive

Concurrent: 64%

Mail/web: 73%

Web/Mail: 44%

Overall effect

0.41 (p=.002)

14

Difference 6.6% 8.2% 7.3% 

14

Results: Response rates by use of email

Email use

Concurrent: 1 study used email reminder

Mail/web: 1 study sent web survey via email

Web/Mail:

7%: email invitation

14%: email reminder

11%: both

66%: did not use email

15

15

Results: Response rates by timing of second mode

16

Most studies offered 2nd mode 2 contacts after 1st mode

Summary of findings on Response Rate

17
| Design Feature        | Impact on Response Rate                                                                                                          |
|-----------------------|----------------------------------------------------------------------------------------------------------------------------------|
| Type of design        | mail/web > (concurrent = web/mail)                                                                                            |
| Number of contacts    |                                                                                                                                  |
| Pre-notification      |                                                                                                                                  |
| Incentive             | concurrent and web/mail: mail/web:                                                                                               |
| Email (web/mail only) | Email reminder:                                                                                                                  |
| Timing of 2nd mode    | mail/web: higher RR when offering 2nd mode 1 or 2 contacts laterweb/mail: higher RR when offering 2nd mode 2 or 3 contacts later |

Results: Web completion rates by type of design

18

Results: Web completion rates by number of contacts

19

Results: Web completion rates by Pre-notification

20

Results: Web completion rates by Incentive

21

Results: Web completion rates by use of email (Sequential web/mail only)

22

Results: Web completion rates by timing of second mode

23

Summary of findings on web completion rate

24
| Design Feature        | Impact on web completion rate                                                                                                 |
|-----------------------|-------------------------------------------------------------------------------------------------------------------------------|
| Type of design        | web/mail > concurrent > mail/web                                                                                        |
| Number of contacts    | concurrent and web/mail:                                                                                                      |
| Pre-notification      | concurrent and mail/web: web/mail:                                                                                            |
| Incentive             | concurrent and web/mail:  mail/web:                                                                                           |
| Email (web/mail only) | Email invitation+ email reminder:                                                                                             |
| Timing of 2nd mode    | mail/web: higher WCR when offering 2nd mode 1 contact laterweb/mail: higher WCR when offering 2nd mode 2 or 4+ contacts later |

Recommended sequence

If goal is to maximize response rate

Concurrent: 6 contacts, pre-notification, incentive, 

Mail/web: 4 contacts, pre-notification, no incentive, offering web at 3rd contact

Web/mail:7+ contacts, pre-notification, incentive, offering mail at 4th contact, email reminder

If goal is maximize web completion rate

Concurrent: 6 contacts, no pre-notification, incentive

Mail/web: 3 contacts, no pre-notification, no incentive, offering web at 2nd contact

Web/mail: 6 contacts, pre-notification, incentive, offering mail at 6th contact, email invitation+email reminder 

25

Limitations and Future Research

Only examined a few selected moderators

Future research should examine other moderators such as target population, type of sampling frame, type and amount of incentives, survey topic

Only examined one moderator at a time

Future research could employ a meta-regression approach to examine the impact of all moderators at the same time

Limited to studies involving an experiment

Future research could expand to include empirical studies that do not include an experiment

26

Thank You

tingyan@Westat.com

*A big thank-you to Alexis Kokoska and Winnie Xu for their help with coding

27

PHOTOGRAPHY LEGAL DISCLAIMER – DO NOT DELETE THIS SLIDE IF PRESENTATION INCLUDES PEOPLE IMAGES

# image2.png

# image3.png

# image7.png

# image80.png

# image9.png

# image8.png

# image10.png

# image11.png

# image12.png

# image13.png

# image14.png

# image15.png

# image16.png

# image17.png

# image18.png

# image19.png

# image20.png

# image4.png

# image5.png

# image1.png

# image6.png