Innovative Retention 

Methods in Panel Research: 

Can Smartphones Improve Long-term Panel Participation? 

November 6, 2013 

Andrew Dyer 

ICF International 

icfi.com | 
 

 

Agenda 

 Panel Management for Longitudinal Research 

 Panel Retention 

− Traditional Tactics 

− Alternative Tactics 

 ICF Panel Management 

− Quantitative Experiment 1 

− Quantitative Experiment 2 

− Next Steps 

 Conclusions 

icfi.com | 2
 

 

Panel Retention 

 Potential damage from attrition 

− Can bias sample 

− Can reduce statistical power 

− Can be costly to replace panelists 

 Traditional tactics 

− Managing contact information 

− Good rapport 

− Participant-oriented actions2 

• Calls, mailings, visits 

2http://www.wpic.pitt.edu/research/famhist/PDF_Articles/Springer/RI%206.pdf 

icfi.com | 3
 

 

Panel Retention 

 56% of American adults own a smartphone3 

−	 Smartphones used to perform activities previously reserved for 

PCs or tablets 

 In evolving landscape...

− How do we best facilitate access and engagement? 

 New/alternative tactics 

− Electronic information searches to locate “lost” panelists

− Smartphone engagement to avoid losing them in the first place 

•	 Native app 

•	 Mobile-optimized online 

•	 QR code 

3http://www.pewinternet.org/~/media/Files/Reports/2013/PIP_Smartphone_adoption_2013.pdf 

icfi.com | 4
 

Quantitative Experiment 1 

Smartphone App 

 Panel of recreational boat owners 

− Q: Would a smartphone app support panel retention? 

 Which panelists own a smartphone? 

icfi.com | 5
Quantitative Experiment 1 

Who are these Panelists? 

 Recreational boat owners recruited from a dual-frame, 

dual-mode study 

− National RDD, CATI 

− State lists of registered boat owners, Mail 

 Approximately 24,000 people—all states, D.C. and Puerto 

Rico 

 Range of boat types 

 Report boating activities periodically throughout year 

 $10 Amazon.com gift code for each interview 

icfi.com | 6
icfi.com | Passion. Expertise. Results. 7 
 

Quantitative Experiment 1 

Sample Selection 

MAIL 

EMAIL 

SMS 

Selected Sample 

N = 2,412 

icfi.com | 8
 

Quantitative Experiment 1 

Sample by Experimental Mode of Contact 

SMS, 804 

EMAIL, 804 

MAIL, 804 

N = 2,412 

icfi.com | 9
Quantitative Experiment 1 

 We asked panelists...

− Do you own a smartphone? 

− Would you download a smartphone panel app? 

− How do you prefer to be contacted in the future? 

 Mini-survey identified effect of contact mode on...

− Response rate 

− Willingness to download smartphone panel app 

− Preferred mode of future contact 

 ...and we learned who owns a smartphone!

icfi.com | 10
 

 

Quantitative Experiment 1 

Smartphone Ownership 

0% 

20% 

40% 

60% 

80% 

100% 

69.1% 

Own SP 

Don’t Own SP

OVERALL 

n = 561 

icfi.com | 11
 

 

 

 

Willing to Download Panel App 

(among SP Owners) 

Quantitative Experiment 1 

OVERALL SMS EMAIL MAIL 

56.0% 62.6% 47.4% 

χ2(2) = 11.80, p = .003 

Yes 

No 

n = 386 n = 91 n = 194 n = 101 

66.3% 

Experimental Mode of Contact 100% 

80% 

60% 

40% 

20% 

0% 

icfi.com | 12
 

 

 

Quantitative Experiment 1 

Preferred Mode of Future Panel Contact 

0% 

20% 

40% 

60% 

80% 

100% 

OVERALL SMS EMAIL MAIL 

Mail 

Phone 

Email 

SMS 

Experimental Mode of Contact 

χ2(6) = 47.83, p < .001 

n = 546 n = 94 n = 288 n = 164 

icfi.com | 13
 

 

 

 

Quantitative Experiment 1 

Response Rate 

23.3% 12.7% 36.6% 20.5%

Non-Response 

Response 

Experimental Mode of Contact 100% 

80% 

60% 

40% 

20% 

0% 

OVERALL SMS EMAIL MAIL 

n = 2,412 n = 804 n = 804 n = 804 

χ2(2) = 133.50, p < .001 

icfi.com | 14
 

 

 

Quantitative Experiment 1 

Mode of Web Survey Access (following SMS or EMAIL contact) 

100% 

80% 

60% 

40% 

20% 

0% 30.5% 76.0% 

67.4% 14.6% 

SMS EMAIL 

n = 95 n = 288 

χ2(2) = 99.76, p < .001 

Smartphone 

Tablet 

PC 

icfi.com | 15
 

 

 

 

 

Provided Current Contact Information 

(Address, Phone, or Email) 

Quantitative Experiment 1 

OVERALL PC TABLET SP 

84.3% 83.9% 96.6% 

χ2(2) = 3.73, p = .155 

Yes 

No 

n = 383 n = 248 n = 29 n = 106 

82.1% 

Mode of Access to Web Survey 100% 

80% 

60% 

40% 

20% 

0% 

icfi.com | 16
Quantitative Experiment 1 

Takeaways 

 Email contact produced highest response rate 

 Many panelists willing to download app 

 Contact mode influenced future contact mode preference 

 Smartphones not a barrier to update contact information 

icfi.com | 17
 

 

Quantitative Experiment 2 

 Panelists with a mailing address 

Group Sample Treatment 

Control 7,163 Postcard + URL 

Experiment 7,163 Postcard + URL + QR code 

mysurvey.com 

Q: How does the QR code impact response rates? 

icfi.com | 18
 

 

 

 

Quantitative Experiment 2 

Responses – Web Survey 

250 

200 

150 

100 

50 

0 

167 

89 78 

56 

56 

OVERALL CONTROL 

(Mail+URL) 

EXPER. 

(Mail+URL+QR) 

By Treatment Group 

Web Survey via 

QR Code 

Web Survey 

via URL 

icfi.com | 19
 

 

 

Quantitative Experiment 2 

Completion Rate – Web Survey 

300 

250 

200 

150 

100 

50 

0 

223 

167 

56 

31 

4 

27 

OVERALL URL QR 

By Web Survey 

Response Channel 

88% 98% 67% 

χ2(1) = 47.53, p < .0001 

Incomplete 

Complete 

icfi.com | 20
 

 

 

Quantitative Experiment 2 

Breakoff Point – Web Survey 

4 

0

19 

5 

0 2 0 1

5 

10 

15 

20 

25 

Respondents

Web Survey 

via QR Code 

Web Survey 

via URL 

INTRO Q1 Q2 Q3 Q4 

Breakoff Point 

icfi.com | 21
 

Quantitative Experiment 2 

Takeaways 

Too early to say but...

 QR presence appears to increase overall response 

− Respondents want the “easy button” 

 Challenge to engage smartfone users to complete 

icfi.com | 22
 

 

 

 

Next Steps 

Quant 3 

“App available

for DL!”

DL Starts 

DL Completes 

Notification DL Behavior 

...Experiment 3

Panelists 

with App 

 Quantitative Research Question 

−	 Does intention to download an app predict actual behavior? 

(correlational) 

icfi.com | 23
 

 

 

Quantitative Experiment 3 

Next Steps 

Sample

(App+ Email + Mail)

Smartphone 

App 

Attrition 

Response % 

Email Attrition 

Response % 

Mail Attrition 

Response % 

Experiment 3 

Mode of Retention 

Communications 

Retention 

 Experiment 3 Questions 

− Does mode of contact influence retention? (experimental) 

− Does mode of contact influence survey response rates? (experimental) 

icfi.com | 24
 

 

 

 

 

Conclusions 

 Apps can provide value under the right conditions 

− Large ongoing panel to be cost effective 

• Higher investment and maintenance cost 

− App store gatekeepers 

− End user acceptance 

 Mobile optimized online option 

− Easier/cheaper development vs. app 

− Requires internet connection 

− Slower page loads could trigger break-offs/attrition 

 QR code 

− Smartphone accessibility via print 

 Smartphones more commonplace 

− Researchers must account for this reality 

icfi.com | 25
 

 

Questions? 

Andrew Dyer 

andrew.dyer@icfi.com 

James Dayton 

james.dayton@icfi.com 

icfi.com | 26