icf.com

Do Fences Really Make Good Neighbors? 

A Side-by-Side Comparison of RDD and Geofencing

Methods Using Risk Factor Surveys

Matt Jans, PhD & James Dayton, MBA – ICF

Scott Worthge – Mfour

July 17, 2018

Consumer Expenditure Survey Methods Symposium 

Bureau of Labor Statistics | Washington, DC
ICF proprietary and confidential. Do not copy, distribute, or disclose.

New Sampling and Measurement 

Opportunities

2
ICF proprietary and confidential. Do not copy, distribute, or disclose.

Challenges with Traditional Survey Methods

Measurement

• People must notice and encode events to report them later

• Recall of behaviors and events is imperfect; worse with time

• Intentions to act are less predictive than actual behavior

Sampling

• Must screen many households to identify eligible respondents

– Particularly for rare characteristics or infrequent activities

• Expensive

• Time consuming 

3
ICF proprietary and confidential. Do not copy, distribute, or disclose.

Research Motivation

Measurement

Improve accuracy of health measures by asking respondents about their 

experiences in situ (e.g., at the doctor’s office or at the gym).

Sampling

Know when respondents might be likely to engage in a particular behavior and 

ask them to report in that moment. 

New Source for Public Health Estimates

Can the resulting data produce statistical estimates that are comparable to 

traditional survey methods? 

4
ICF proprietary and confidential. Do not copy, distribute, or disclose.

Our Methodology

5
ICF proprietary and confidential. Do not copy, distribute, or disclose.

Panel

MFour's Surveys On The Go® Opt-in Panel (www.mfour.com) 

• Largest mobile panel in US (2M active members)

• Non-Probability selection, participation largely spread by word-of-mouth

• No advertising, no internet pop-up ads

• Completely owned/managed by MFour

• Limited overlap with other online opt-in panels

• Specializes in: 

– Point-of-Emotion® measurement 

– Diary studies; in-home measurement; and ad, entertainment, and 

behavior trackers

6
ICF proprietary and confidential. Do not copy, distribute, or disclose. 7

So what does it look like?
ICF proprietary and confidential. Do not copy, distribute, or disclose.

Smartphone Panel Interface

8

Panel Member Dashboard A Single Question
ICF proprietary and confidential. Do not copy, distribute, or disclose. 9

But is it representative?
ICF proprietary and confidential. Do not copy, distribute, or disclose. 10

MFour Panel v. US General Population

MFour US

60%

50%

40%

30%

20%

10%

0%

Sex

(Female)

50+

Years

Hispanic/

Latino

Education

(% College +)

Black 35–49

Years

25–34

Years

18–24 

Years

≤17 Years

Race/Ethnicity Age

53% 51%

21%

18%

14% 13%

35%

48%

41%

27%

30%

23%

10%

31%

14%

9%

20% 19%

8%

31%

Marital Status

(% Married) 
ICF proprietary and confidential. Do not copy, distribute, or disclose. 11

MFour Panel v. US General Population

MFour US

60%

50%

40%

30%

20%

10%

0%

Sex

(Female)

50+

Years

Hispanic/

Latino

Education

(% College +)

Black 35–49

Years

25–34

Years

18–24 

Years

≤17 Years

Race/Ethnicity Age

53% 51%

21%

18%

14% 13%

35%

48%

41%

27%

30%

23%

10%

31%

14%

9%

20% 19%

8%

31%

Marital Status

(% Married) 

-

+ +

-

?
ICF proprietary and confidential. Do not copy, distribute, or disclose. 12

Our Study
ICF proprietary and confidential. Do not copy, distribute, or disclose.

Questionnaire

Mode

• Smartphone-based app

Topics

• BRFSS and other health-related questions (for benchmarking)

– Alcohol use

– Smoking 

– Sugar-sweetened beverage consumption

Nontraditional Survey Tasks

• Image capture of in-store tobacco, alcohol, or sugar-sweetened beverage display 

(for respondents completing the survey while still in store)

13
ICF proprietary and confidential. Do not copy, distribute, or disclose.

Sampling and Contact Specifications

• Geofenced grocery, liquor, and convenience stores nation-wide

• Invited when cross geofence 

– Cash register “cha-ching” and visual notification

• Reminders at 1, 24, and 36 hours

14
ICF proprietary and confidential. Do not copy, distribute, or disclose.

Analysis Plan

Sample Size

• Planning for 1,500 completes

Benchmarking

• BRFSS (CATI/RDD)

• Similar health behavior data 

– Web and mail 

– Nonprobability web panels 

15
ICF proprietary and confidential. Do not copy, distribute, or disclose.

Future Research

Test using probability sample

Utilize time-lapse geo-location tracking of panel members

• Geo-Tracking all panel members who grant permission 

– Targeting respondents who:

– Visit fast food restaurants at least x times/week

– Visit hospital or health care provider weekly

16
ICF proprietary and confidential. Do not copy, distribute, or disclose. 17

Matt Jans: matt.jans@icf.com

301-572-0821

Jamie Dayton: james.dayton@icf.com

802-264-3723