Investigating Internet Opt-in Panels 

for Behavioral Surveillance 

Carol A. Gotway Crawford, PhD 

Chief, Population Health Surveillance Branch

Federal Committee on Statistical Methodology Research Conference

November 4, 2013

Washington, DC

National Center for Chronic Disease Prevention and Health Promotion

Division of Population Health

Catherine Okoro, PhD

Epidemiologist

Satvinder Dhingra, MPH 

Epidemiologist

Northrop Grumman
What Are Internet Opt-in Panels? 

 Potential panelists are recruited via the Internet 

 Banner ads, email lists, promotions, and offers

 Double opt-in process to become a panel member

 Panelists become the pool for sample selection 

 

 Panel may or may not be representative of the 

population 

 Coverage is limited to Internet users (~ 80% of the population) 

 Respondent selection and motivation

 
Why Use Internet Opt-in Panels? 

 Lower cost than probability-based sampling 

 Shorter collection and prep time for data release to 

the public than current methods (RDD, face-to-face) 

 Expands the surveillance and study tool-kit 

 Permits longitudinal and in-depth follow-up studies 

 Increases administrative and design flexibility and 

efficiency 
Pilot Study 

 4 States 

 Cooperative agreements in GA, IL, NY, and TX

 3 Vendors 

 Different sampling methodologies

 Cooperating and collaborating

 De-duplication of respondents

 Nearly identical questionnaire format

 3 Levels of Geography 

 National

 State

 Metropolitan Statistical Area
Pilot Objectives 

 Compare sampling methodologies 

 Sample matching, source blending, and quota

 Assess feasibility and accuracy for public health 

 Compare estimates with those from other 

surveys 

 Evaluate across a range of parameters: 

 Cost, geographic granularity, and timeliness
Sampling Methodologies 

 Sample Matching

 Different modes of recruitment are used to ensure representativeness for 

hard-to-reach populations

 Potential respondents are selected by matching to a random sample from 

the American Community Survey 

 Final responses are weighted to known characteristics in the U.S. using 

propensity score weighting

 Sample Blending

 Uses population segments designed to reflect behavioral differences but 

based on Census data

 Apply the segmentation structure locally to balance, weight, and blend 

sample

 Quota Sampling

 A non-probability sample in which respondents take the survey on a first- come, first-served basis according to a fixed quota
Questionnaire Development 

 Survey consists of ~80 questions (20 minutes)

 Questions drawn from:

• CDC: BRFSS, NHANES, & NHIS

• NIH: PROMIS

• SAMHSA: NSDUH

• ONC: Consumer Survey of Attitudes Toward the Privacy and 

Security Aspects of EHR and HIE 

• NPWF (National Partnership for Women and Families)

• NSF supported Cooperative Congressional Election Study
Benchmarking 
National: Demographics (Unweighted) 

NHIS 2012

HH CAPI

BRFSS 2012

DF-RDD CATI

YouGov 2013

IPS Matched CPS 2012

0.0 20.0 40.0 60.0 80.0

White nH

Black nH

Hispanic

Other

0.0 10.0 20.0 30.0 40.0 50.0 60.0

Male

Female

0.0 10.0 20.0 30.0 40.0

 18 -29

 30 - 44

 45 - 64

 65+

0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0

 <HS

 =HS

 >HS
National: Demographics (Weighted)

NHIS 2012

HH CAPI

BRFSS 2012

DF-RDD CATI

YouGov 2013

IPS Matched

0.0 10.0 20.0 30.0 40.0

 18 -29

 30 - 44

 45 - 64

 65+

0.0 10.0 20.0 30.0 40.0 50.0 60.0

Male

Female

0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0

 <HS

 =HS

 >HS

0.0 20.0 40.0 60.0

White nH

Black nH

Hispanic

Other
State: Age

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

GA

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

IL 

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

NY

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

TX

Mktg Inc. 2013

Blended

NHIS 2011

HH CAPI

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota

BRFSS 2011

DF-RDD CATI
MSA: Age

Mktg Inc. 2013

Blended

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota

SMART BRFSS 2012

DF-RDD CATI

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

Houston

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

New York

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

Chicago

0

10

20

30

40

50

60

18-25 26-34 35-64 >=65

Atlanta 
State: Race/Ethnicity

0

20

40

60

80

White nH Black nH Hispanic Other

TX

0

20

40

60

80

White nH Black nH Hispanic Other

NY

0

20

40

60

80

White nH Black nH Hispanic Other

IL 

Mktg Inc. 2013

Blended

NHIS 2011

HH CAPI

BRFSS 2011

DF-RDD CATI

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota

0

20

40

60

80

White nH Black nH Hispanic Other

GA
MSA: Race/Ethnicity

Mktg Inc. 2013

Blended

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota

SMART BRFSS 2012

DF-RDD CATI

0

10

20

30

40

50

60

70

White nH Black nH Hispanic Other

Atlanta 

0

10

20

30

40

50

60

70

White nH Black nH Hispanic Other

Chicago

0

10

20

30

40

50

60

70

White nH Black nH Hispanic Other

New York

0

10

20

30

40

50

60

70

White nH Black nH Hispanic Other

Houston
State: Education

0

10

20

30

40

50

60

70

 <HS =HS >HS

TX

0

10

20

30

40

50

60

70

 <HS =HS >HS

NY

0

10

20

30

40

50

60

70

 <HS =HS >HS

IL 

0

10

20

30

40

50

60

70

 <HS =HS >HS

GA

Mktg Inc. 2013

Blended

NHIS 2011

HH CAPI

BRFSS 2011

DF-RDD CATI

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota
MSA: Education

Mktg Inc. 2013

Blended

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota

SMART BRFSS 2012

DF-RDD CATI

0

10

20

30

40

50

60

70

 <HS =HS >HS

Houston

0

10

20

30

40

50

60

70

 <HS =HS >HS

New York

0

10

20

30

40

50

60

70

 <HS =HS >HS

Chicago

0

10

20

30

40

50

60

70

 <HS =HS >HS

Atlanta 
National: Outcomes

NHIS 2012

HH CAPI

BRFSS 2012

DF-RDD CATI

YouGov 2013

IPS Matched

0.0 10.0 20.0 30.0 40.0

Obesity (BMI ≥30)

NHANES (MEC) NHANES (SR) NHIS 2012

BRFSS 2012 YouGov 2013

0.0 2.0 4.0 6.0 8.0 10.0 12.0

Diabetes

NHANES (MEC) NHANES (SR) NHIS 2012

BRFSS 2012 YouGov 2013

0.0 20.0 40.0 60.0 80.0 100.0

Ex/VG/GD

Fair/Poor

NHIS 2012 BRFSS 2012 YouGov 2013
State: Obesity (BMI ≥30)

0

10

20

30

40

50

Yes

TX

0

10

20

30

40

50

Yes

NY

0

10

20

30

40

50

Yes

IL 

0

10

20

30

40

50

Yes

GA

Mktg Inc. 2013

Blended

NHIS 2011

HH CAPI

BRFSS 2011

DF-RDD CATI

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota
MSA: Obesity (BMI ≥30)

Mktg Inc. 2013

Blended

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota

SMART BRFSS 2012

DF-RDD CATI

0

10

20

30

40

50

Atlanta 

0

10

20

30

40

50

Chicago

0

10

20

30

40

50

New York

0

10

20

30

40

50

Houston
State: Diabetes

0

5

10

15

Yes

TX

0

5

10

15

Yes

NY

0

5

10

15

Yes

IL 

0

5

10

15

Yes

GA

Mktg Inc. 2013

Blended

NHIS 2011

HH CAPI

BRFSS 2011

DF-RDD CATI

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota
MSA: Diabetes

Mktg Inc. 2013

Blended

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota

SMART BRFSS 2012

DF-RDD CATI

-2

3

8

13

18

Houston

-2

3

8

13

18

New York

-2

3

8

13

18

Chicago

-2

3

8

13

18

Atlanta 
State: Disability

0

10

20

30

40

Activity or Eqpt

TX

0

10

20

30

40

Activity or Eqpt

NY

0

10

20

30

40

Activity or Eqpt

IL 

0

10

20

30

40

Activity or Eqpt

GA

Mktg Inc. 2013

Blended

NHIS 2011

HH CAPI

BRFSS 2013 

Preliminary

DF-RDD CATI

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota
MSA: Disability

0

5

10

15

20

25

30

35

Activity or Eqpt

Houston

0

5

10

15

20

25

30

35

Activity or Eqpt

New York

0

5

10

15

20

25

30

35

Activity or Eqpt

Chicago

0

5

10

15

20

25

30

35

Activity or Eqpt

Atlanta 

Mktg Inc. 2013

Blended

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota

SMART BRFSS 2012

DF-RDD CATI
National: Health Care Access, 

Utilization, Behaviors & Outcomes

NHIS 2012

HH CAPI

BRFSS 2012

DF-RDD CATI

YouGov 2013

IPS Matched

0.0 20.0 40.0 60.0 80.0

 No Pap

No Mam

No Colorectal

No HIV Test

No Flu

0.0 10.0 20.0 30.0 40.0 50.0

No Insurance

No Usual Source

Cost Barrier

No Past year Checkup

0.0 5.0 10.0 15.0 20.0 25.0 30.0

Current Smoker

Former Smoker

Heavy Drinker

0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0

Diabetes

Hypertension

Any Cancer

Asthma

Arthritis

CHD
State: Health Insurance

0

5

10

15

20

25

30

35

No

TX

0

5

10

15

20

25

30

35

No

NY

0

5

10

15

20

25

30

35

No

IL 

0

5

10

15

20

25

30

35

No

GA

Mktg Inc. 2013

Blended

NHIS 2011

HH CAPI

BRFSS 2011

DF-RDD CATI

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota
State: Primary Care Provider

0

5

10

15

20

25

30

35

No

TX

0

5

10

15

20

25

30

35

No

NY

0

5

10

15

20

25

30

35

No

IL 

0

5

10

15

20

25

30

35

No

GA

Mktg Inc. 2013

Blended

NHIS 2011

HH CAPI

BRFSS 2011

DF-RDD CATI

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota
State: Cost Barrier

0

10

20

30

40

Yes

IL 

0

10

20

30

40

Yes

GA

Mktg Inc. 2013

Blended

NHIS 2011

HH CAPI

BRFSS 2011

DF-RDD CATI

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota

0

10

20

30

40

Yes

TX

0

10

20

30

40

Yes

NY
State: Current Smoker

0

10

20

30

40

100 + Ev/Some

TX

0

10

20

30

40

100 + Ev/Some

NY

0

10

20

30

40

100 + Ev/Some

IL 

0

10

20

30

40

100 + Ev/Some

GA

Mktg Inc. 2013

Blended

NHIS 2011

HH CAPI

BRFSS 2011

DF-RDD CATI

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota
State: Heavy Drinker

0

5

10

15

20

Yes

TX

0

5

10

15

20

Yes

NY

0

5

10

15

20

Yes

IL 

0

5

10

15

20

Yes

GA

Mktg Inc. 2013

Blended

NHIS 2011

HH CAPI

BRFSS 2011

DF-RDD CATI

YouGov 2013

IPS Matched

uSamp 2013

IPS Quota
Quantifying Uncertainty 

 The use of Frequentist confidence intervals with data 

from a non-probability sample is theoretically 

inappropriate

 Bayesian credible intervals are a more appropriate 

way to quantify uncertainty when analyzing data 

from a non-probability sample

 In our pilot studies, however, both methods yielded 

highly similar, if not identical, results
Uncertainty Comparison 

Variable

Confidence 

Interval

Credible 

Interval

Obesity 29.22 32.61 29.12 32.56

Diabetes 9.88 11.95 9.85 11.94

High BP 26.64 29.62 26.56 29.58
Major Benefits 

• Time (samples constructed to be representative):

< 15 days for a national survey ~ 4,000 interviews

~ 30 days for most states ~3,000 interviews

~ 30 days for large (5+ million) MSAs ~2,000 interviews

• Cost: 

– Internet opt-in panels: $5-$15 per completed interview

• Costs include editing and weighting 

– Dual-frame RDD State direct costs average ~$70/CI

• Considerable additional costs for editing and weighting
Preliminary Results 

 • Great deal of similarity 

– Results of sample matching comparable with BRFSS and NHIS 

– Variation among surveys consistent across states

– Internet opt-in panels fairly accurate at lower levels of geography 

– Quota sampling not as accurate

• Differences can be attributed to: 

– Coverage effects (sample selection*outcome interaction)

– Use of different control totals and weighting methods

– Mode effects (face-to-face, telephone, Internet)

– Question differences and order effects

– Temporal changes (2013 vs. 2011)

– Sample size differences

– Cross-sectional differences

 
Acknowledgements 

 Stephen Ansolabehere, Harvard University & CCES

 Steven Gittelman, Mktg. Inc. 

 Douglas Rivers, Stanford University & YouGov

 Meena Khare, NCHS

 Georgia, Illinois, New York, and Texas Departments of Health, BRFSS

 RanaBayakly, Madhavi Vajani, Francis Annor, GA

 Bruce Steiner, IL

 Bethany Hawke, NY

 Rebecca Wood, TX

 Haci Akcin, CDC

 Derek Ford, CDC

 Guixiang Zhao, CDC

 Soyoun Park, CDC