© 2021 Ipsos 1

Who’s Left Out?: Noミrespoミse Bias Assessマeミt for aミ

Online Probability-based Panel Recruitment 

Frances M. Barlas and Randall K. Thomas

Ipsos Public Affairs

2021 FCSM Research & Policy Conference

November 2nd – November 4th, 2021
© 2021 Ipsos 3

• For two decades now, probability-based online panels have been 

relied upon as credible sampling frames with numerous studies 

showing comparable quality to traditional methodologies like 

telephone studies (MacInnis et al., 2019; Yeager et al. 2011) 

• Like all surveys, response rates to the panel recruitment invitation 

has been declining over recent years

• Although the Total Survey Error framework points to nonresponse as 

one of many possible threats to data quality and studies have called 

into question the connection between response rates and bias, they 

are still a commonly reported metric 

Study Background
© 2021 Ipsos 4

KnowledgePanel Recruitment Methodology

Mail with Pre-paid 

envelope

Toll-free number

join.knpanel.com

Responders

Have phone number:

Outbound call

No phone number:

No further action

• Recruitment is primarily through Address-Based Sampling (ABS)

• Follow-up efforts and incentives used to maximize response

Mail Sample Recruit 

cover letter, brochure, business reply 

envelope, mail acceptance form, cash 

incentive ($2) Non-Responders

Reminder Letter

Reminder Postcard
© 2021 Ipsos 5

Study Design

• To investigate the possibility of nonresponse bias at the panel 

recruitment stage, we conducted two analyses:

o Frame analysis – comparisons of recruited and nonrecruited 

households based on the available ancillary data appended to 

the Delivery Sequence File (DSF) used for recruitment sampling.

o NRFU – follow-up survey of a random sample of nonrecruited

households and comparisons of their responses against those 

who have been recruited.
© 2021 Ipsos 6

Study Design – Frame Analysis

• We investigated one wave of KnowledgePanel recruitment 

• Recruitment between May and September of 2019

• Sample vendor was MSG who worked with data compiler, Neustar, to 

append a long list of ancillary data items to each address:

Total Sample Size Recruits Recruitment Rate

40,000 2,019 5%

1. Activity Date

2. Age Category

3. Birth Date

4. Business Owner

5. County Size Code

6. Children: Age 0-3

7. Children: Age 10-12

8. Children: Age 13-15

9. Children: Age 16-18

10. Children: Age 4-6

11. Children: Age 7-9

12. CBSA

13. Country of Origin

14. Dwelling Type

15. Dwelling Unit Size

16. Home Value

17. Household Income

18. Ethnic Group

19. Gender

20. Home Business

21. Home/Business

22. Homeownership

23. Household Type

24. Residence Years

25. Marital Status

26. Number of Adults 

27. Number of Children

28. Number of Persons

29. Occupation Group

30. Person Type

31. Credit Card

32. Property indicator
© 2021 Ipsos 7

NRFU Frame Analysis - Results

• Across the list of 32 variables, we saw very few differences between 

recruits and non-recruits based on ABS frame data

5.8

35.3

22.4

14.5

10.5 11.6

4.5

35.7

23.7

13.3 10.7 12.2

None One Two Three Four Five+

Number of Adults in HH

Nonrecruited Recruited

11.0 11.0 11.2

8.2

10.3

8.6

14.6

13.1 12.9

9.6 10.3

8.7

0 to 3 4 to 6 7 to 9 10 to 12 13 to 15 16 to 18

Presence of Children by Age

Nonrecruited Recruited
© 2021 Ipsos 8

NRFU Frame Analysis - Results

• Across the list of 32 variables, we saw very few differences between 

recruits and non-recruits based on ABS frame data

11.2

9.3 9.9

12.0

17.2

12.1

7.8

4.9

3.0 1.6 2.2 3.0

5.8

14.2

9.5 10.4 12.0

18.6

10.8

7.7

4.7

2.2 1.4 1.6 2.5

4.5

$1<$15 $15<$25 $25<$35 $35<$50 $50<$75 $75<$100 $100<$125 $125<$150 $150<$175 $175<$200 $200<$250 $250+ Unknown

Estimated HH Income

Nonrecruited Recruited

77.4

5.0 5.9

11.7

76.0

5.3 6.7

12.1

Single Family Up to 4 Units 5 to 19 Units 20+ Units

Dwelling Type/Size

Nonrecruited Recruited
© 2021 Ipsos 9

NRFU Frame Analysis - Results

• Across the list of 32 variables, we saw very few differences between 

recruits and non-recruits based on ABS frame data

9.3

26.4

11.1

17.3

34.2

1.8

9.2

26.1

12.2

16.6

34.1

1.8

0 to 24 25 to 34 35 to 44 45 to 54 55+ Unknown

Estimated Age of Head of Household

Nonrecruited Recruited

42.8 43.6

13.6

45.3

40.9

13.8

Male Female Unknown

Head of Household Inferred Gender

Nonrecruited Recruited
© 2021 Ipsos 10

NRFU Frame Analysis - Results

• Across the list of 32 variables, we saw very few differences between 

recruits and non-recruits based on ABS frame data

8.3

22.8

43.8

25.1

10.5

18.2

45.5

25.8

African American Hispanic Others Unknown

Inferred Race/Ethnicity of Head of Household

Nonrecruited Recruited

White/Other

17.5

27.6 25.6

14.2 11.3

3.9

15.4

30.4

26.5

14.7

10.0

3.1

No High School High School Some College Bachelor Masters+ Unknown

Inferred Level of Education of Head of Household

Nonrecruited Recruited
© 2021 Ipsos 11

Study Design – NRFU

• We selected one wave of KnowledgePanel recruitment for NRFU

• Recruitment in Fall of 2019

• For the NRFU sample:

o All 997 recruits were sampled with certainty

o A subsample of 3,300 nonrespondents was selected in

anticipation of a lower completion rate

Total Sample Size Recruits Recruitment Rate

17,500 997 6%
© 2021 Ipsos 12

Study Design – NRFU

• For the NRFU, we developed a separate brand from 

KnowledgePanel to avoid possibility of connection with 

the KP recruitment effort.

Mailed Invitation 

with Survey Packet

$10

Postcard with 

Web URL 

Mailed Survey 

Packet

Postcard with 

Web URL

Promise

of 

$20
© 2021 Ipsos 13

Study Design – NRFU

• With multi-person HH, adult with most recent birthday was asked to 

complete

• Questionnaire was designed for NRFU with a mix of demographics, 

attitudes and behaviors – relatively short with 44 questions. 

• Fielding was January 31 to April 15, 2020 with samples divided into 3 

replicates
© 2021 Ipsos 14

Results – NRFU Completes

• Study design was successful with 26% completion rate from 

nonrespondents and 68% from those who responded to the KP 

recruitment invitation

KP Recruitment 

Status

Invited to 

NRFU

Completed

NRFU

Completion 

Rate

Mail 

Response

Online 

Response

Nonrespondent 3,300 866 26% 92% 8%

KP Recruit 997 673 68% 94% 6%
© 2021 Ipsos 15

NRFU – Survey follow-up

• We saw very few differences between recruits and non-recruits

46

54

9

24 27

40

62

9

12

18

35

65

10

25 29 35

62

11 8

18

Male Female 18 to 29 30 to 44 45 to 59 60+ White Black Other,2+ Hispanic

*Gender Age Race/Ethnicity

Nonrecruited Recruited

9

22

34 35

21 20 17 13 15 13

5

20

37 37

22 23

18

12 14 11

Less than HS High School Some college College plus Under $25k $25k-$50k $50k-$75k $75k-$100k $100k-$150k $150k or more

Education Income

Nonrecruited Recruited

*p<0.05
© 2021 Ipsos 16

NRFU – Survey follow-up

• We saw very few differences between recruits and non-recruits

*p<0.05

49 51 53 47

97

3

42

58 49 51

97

3

Full-time Not full-time Married Not married Citizen Non-citizen

*Employment Marital status Citizenship

Nonrecruited Recruited

16 21

39

24 22

78 71

28

15

24

38

23 22

78

69

30

Northeast Midwest South West Rural Urban Owner Renter

Census Region Urbanicity Homeownership

Nonrecruited Recruited
© 2021 Ipsos 17

NRFU – Survey follow-up

• We saw very few differences between recruits and non-recruits

*p<0.05

25

37 39 33

67

9

65

25 26 32

44 36

65

11

63

25

One Two Three plus Yes No Studio or One 2 or 3 4 plus

HH Size Kids in HH Bedrooms at home

Nonrecruited Recruited

8

28 37

27

8

92

64

36

8

31 37

25

9

91

60

40

None One Two Three plus No Yes No Yes

Vehicles Driver's license Landline at home

Nonrecruited Recruited
© 2021 Ipsos 18

NRFU – Survey follow-up

• We saw very few differences between recruits and non-recruits

*p<0.05

26 22 18

34

23 26 23 17 11

64

36

21 22 18

40

12

27 26 21 15

59

42

1 hr or less 2 hrs 3 hrs 4 hrs plus 1 hr or less 2 to 5 hrs 6 to 10 hrs 11 to 20 hrs 21 hrs plus Never Sometimes

TV viewing per day *Internet in typical week *Politics opinions online

Nonrecruited Recruited

40

60

28

72

21

80

42

58

32

69

22

78

Can trust most Can't be too careful Not concerned Concerned Not concerned Concerned

Trust people Gov't and personal data Corporation and personal data

Nonrecruited Recruited
© 2021 Ipsos 19

NRFU – Survey follow-up

• We saw very few differences between recruits and non-recruits

*p<0.05

84

14

2

27

73

34 30 34

2

85

12 2

23

77

27

39 33

1

Yes No Not sure No Yes Republican Democrat Independent Another party

Voter registration Vote Nov 2016 *Party ID

Nonrecruited Recruited

32

68 61

26

13

84

13 3

28

73 62

23 15

84

11 5

Non-drinker Drinker Never Former Current Never Former Current

Alcohol use Smoker Vaper

Nonrecruited Recruited
© 2021 Ipsos 20

Conclusions and Discussion 

• This analysis provided a look at households that were invited to join 

KP but declined to participate

• Households recruited to KP exhibited distributions almost identical to 

those of households not recruited 

• Differences between recruited and nonrecruited households were 

rarely statistically significant
© 2021 Ipsos 21

Conclusions and Discussion 

• Taken together, both the frame analysis and the NRFU provides 

reassurances that nonresponse bias with regard to panel recruitment 

is of minimal concern among the variables we examined

• The panel recruitment process is robust to differential nonresponse

• We also saw reasonably high completion rates, even among initial 

nonrespondents to panel recruitment, when combined with:

• the mail-first followed by push to web approach 

• high incentives
Frances M. Barlas, Ph.D.

frances.barlas@ipsos.com

Thank you!