Predicting Future Participation in

AmeriSpeak Panel

Bayesian mixed model analysis

FCSM / October 25, 2022

Stas Kolenikov, David Dutwin, Ipek Bilgen, Poulami Maitra
Problem statement / background

In both online panels and longitudinal surveys, panel attrition and low

participation rates impact survey estimates, total survey error, as well as

panel operation costs.

Panel retention and high survey cooperation rates are also crucial to support

studies of lower incidence populations and to potentially minimize

nonresponse bias.

A potential indicator of low participation and panel attrition/turnover is

panelist dissatisfaction.

MOTIVATION : Problem statement 2

Panel Attrition
WHY?

Can we predict attrition and decline in participation over time? If so, how?

Predicting panel attrition (i.e., drop out) and nonresponse to subsequent surveys using

panelist’s self-reported data, as well as paradata

Identification of attrition and nonresponse patter

Preventing or forestalling the decline of interest and dropout

cheaper than recruiting new panelists

helps maintaining the panel representativeness

PANEL ATTRITION : RESEARCH QUESTIONS 5
WHAT?

Is it feasible to predict panel attrition and nonresponse to subsequent surveys using overall

and over time panelist self-reported future participation scores?

Who has lower panelist self-reported future participation scores?

If (and how) their participation scores change over time?

Who is expected to drop out faster based on our models?

HOW?

Predicting future drop out and nonresponse to subsequent surveys using change in surveylevel panelist participation propensities and self-reported future participation scores over

time

Bayesian cross-classified latent trajectory model combining paradata (unit response; surv

features) and survey data (FUTURE)

PANEL ATTRITION : RESEARCH QUESTIONS 6
FUTURE question:

How likely are you to participate in subsequent survey invitations from AmeriSpeak?

(1) Not at all,

(2) somewhat,

(3) very, or

(4) extremely likely?

Introduced in June 2021, asked no more frequently than once every two weeks

PANEL ATTRITION : SELF-REPORT 7
Analysis
AmeriSpeak data Year 2021 recruits

98 surveyswithat least 2,500 invited participants

small surveysare usuallyvariouspilots

3,366 panelists

65,277 invitations

13,911 completesand51,366 incompletes/refusals. ANALYSIS : DATA

9
Model in equations

A two-way/cross classified latent trajector mod

sequential number of survey among the -th panelist roster of invites.

ANALYSIS : MODEL

θij = logit P[rij = 1], rij ≡ panelist i completes survey j

θij = β

′

p⋅sxij + Intercepti + Slopei mi(j) + νj

Intercepti = β

′

personxi + e1i, Slopei = β

′

slope

xi + e2i

νj = β

′

svyzj + εj

,

P[futureij = m|rij] = PΛ(κm−1 < γ1 + γ2θij ≤ κm)

PΛ(κm−1 < x ≤ κm) = Λ(κm) − Λ(κm−1), Λ(z) = [1 + exp(−Z)]

−1

mi(j) ≡ i j

10
Model as a diagram ANALYSIS : MODEL 11
Need more slopes ANALYSIS : MODEL WITH A KINK 12
Significant variabl

parameter mean 2.5% 97.5%

beta_person_intercept[Const.] -3.193 -3.534 -2.862

beta_person_intercept[race4] -0.056 -0.104 -0.006

No significant variables is... goo

Initial recruitment is effective in bringing in a variety of respondents who are equally

motivated to start

ANALYSIS : INITIAL PROPENSITY 13
Significant variabl

parameter mean 2.5% 97.5%

beta_person_slope1[Const.] -0.108 -0.186 -0.023

beta_person_slope1[ln_weights] -0.038 -0.055 -0.024

beta_person_slope1[age] 0.004 0.002 0.005

beta_person_slope1[educ_5] 0.022 0.002 0.042

beta_person_slope1[race4] -0.069 -0.113 -0.023

beta_person_slope2[ln_weights] 0.044 0.022 0.067

beta_person_slope2[age] -0.004 -0.006 -0.002

Faster attrition:

younger

less educated

Hispanic (race==4)

ANALYSIS : SLOPES 14
Significant variabl

The FUTURE question appears twice in the model:

(1) as the contemporaneous outcome of the latent propensity to respond...

parameter mean 2.5% 97.5%

gamma_future[slope] 3.797 3.597 4.005

gamma_future[Const.] 4.632 4.333 4.951

The relevant part is the slope coefficient -- which is the slope of the regression of the FUT

question on the latent response propensity.

ANALYSIS : FUTURE QUESTION 15
Significant variabl

The FUTURE question appears twice in the model:

... (2) as the (time-varying) predictor of the response propensity in the future

have to jump through some data management hoops to define "last known value 

FUTURE"

parameter mean 2.5% 97.5%

beta_person_by_survey[future12] 1.022 0.869 1.176

beta_person_by_survey[future3] 1.616 1.471 1.771

beta_person_by_survey[future4] 2.832 2.689 2.979

The "desirable" monotonicity is observed: higher values of FUTURE move the propensity up quite

substantially:

hypothetical unit response rate = 20% with FUTURE == 4

counterfactual unit response rate = 4% with FUTURE == 1, 2

ANALYSIS : FUTURE QUESTION 16
Bayesian diagnostics

4 chains; 1000 warmup draws; 500 simulation draws; adaptive delta parameter of 0.92; tree

depth of 10; thinning by 2

For regression slopes, all values of the variance reduction factor (Rhat statistic) were below

1.05.

For the model estimates, the values ranged from 0.997 to 1.040.

For the person-level random effects, the values ranged from 0.996 to 1.020.

The Rhat statistics for the variance components indicated lack of chain convergence. The

Rhat for was 1.157, and for , 1.130.

There were no divergent transitions in the simulation part of the chain.

The treedepth never approached the maximum value in the simulation part of the chain.

ANALYSIS : BAYESIAN DIAGNOSTICS

τ1 τ2

17
Ba

yesian diagnostics

ANALYSIS : BAYESIANDIAGNOSTICS 18
Goodness of f

Nowcast of the response propensity: iron out the kinks and wiggles

ANALYSIS : GOODNESS OF FIT 19
Parting words
Conclusions and discussion

Moderately sophisticated Bayesian modeling exercise

Explainable patterns of decline in participation

lower education, younger, ethnic minorities

Self-reported intent to continue panel participation is valid

Identify high-risk panelists

greatest (most negative) slopes

decline in self-reports

Next steps: interventions for the highest risk panelists

Still needed: social theory of attrition

accelerated time?

mixture model?

Still needed: social theory of panel time

current model: number of invitations

calendar time?

Parting words : WHAT WE LEARNED 21
Thank you.

StasKolenikov

kolenikov-stas@norc.org

@StatStas

IpekBilgen

DavidDutwin

PoulamiMaitra