1/ 32

Maximum likelihood estimation of response

propensity to a nonprobability survey

Vladislav Beresovsky 1

1 U.S. Bureau of Labor Statistics, OSMR

2023 FCSM Research & Policy Conference

October 24-26, 2023
2/ 32

This is a joint work with Julie Gershunskaya (OEUS) and Terrance

Savitsky (OSMR)

The views expressed are those of the authors and do not reflect the

official policy or position of U.S. Bureau of Labor Statistics
3/ 32

Outline

Survey statistics: from probability samples to data integration

Modeling response propensity to a nonprobability survey

Theoretical properties of the estimators

Simulation study to compare estimates by different methods

Discussion and next steps

References
4/ 32

▶ Problems with traditional probability surveys

▶ Labor intensive and time consuming data collection

▶ Increasing costs

▶ Decreasing response rates

▶ Restricted geography of data collection

▶ Proliferation of relatively inexpensive and expedient

nonprobability data sources:

▶ Web surveys (opt-in or online panels)

▶ Extracts from administrative records
5/ 32

Outline

Survey statistics: from probability samples to data integration

Modeling response propensity to a nonprobability survey

Theoretical properties of the estimators

Simulation study to compare estimates by different methods

Discussion and next steps

References
6/ 32

IPW estimation from nonprobability samples

▶ Na ̈ıve unweighted estimates of population parameters

from nonprobability sample may be biased

▶ To correct for a bias, we focus on Inverse Propensity

Weighting (IPW) methods, based on estimated latent

response propensity πc (x) to a nonprobability survey

▶ Methods for estimation of πc (x) from combined

nonprobability (convenience) sample Sc and probability

(reference) sample Sr:

▶ pseudo-likelihood based approaches (Chen, P. Li, and Wu

(2020),Wang, Y. Li, and Valliant (2021))

▶ likelihood based approach (Savitsky et al., 2022)

▶ We present and compare theoretical properties of these

methods and illustrate the differences using numerical

study and simulations
7/ 32

Combined sample S = Sc ⊕ Sr

πr (xi) = P (Ii = 1|i ∈ U, xi) are known reference sample selection

probabilities

πc (xi) = P (Ri = 1|i ∈ U, xi) are unknown response propensity to

convenience survey

Stack samples: S = Sc ⊕ Sr (overlapping units included twice)

Z is observed indicator on S: Zi = 1|i ∈ Sc and Zi = 0|i ∈ Sr

πz (xi) = P (Zi = 1|i ∈ S, xi) can be estimated
8/ 32

Core Relationship for Independent Sampling

Probabilities (CRISP)

Response propensity πc (xi) to convenience survey can be

found from CRISP:

πz (xi) = πc (xi)

πc (xi) + πr (xi)

Elliott (2009) and Elliott and Valliant (2017) present this formula

assuming negligible fraction of overlapping units.

The proof by Savitsky et al. (2022) makes no assumption about

the overlapping units.
9/ 32

Implicit Logistic Regression (ILR)

Bernoulli log-likelihood for observed Z on ‘stacked’ S = Sc ⊕ Sr :

l

ILR (Z, πz (φ)) = X

Sc⊕Sr

Zilog πzi + (1 − Zi) log (1 − πzi)

Use CRISP to parameterize πzi (φ) as composite function:

πzi (πci (φ)) = πci (φ)

πci (φ) + πri

, where logit [πci (φ)] = xiφ

Parameters φ can be estimated by solving score equations ∂lILR

∂φ = 0

with Newton-Raphson iterations.
10/ 32

Pseudo-ILR (PILR)

Bernoulli log-likelihood on ‘stacked’ Sc ⊕ U (Wang, Y. Li, and Valliant,

2021):

l(Z, πz (φ)) = X

Sc⊕U

Zilog πzi + (1 − Zi) log (1 − πzi)

Pseudo log-likelihood on S = Sc ⊕ Sr with wri = π

−1

ri :

l

PILR (Z, πz (φ)) = X

Sc⊕Sr

Zilog πzi + wri (1 − Zi) log (1 − πzi).

Use CRISP to parameterize πzi (φ) as composite function:

πzi (πci (φ)) = πci (φ)

πci (φ) + 1, where logit [πci (φ)] = xiφ.

If Sr ≡ U then l

PILR (φ) ≡ lILR (φ).
11/ 32

Likelihood Estimating Equations (CLW)

Bernoulli log-likelihood of unobserved R on U (Chen, P. Li, and Wu,

2020):

l(R, πc (φ)) = X

U

Rilog πci + (1 − Ri) log (1 − πci)

=

X

Sc

log 

πci

1 − πci 

+

X

U

log (1 − πci)

Pseudo log-likelihood with survey weights wri = π

−1

ri

l

CLW (πc (φ)) = X

Sc

log 

πci

1 − πci 

+

X

Sr

wri log (1 − πci),

Parameterize logit(πci (φ)) = x

′

iφ and solve score equations for φ.
12/ 32

A key difference between the methods

▶ CLW directly models the response indicator to

nonprobability survey Ri ∼ Bernoulli(πc(xi)). If Riis

observed, then CLW is optimal. However, Riis generally

unobserved, unless Sr = U.

▶ ILR and PILR use CRISP to implicitly model Ri by

modeling Zi ∼ Bernoulli(πz (πc(xi))), the observed

indicator of Sc on ‘stacked’ sample S = Sc ⊕ Sr
13/ 32

Outline

Survey statistics: from probability samples to data integration

Modeling response propensity to a nonprobability survey

Theoretical properties of the estimators

Simulation study to compare estimates by different methods

Discussion and next steps

References
14/ 32

Hajek estimator of population mean

Hajek IPW estimator of finite population mean

μ = N −1 P

i∈U

Yifrom the convenience sample with estimated

response propensity:

μˆ =

1

Nˆ

X

i∈Sc

yi

πˆci

,

where Nˆ =

P

i∈Sc

(ˆπci)

−1

.

How Var(ˆμ) is inflated due to estimation of πˆci?
15/ 32

Estimating equations for η = (μ, φ)

μ - population mean, φ - propensity model parameters.

Consider the system of estimating equations:

Φ (η) = 

S (μ)

S (φ)



= 0,

where

S (μ) = N

−1 X

i∈Sc

π

−1

ci (yi − μ) = 0

and S (φ) = Sc (φ) + Sr (φ) are score equations for ILR, PILR

or CLW.
16/ 32

Variances of η = (μ, φ)

Variances of parameter estimates are

Var(ηˆ) = H−1Var {Φ (η)} H−1,

where

Var {Φ (η)} =



Var(S (μ)) Cov (S (μ),Sc (φ))

Cov (S (μ),Sc (φ)) Var(S (φ)) 

,

H = −E



∂Φ (η)

∂η



is Hessian.
17/ 32

Variances of η = (μ, φ) (cont’d)

Var(ˆμ) = Var(S (μ)) − 2bCov (S (μ),Sc (φ)) + bVar(S (φ)) b

T

Var(φˆ) = H−1

φφ Var(S (φ)) H−1φφ

where

Var(S (φ)) = Var(Sc (φ)) + Var(Sr (φ))

b = H−1

φφ Hφμ, Hαβ = −E



∂Φ (β)

∂α 

All methods are asymptotically equivalent:

Var(ηb) ∼ O



n

−1

r + n

−1

c



However, Var(Sr (φ)) is a major contributor to the differences

between ILR, PILR and CLW methods.
18/ 32

Design variance and “overlap” between Sc and Sr

Vd (.) is design variance with respect to sampling indicator Iri:

ILR: Var(Sr (φ)) = Vd

X

U

Iri

gi

1 + gi

(1 − πci) xi



PILR: Var(Sr (φ)) = Vd

X

U

Irigi

1 − πci

1 + πci

xi



CLW: Var(Sr (φ)) = Vd

X

U

Irigixi



gi = πc (xi)/πr (xi) - depends on overlap between Sc and Sr in

covariate-defined domains xi.

▶ “High” overlap: πc (xi) and πr (xi) are similar for all xi

, and

gi ∼ nc/nr fluctuate around constant.

▶ “Low” overlap: πc (xi) and πr (xi) are dissimilar, and

gi ∈ (0, ∞).

For ILR, gi

1+gi

∈ (0, 1).

For PILR, variability due to large giis reduced by 1−πci

1+πci

.
19/ 32

High and Low overlaps between Sc and Sr in covariate domains

▶ Covariates: X ∼ N (0, 1)

▶ Sc response propensity: logit(πci) = φ0 + φc ∗ xi

, φc = 1.0

▶ Sr PPS size measure: logit(Mri) = 1.0 + φr ∗ xi

Figure: Low overlap: φr = −φc; High overlap: φr = φc
20/ 32

Ratio of SE φˆ

1

ILR

to SE φˆ

1

CLW

Figure: Contour plots of the ratio of SE depending on sampling

fractions (fr, fc)
21/ 32

Ratio of SE μˆ

ILR

to SE μˆ

CLW

Figure: Contour plots of the ratio of SE depending on sampling

fractions (fr, fc)
22/ 32

Outline

Survey statistics: from probability samples to data integration

Modeling response propensity to a nonprobability survey

Theoretical properties of the estimators

Simulation study to compare estimates by different methods

Discussion and next steps

References
23/ 32

Simulations scenarios

The purpose of simulations is to illustrate effects of sample

fractions, sizes and overlap (“high” and “low” ) between Sr and

Sc on estimates ηˆ =



μ, ˆ φˆ



.

Sample PPS sample Sr Poisson sample Sc

fraction logit(Mri) = 1.0 + φr1 ∗ xilogit(πci) = φc0 + 1.0 ∗ xi

fr = .1, fc ≈ .1 nr = 100 nc ≈ 100

nr = 600 nc ≈ 600

fr = .01, fc ≈ .01 nr = 100 nc ≈ 100

nr = 600 nc ≈ 600

fr = .1, fc ≈ .01 nr = 1, 000 nc ≈ 100

fr = .01, fc ≈ .1 nr = 100 nc ≈ 1, 000

Covariates: xi ∼ N (0, 1)

fc ≈ .1, φc0 = −2.5 and fc ≈ .01, φc0 = −5.0

High overlap: φr1 = 1.0, Low overlap: φr1 = −1.0

Outcome variable on Sc: yi ∼ N



1 + xi, 1.5

2



Sc and Sr are sampled 500 times from a simulated population.
24/ 32

Propensity parameter φb1 (true value is φ1 = 1.0 ) for sample fractions

(fc, fr) = (0.01, 0.1) and (0.1, 0.01), population size N = 10, 000

High overlap Low overlap

φb1 SD SE SE d 95%CI φb1 SD SE SE d 95%CI

nc = 100, nr = 1, 000

ILR 1.00 0.12 0.11 0.11 0.95 1.01 0.14 0.13 0.13 0.95

PILR 1.00 0.12 0.12 0.11 0.95 1.03 0.16 0.15 0.18 0.94

CLW 1.01 0.12 0.12 0.12 0.95 1.05 0.19 0.16 0.23 0.93

nc = 1, 000, nr = 100

ILR 1.00 0.14 0.14 0.15 0.94 1.01 0.15 0.15 0.15 0.95

PILR 1.01 0.16 0.16 0.16 0.95 1.05 0.26 0.25 0.29 0.94

CLW 1.05 0.23 0.22 0.22 0.95 1.29 0.62 0.51 0.68 0.92

SD - SD of φb1 over simulations

SEd - estimated SE averaged over simulations.

SE - calculated SE for population.
25/ 32

Figure: Relative residuals of the estimated propensity parameters φb1

for sample fractions (fc, fr) equal to (0.01, 0.1) (upper row) and to

(0.1, 0.01) (lower row) over the simulations
25/ 32

Estimate of population mean μb (true value is μ = 1.0) for sample fractions

(fc, fr) = (0.01, 0.1) and (0.1, 0.01), population size N = 10, 000

High overlap Low overlap

μb SD SE SE d 95%CI μb SD SE SE d 95%CI

nc = 100, nr = 1, 000

ILR 1.07 0.30 0.29 0.31 0.87 1.03 0.32 0.28 0.31 0.88

PILR 1.07 0.30 0.30 0.32 0.88 1.00 0.35 0.34 0.36 0.90

CLW 1.07 0.30 0.31 0.32 0.88 0.98 0.37 0.37 0.39 0.91

nc = 1, 000, nr = 100

ILR 1.02 0.14 0.14 0.14 0.94 1.01 0.13 0.13 0.13 0.95

PILR 1.00 0.17 0.18 0.16 0.96 0.96 0.24 0.27 0.26 0.96

CLW 0.97 0.23 0.29 0.21 0.95 0.75 0.54 Inf 0.60 0.95

SD - SD of μb over simulations

SEd - estimated SE averaged over simulations.

SE - calculated SE for population.
26/ 32

Figure: Relative residuals of the estimated population mean μb for

sampling fractions (fc, fr) equal to (0.01, 0.1) (upper row) and to

(0.1, 0.01) (lower row) over the simulations
26/ 32

Outline

Survey statistics: from probability samples to data integration

Modeling response propensity to a nonprobability survey

Theoretical properties of the estimators

Simulation study to compare estimates by different methods

Discussion and next steps

References
27/ 32

Comparison of the three methods for estimation of πc

▶ All three methods are asymptotically consistent and

similarly efficient under favorable conditions.

▶ ILR and PILR use “stacked” sets and employ CRISP

formula πz = πc/(πc + πr).

CLW models unobserved indicator of Sc on the population.

▶ ILR uses likelihood; PILR and CLW use pseudo-likelihood.

▶ ILR requires sampling probabilities πr for convenience

sample units. If they cannot be inferred from the probability

sample design, PILR should be used.

▶ ILR is more efficient especially in case of low overlap in

covariate domains, which is important for multivariate

models.
28/ 32

What’s next ...

▶ Explore applying ILR to integrate BLS probability surveys with

administrative data. This may substantially improve estimates in

small domains (SAE) and for rare populations.

▶ The presented methods use CRISP to define composite

dependence of likelihood on model parameters. The CRISP

formula is general. Bayesian estimation has been implemented

by Savitsky et al. (2022). The methodology can be extended to

popular ML algorithms: CART, LASSO, Random Forest, Neural

Networks, etc.
29/ 32

Outline

Survey statistics: from probability samples to data integration

Modeling response propensity to a nonprobability survey

Theoretical properties of the estimators

Simulation study to compare estimates by different methods

Discussion and next steps

References
30/ 32

References I

Chen, Y., P. Li, and C. Wu (2020). “Doubly Robust Inference

With Nonprobability Survey Samples”. In: Journal of the

American Statistical Association 115.532, pp. 2011–2021.

eprint: https:

//doi.org/10.1080/01621459.2019.1677241.

Elliott, M. R. (2009). “Combining Data from Probability and

Non-Probability Samples Using Pseudo-Weights”. In:

Survey Practice 2 (6), pp. 813–845.

Elliott, M. R. and R. Valliant (2017). “Inference for

Nonprobability Samples”. In: Statistical Science 32.2,

pp. 249–264.

Savitsky, T. et al. (2022). “Methods for Combining Probability

and Nonprobability Samples Under Unknown Overlaps”. In:

eprint: https://arxiv.org/abs/2208.14541.
31/ 32

References II

Wang, L., Y. Li, and R. Valliant (2021). “Adjusted logistic

propensity weighting methods for population inference using

nonprobability volunteer-based epidemiologic cohorts”. In:

Stat Med. 40.4, pp. 5237–5250.
32/ 32

Thank you for attending FCSM!

Vladislav Beresovsky

Beresovsky.Vladislav@bls.gov