1/ 22

Combining Probability and Nonprobability

Samples Under Unknown Overlaps

Terrance D. Savitsky 1 Matthew R. Williams 2

Julie Gershunskaya 3 Vladislav Beresovsky 4

Nels G. Johnson 5

1 U.S. Bureau of Labor Statistics (Office of Survey Methods Research)

2RTI International (Division for Statistical and Data Sciences)

3U.S. Bureau of Labor Statistics (OEUS Methods Division),

4National Center for Health Statistics

5USDA Forest Service

FCSM

October, 2022
2/ 22

Outline

Motivation

Methods

Bayesian Hierarchical Model

Simulation Performance Study
3/ 22

Combine convenience sample with reference sample

I Improve estimation efficiency with convenience sample

I Non-probability sample inexpensive and easily accessible

I Often has a lot more units than reference probability sample

I Treat convenience sample as from latent random sampling

mechanism:

I Estimate latent inclusion probabilities, πc(xi)

I Use overlap of predictor values (xci, xri) and known

reference sample πr(xi)

I Reference and convenience samples may overlap units

I Exclude convenience units that inflate estimator variance

I Remove convenience units very different from reference

I xci values very different from xri
4/ 22

Outline

Motivation

Methods

Bayesian Hierarchical Model

Simulation Performance Study
5/ 22

Terminology

I Sc and Sr observed convenience and reference samples

I Population frames Uc and Ur

I Target population U

0

, such that Uc ⊆ U

0 and Ur ⊆ U0

.

I Known coverage probabilities of U

0 by frames Uc and Ur

I pc (xi) = P



i ∈ Uc|i ∈ U

0

, xi

	

pr (xi) = P



i ∈ Ur|i ∈ U

0

, xi

	

I inclusion probabilities into Sc and Sr

I πc (xi) = P {i ∈ Sc|i ∈ Uc, xi}

πr (xi) = P {i ∈ Sr|i ∈ Ur, xi}

I Consider combined sample, S = Sc + Sr.

I Indicator zi = 1 when i ∈ Sc, and zi = 0 when i ∈ Sr

I πz(i) = P {i ∈ Sc | i ∈ S, xi} → propensity scores
6/ 22

Proposition: The following relationship holds:

πz (xi) = πc (xi) pc (xi)

πc (xi) pc (xi) + πr (xi) pr (xi)

.

Proof: two copies of U

0

: U = U

0 + U0

.

P {i ∈ Sc|i ∈ U, xi} = P{i∈Sc|i∈Uc,i∈U0,xi}P{i∈Uc|i∈U0

,xi}P{i∈U0

|i∈U}

=

1

2

πc (xi) pc (xi)

Similarly, P {i ∈ Sr|i ∈ U, xi} =

1

2

πr (xi) pr (xi).
7/ 22

Hence, for units in S = Sr + Sc, we have

P {i ∈ S|i ∈ U, xi} = P {i ∈ Sc|i ∈ U, xi} + P {i ∈ Sr|i ∈ U, xi}

=

1

2

πc (xi) pc (xi) + 1

2

πr (xi) pr (xi).

By definition of conditional probability,

P {i ∈ Sc|i ∈ S, i ∈ U, xi} =

P {i ∈ Sc|i ∈ U, xi}

P {i ∈ S|i ∈ U, xi}
8/ 22

Exact Likelihood Method when Uc = Ur

I Same Population Frame for each sampling arm

I pc(xi) = P(i ∈ Uc | i ∈ U

0

) = pr(xi)

πz (xi) = πc (xi)

πc (xi) + πr (xi)

I Produces exact likelihood for observed data

zi ∼ Bernoulli(πz(xi)), which allows to implicitly estimate

parameters of πc(xi, β)

I Elliot, 2009 derived the same formula assuming no-overlap

between samples

I Sc and Sr may be overlapping
9/ 22

Outline

Motivation

Methods

Bayesian Hierarchical Model

Simulation Performance Study
10/ 22

Joint model for [(zi),(πri)i∈Sr]

1. Parameterize our model using π`i = P {i ∈ S`| i ∈ U`, xi}.

I Unit i ∈ 1, . . . ,(n = nr + nc)

I Sampling arm ` ∈ (r, c)

I Estimate (π`i) for all units for both ` = r and ` = c

2. logit(π`i) = μx,`i = xiγx,` +

PK

k=1 g(xki)β`k

I B-spline basis for each predictor where C × 1, g(xki), with

C = knots + spline degrees - 1

I Autoregressive smoothing of the C × 1, β`k

I Sparsity over K predictors with β`kc ∼ N (β`kc−1, κ`kτ`)

3. Joint likelihood for [(zi),(πri)i∈Sc

]

I zi

| πzi

ind∼ Bernoulli(πzi)

I logit(πri)

ind∼ N (μx,ri, φ) only for units i ∈ Sr
11/ 22

Outline

Motivation

Methods

Bayesian Hierarchical Model

Simulation Performance Study
12/ 22

Compare Exact and Pseudo Likelihood Methods

I Exact Likelihood Methods (Bayesian Implementation)

I Two-arm option:

(Sc, Sr) : πz(xi) = πc(xi)/ (πc(xi) + πr(xi))

I One-arm option: (Sc, U) → πr(xi) = 1: πz (xi) = πc(xi)

πc(xi)+1

I One-arm gold standard since know whole population of X.

I Pseudo Likelihood Methods (Bayesian Implementation)

I Competitors define likelihood on population indicator

I Approximate on observed sample using weights ∝ 1/πr(xi)

I Chen, P. Li, and Wu (2020)(LCW) specify Bernoulli (πc(xi)) for pop

I Wang, Valliant, and Y. Li (2021) (WVL) specify Bernoulli (πz(xi)) for

pop - same as One-arm
13/ 22

Data Generation Process

I We generate M = 30 distinct populations of size N = 4000.

I Let X have K = 5 predictors (one continuous)

I Outcome yi has a lognormal distribution

log(yi) ∼ N (xiβ, 2).

I We chose a large sampling fractions to explore the full

range of πc ∈ [0, 1] (establishment surveys).

I Select reference sample of nr = 400 using PPS sampling:

sri = log(exp(xiβ) + 1)

I Select two convenience samples of nc ≈ 800 using Poisson

sampling: πci = logit−1

(xiβc + offset)

I We control ‘high’ and ‘low’ overlap by varying βc compared

to the reference sample (next slide)
14/ 22

High and Low Overlap of Xr and Xc Datasets

0.0

0.2

0.4

0.6

0 300 600 900

Unit

Inclusion Probability

p_c p_r

0.00

0.25

0.50

0.75

0 300 600 900 1200

Unit

Inclusion Probability

p_c p_r

Figure: πc versus πr LHS high overlap and RHS low overlap.
15/ 22

Higher Percent of Pooled Sample in High Overlap

2.5

5.0

7.5

10.0

High Low

overlap

percent

Figure: Distributions over 30 population and sample realizations.
16/ 22

Two-arm Method is More Efficient

two−arm CLW CLW−A WVL WVL−A

High Overlap Low Overlap

0.00 0.25 0.50 0.75 1.000.00 0.25 0.50 0.75 1.000.00 0.25 0.50 0.75 1.000.00 0.25 0.50 0.75 1.000.00 0.25 0.50 0.75 1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

πc

pre

dicte

d

πc

^

two−arm CLW CLW−A WVL WVL−A

Average Predicted Inclusion Probabilities 

with 95% Quantiles Across 30 Simulations

Figure: Avg and 95% frequentist quantiles for posterior mean of πc .
17/ 22

Coverage degrades for pseudo likelihood options

bias rmse cov wid

High Overlap Low Overlap

0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00

−0.25

0.00

0.25

0.50

0.75

1.00

0.0

0.5

1.0

true πc

two−arm CLW CLW−A WVL WVL−A

Mean and Median Difference, Mean Absolute Difference, 

Root MSE, 90% coverage for predicted convenience sample inclusion probabilities

Figure: Pointwise coverage comparisons of 90% credibility intervals in

3rd column
18/ 22

Application to Estimation of Government Employment

I Estimate pseudo weights for quota sample of government

employment.

I Use census instrument as reference sample; we set

πri = 1 for all units

I We observe: z = 1 for units in the quota sample and z = 0

for units in the census.

I Quota sample units are a subset of census.

I Estimate πc (xi) of inclusion into the quota sample, where

xiis employment level of unit i

I Produce employment estimates for Metropolitan Statistical

Areas (MSAs).
19/ 22

Pseudo Weighted Link Relative Estimator (WLR)

Yˆ

d,12 = Yd,0

Y

12

τ=1

Rˆ

d,τ .

I Starting level, Yd,0

, available from census at end of year

I Monthly ratio estimates Rˆ

d,τ are obtained using a link

relative (LR) estimator

RˆLR

d,τ =

X

i∈sd,τ

yi,τ /

X

i∈sd,τ

yi,τ−1

I We fear LR induces bias by use of unweighted (yi,τ−1, yi,τ ).

I So, use a weighted LR estimator.

RˆW LR

d,τ =

X

i∈sd,τ

wiyi,τ /

X

i∈sd,τ

wiyi,τ−1
20/ 22

Estimations for Selected MSAs

3 4

1 2

0 1 2 3 4 5 6 7 8 9 10 11 12 0 1 2 3 4 5 6 7 8 9 10 11 12

0 1 2 3 4 5 6 7 8 9 10 11 12 0 1 2 3 4 5 6 7 8 9 10 11 12

130000

140000

150000

10000

11000

12000

13000

14000

350000

400000

450000

70000

75000

80000

85000

month

employment

Hist LR WLR
21/ 22

Future Work

I Joint estimation of (πc, y).

I Create efficient survey estimator for domains.

I Incorporates full uncertainty quantification.
22/ 22

References I

Chen, Yilin, Pengfei Li, and Changbao Wu (2020). “Doubly

Robust Inference With Nonprobability Survey Samples”. In:

Journal of the American Statistical Association 115.532,

pp. 2011–2021.

Elliot, Michael R. (2009). “Combining Data from Probability and

Non-Probability Samples Using Pseudo-Weights”. In: Survey

Practice 2 (6), pp. 813–845.

Wang, L., R. Valliant, and Y. Li (2021). “Adjusted logistic

propensity weighting methods for population inference using

nonprobability volunteer-based epidemiologic cohorts”. In:

Stat Med. 40.4, pp. 5237–5250.