A Review of Methods to Combine 

Probability Surveys and/or Calibrate 

One Probability Survey with Official 

Federal Statistical Survey Data

2023-10-24

NORC Team + NCHS Team
ACKNOWLEDGEMENTS AND DISCLAIMERS 2

NORC Literature Review Team

David Dutwin

Ipek Bilgen

Stas Kolenikov

Michael Yang

Chien-Min Huang

Margrethe Montgomery

NCHS Team

Paul Scanlon

J ames Dahlhamer

Katherine Irimata

FCSM 2023 – Was hington, DC – 24-10-2023

Views expressed in this presentation are those of the authors, 

and do not represent CDC, NCHS or NORC.

NORC R code team

Stas Kolenikov

Soubhik Barari

Nuria Adell Raventos

Emers on Berry

Chien-Min Huang

J iazhi Yang

Matt Gunther
Agenda

01 Why combined data

02 Methodological considerations

03 Statistical methods: Combine probability 

sources

04 Statistical methods: Combine probability and 

non-probability sources

05 Current work
Why combined data
SECTION : WHY COMBINED DATA 5

Fitness for use and various aspects of quality

FCSM 2020 Data Quality Framework

• Three domains, 11 dimensions

Gold s tandard in-person surveys

• NHIS: National Health Interview Survey; NHANES: National Health and Nutrition Examination Survey

• Better accuracy due to decades of experience, underlying methods research, higher response rates in 

active interview modes

Online panels

• RANDS: Research and Development Survey; RSS: Rapid Survey System

• Better timeliness (and hence, potentially, relevance) due to faster turnaround times

Variations in granularity, accuracy (specimen instrumentation in NHANES vs. self-report in NHIS, RSS).

The question of combining the data is that of coherence between sources.
Methodological considerations
METHODOLOGICAL CONSIDERATIONS : ONLINE P ANELS 7

Methodological considerations in online panels

• Probability vs. non-probability nature of recruitment

• Sampling frames

– Coverage of the U.S. population

• Panel refresh frequency

• Panelist interview load / burden

• Retention, incentivization strategies

• Documentation of the above in public-facing documents, on demand, not at all
METHODOLOGICAL CONSIDERATIONS : METRICS 8

How can one compare the different panels?

• Response rates

• Panel composition, representation of 

key subpopulations

– Racial and ethnic minorities

– Young adults

– Non-English speakers

• Panel recruitment

– Coverage

– Protocols

• Panel maintenance and retainment

• Weighting methodology

– Complex sampling designs

– Replenishment

– Adjustments for eligibility

– Adjustments for nonresponse in 

recruitment

– Adjustments for nonresponse in 

individual surveys

• Transparency

See als o: ESOMAR 37 ques tions 

• The European Society for Opinion and Marketing Research

• https:/ / esomar.org/ code-and-guidelines/ 37-questions-to-help-buyers-of-online-samples
METHODOLOGICAL CONSIDERATIONS : S OURCES OF P OTENTIAL BIAS 9

Total survey error and sources of biases

• Coverage of the non-Internet population

– Additional response mode(s) vs. access provision

• Recruitment mode(s)

– Mixed modes more likely to recruit diverse panels

– Recruitment mode ≠ data collection mode

• Recruitment materials design features

• Within household selection

– Recruitment of every HH member

• Main data collection mode effects

• Panel conditioning

• Low quality respondents

– Ineligible (e.g. does not reside in the U.S.)

– Cheaters

– Speeders

– Bots
Combining survey data
COMBINED DATA : P RINCIP AL REFERENCES 11

Review of reviews

• Citro (2014)

• Elliott and Valliant (2017)

• Lohr and Raghunathan (2017)

• Rao (2020)

• Yang and Kim (2020)

• Beaumont (2020)

• Wu (2022)
Combining multiple probability surveys
MULTIP LE P ROB SOURCES : DUAL FRAME 13

Dual/multiple frame estimation

• Each source is considered a separate frame

• For each combined data observation, determine (potential) frame membership

• Compositing: YY�

λλ = YY�aa + λλYY�

aaaa

AA + 1 − λλ YY�

aaaa

BB + YY�

– Optimize λλ to minimize sampling variance or coverage bias

• Single frame: YY� = ∑ii

yyii

ππii;frame 1+ππii;frame 2−ππii;both

• More than two frames: frame count estimator

Hartley (1962), Kalton and Anderson (1986), Bankier (1986), Lohr (2009) 
MULTIP LE P ROB SOURCES : REGRES S ION AND CALIBRATION 14

Regression- and 

calibration-type methods

• First survey regression estimator:

yy�RR = yy�HHHH + BB�′ TTZZ − ZẐ

HHHH ; BB� = ZZ′

WW −1ZZZ

• Second survey adjustment:

yy�AA = yy�RR + DD�′ XX� − XX�

RR ; DD� = ZZ′

RR −1ZZ′RR ; RR = WW − WW ZZ′WW −1ZZ′WW

• +1/-1 raking trick 

• Empirical likelihood family of approaches

Renssen and Nieuwenbroek (1997), Hidiroglou (2001), Wu (2004), Fu et al. (2008) 

Outcome y Covariate X Covariate Z

Survey 1   

Survey 2 

Population 
MULTIP LE P ROB SOURCES : IMP UTATION 15

Multiple and mass imputation

• Fit imputation models on survey 1 

• Impute inside survey 2

• Rubin’s combine formula:

θθ̂

MM

MMMM =

1

MM �

mm=1

MM

θθ̂

mm ; VV�

MM

MMMM =

1

MM �

mm=1

MM

vv�mm + 1 +

1

MM �

mm=1

MM

θθ̂

mm − θθ̂

MM

MMMM θθ̂

mm − θθ̂

MM

MMMM 

• Complex surveys: 

– Subsample survey 1 according to complex survey design per each imputation

– Internal vv�mm must account for the complex survey design (of survey 2)

Rubin (1976), Shao and Sitter (1996), Raghunathan (2006), Schenker et al. (2010), Kim and 

Rao (2012), van Buuren (2018)

Outcome y Covariate X

Survey 1  

Survey 2 
MULTIP LE P ROB SOURCES : BAYES IAN METHODS 16

Bayesian methods

Macro-level

• Bayesian updating / combining:

yy1~NN μμ, σσ1

2 & yy2~NN μμ, σσ22 ⇒ yycc|yy1, yy2~NN μμcc, σσcc

2 , μμcc =

yy1/σσ1

2 + yy2/σσ22

1/σσ1

2 + 1/σσ22 , σσcc

2 =

1

1/σσ1

2 + 1/σσ22

Micro-level

• Model estimation for survey 2 with priors obtained from survey 1

Erciulescu, Opsomer and Breidt (2021)
MULTIP LE P ROB SOURCES : SAE 17

Small area estimation methods

yyii = θθii + sampling errorii; θθii = xxii

′

ββ + model errorii

As ymmetric roles of data

• Survey 1 provides covariates for SAEs built on Survey 2

Symmetric roles of data

• Different sources are random area-level effects (possibly with bias) 

Raghunathan et al. (2007), Ybarra and Lohr (2008), Kim et al. (2015)
Combining probability and nonprobability surveys
P ROB + NONP ROB SOURCES : US UAL S US P ECTS 19

Applicable prob + prob methods

• Superpopulation modeling

• Mass imputation

• Calibration

– Base weights?

– Lasso selection for the outcome model (Chen et al. 2018)

– Behavior variables
P ROB + NONP ROB SOURCES : SAE 20

Small area estimation methods

J oint detailed domain modeling:

yydd

PP = xxdd′ ββ + ννdd + εεddPP

yydd

NN = xxdd′ ββ + ννdd + εεddNN + ααddNN

• ννdd: domain model error

• εεdd

PP

, εεdd

NN : domain sampling error (estimable)

• ααdd

NN : systematic error in the low quality source (may have nonzero mean)

Ganesh et al. (2017)
P ROB + NONP ROB SOURCES : P ROP ENS ITY S CORE METHODS 21

Propensity score adjustments

• Model membership in the non-probability sample (over combined data set):

Pr δδii = 1 xxii = parametric or machine learning model

• Estimating equations:

∑ii∈SSNNNN 1 − ppii αα xxii − ∑ii∈SS wwiippii αα xxii = 0 or ∑ii∈SSNNNN

xxii

ppii αα − ∑ii∈UU xxii = 0

• Non-prob sample weights:

– Inverse pseudo-probabilities of inclusion

– Propensity classes / cells

– Weight imputation from PS-matched donors

– Kernel estimates

Kim and Wang (2019), Chen et al. (2020), Wang et al. (2022), Shin et al. (2022)
P ROB + NONP ROB SOURCES : DOUBLE ROBUS T 22

Doubly robust methods

yy�DD = �

ii∈SSNNNN

wwii

NN yyii − mm(xxii, ββ̂) + �

ii∈SS

wwii

PP mm(xxii, ββ̂)

where mm(xxii, ββ̂) is the model for outcome (parametric or machine learning) and wwii

NN is the 

pseudo-weight for the non-probability sample (usually obtained via PS-type methods)

• If the propensity model is right, ∑ii∈SSNNNN wwii

NN mm(xxii, ββ̂) and ∑ii∈SS wwiiPP mm(xxii, ββ̂) cancel one another, 

and the estimate is essentially ∑ii∈SSNNNN wwii

NN yyii

• If the outcome model is right, ∑ii∈SSNNNN wwii

NN yyii − mm(xxii, ββ̂) is zero, and the estimate is essentially 

∑ii∈SS wwii

PP mm(xxii, ββ̂)

Chen et al. (2020); Kim and Wang (2019); Valliant (2020); Yang et al. (2020)
Simulation study
SIMULATION : METHODS AND S CENARIOS 24

Simulation study

Es timators

• Calibration to standard demographics

• Calibration to demographics + health

• Calibration with lasso selection

• Propensity score

• Double robust

• NORC SAE-type estimator

Scenarios for online panels

• SRS

• Mild to strong correctable nonresponse

• Non-correctable nonresponse

• Mild to strong coverage error
Thank you. Stas Kolenikov 

Principal Statistician

kolenikov-stas@norc.org
26

References - I

Bankier, M. D. (1986). Estimators based on several 

stratified samples with applications to multiple frame 

surveys. Journal of the American Statistical Association

81(396), 1074–1079.

Beaumont, J.-F. (2020). Are probability surveys bound to 

disappear for the production of official statistics? Survey 

Methodology 46(1), 1–29. 

van Buuren, S. (2018). Flexible Imputation of Missing Data

(2nd ed.). Interdisciplinary Statistics. Boca Raton, FL: 

Chapman and Hall/CRC.

Chen, J. K. T., R. L. Valliant, and M. R. Elliott (2018). 

Model-assisted calibration of non-probability sample 

survey data using adaptive lasso. Survey Methodology 

44(1), 117–144.

Chen, Y., P. Li, and C. Wu (2020). Doubly robust 

inference with nonprobability survey samples. Journal of 

the American Statistical Association 115(532), 2011–

2021.

Citro, C. F. (2014). From multiple modes for surveys to 

multiple data sources for estimates. Survey 

Methodology 40(2), 137–162.

Elliott, M. R. and R. Valliant (2017). Inference for 

nonprobability samples. Statistical Science 32(2), 249–

264.

Erciulescu, A. L., J. D. Opsomer, and F. J. Breidt (2021). 

A bridging model to reconcile statistics based on data 

from multiple surveys. The Annals of Applied Statistics

15(2), 1068–1079.

Fu, Y., Wang, X. and Wu, C. (2008). Weighted Empirical 

Likelihood Inference for Multiple Samples. Journal of 

Statistical Planning and Inference, 139, 1462–1473.
27

References - II

Ganesh, N., V. Pineau, A. Chakraborty, and J. M. Dennis 

(2017). Combining probability and non-probability samples 

using small area estimation. In JSM Proceedings, Survey 

Research Methods Section, 1657–1667. Alexandria, VA: 

American Statistical Association.

Hartley, H. O. (1962). Multiple frame surveys. In JSM 

Proceedings, Social Statistics Section, 203–206. 

Alexandria, VA: American Statistical Association.

Hidiroglou, M. (2001). Double sampling. Survey 

methodology 27(2), 143–154.

Kalton, G. and D. W. Anderson (1986). Sampling rare 

populations. Journal of the Royal Statistical Society: 

Series A (General) 149(1), 65–82.

Kim, J. K., S. Park, and S.-Y. Kim (2015). Small area 

estimation combining information from several sources. 

Survey Methodology 41(1), 21–36.

Kim, J. K. and J. Rao (2012). Combining data from two 

independent surveys: a model-assisted approach. 

Biometrika 99(1), 85–100.

Kim, J. K. and Z. Wang (2019). Sampling techniques for 

big data analysis. International Statistical Review 

87(S1), S177–S191.

Lohr, S. L. (2009). Multiple-frame surveys. In Handbook 

of Statistics; Sample Surveys: Design, Methods and 

Applications, Volume 29, 71–88. Elsevier.
28

References - III

Lohr, S. L. and T. E. Raghunathan (2017). Combining 

survey data with other data sources. Statistical Science

32(2), 293–312.

Raghunathan, T. E. (2006). Combining information from 

multiple surveys for assessing health disparities. 

Allgemeines Statistisches Archiv 90(4), 515–526.

Raghunathan, T. E., D. Xie, N. Schenker, V. L. Parsons, 

W. W. Davis, K. W. Dodd, and E. J. Feuer (2007). 

Combining information from two surveys to estimate 

county-level prevalence rates of cancer risk factors and 

screening. Journal of the American Statistical Association

102(478), 474–486.

Rao, J. N. K. (2020). On making valid inferences by 

integrating data from surveys and other sources. Sankhya

B 83(1), 242–272.

Renssen, R. H. and N. J. Nieuwenbroek (1997). Aligning 

estimates for common variables in two or more sample 

surveys. Journal of the American Statistical Association

92(437), 368–374.

Rubin, D. B. (1976). Inference and missing data. 

Biometrika 63(3), 581–592.

Schenker, N., T. E. Raghunathan, and I. Bondarenko

(2010). Improving on analyses of self-reported data in a 

large-scale health survey by using information from an 

examination-based survey. Statistics in Medicine 29(5), 

533–545.

Shao, J. and R. R. Sitter (1996). Bootstrap for imputed 

survey data. Journal of the American Statistical 

Association 91(435), 1278–1288.

Shin, H.-C., J. Parker, V. Parsons, Y. He, K. Irimata, B. 

Cai, and V. Beresovsky (2022). Propensity-score 

adjusted estimates for selected health outcomes from 
29

References - IV

Shin, H.-C., J. Parker, V. Parsons, Y. He, K. Irimata, B. 

Cai, and V. Beresovsky (2022). Propensity-score adjusted 

estimates for selected health outcomes from the research 

and development survey. Vital and Health statistics. Ser. 

1, Programs and Collection Procedures (196), 1–20.

Valliant, R. (2020). Comparing alternatives for estimation 

from nonprobability samples. Journal of Survey Statistics 

and Methodology 8(2), 231–263.

Wang, L., B. I. Graubard, H. A. Katki, and Y. Li (2022). 

Efficient and robust propensity-score-based methods for 

population inference using epidemiologic cohorts. 

International Statistical Review 90(1), 146–164.

Wu, C. (2004). Combining information from multiple 

surveys through the empirical likelihood method. Canadian 

Journal of Statistics 32(1), 15–26.

Wu, C. (2022). Statistical inference with non-probability 

survey samples. Survey Methodology 48(2), 283–311.

Yang, M., N. Ganesh, E. Mulrow, and V. Pineau (2019). 

Evaluating estimation methods for combining probability 

and nonprobability samples through a simulation study. 

In JSM Proceedings, Survey Research Methods 

Section, 1714–1727. Alexandria, VA: American 

Statistical Association.

Yang, S. and J. K. Kim (2020). Statistical data 

integration in survey sampling: A review. Japanese 

Journal of Statistics and Data Science 3(2), 625–650.

Ybarra, L. M. and S. L. Lohr (2008). Small area 

estimation when auxiliary information is measured with 

error. Biometrika 95(4), 919–931.