Generating survey weights using a

machine-learning method and the entropy

balancing calibration technique

Joseph Kang, Darcy Morris, Patrick Joyce, Isaac Dompreh

US Census Bureau

October 6, 2022
Overview

Raked inverse of probability weighting (IPW) method

Review of IPW

What is IPW?

A machine learning-based weights wgt

Raking methods based on causal inference

The entropy balancing technique [Hainmueller, 2012]

Simulation study

1000 simulated samples for ebal

1000 simulated samples for gbm

1000 simulated samples for balancing results

Summary
Pew Institute’s example

▶ Raking alone was reported to be inferior to combined methods
Review of the IPW weight construction

▶ Let R denote a subgroup indicator including a binary response

condition, X a vector of covariates, and Y a continuous

outcome of interest.

▶ Weights wgt to equate covariate distributions among different

populations

wgt × P(X|R = 1) = P(X).

▶ IPW wgt ≈

1

P(R=1|X)

.
Interpretation of IPW weights

ID R X Y Y 1 Y 0

1 1 X1 Y1 Y1 ??

2 1 X2 Y1 Y1 ??

. . . . . . . . . . . . . . . . . .

m 1 Xm Ym Ym ??

m + 1 0 Xm+1 Ym ?? Ym

. . . . . . . . . . . . . . . . . .

n 0 Xn Yn ?? Yn

n: sample size, m: Σn

i=1(Ri = 1), ??: missing Y

▶ wgt to weight the subgroup (R = 1) data to make it look like

the entire data set, i.e., both R = 1 and R = 0.

▶ Y 1: the potential outcome of Y that could have been

observed if every unit had R = 1.

▶ E(Y 1) =

Pwgt

P ×R×Y

wgt and E(Y 0) =

Pwgt

P×(1−R)×Y

wgt .

▶ Average causal effect: E(Y 1) − E(Y 0) under reasonable

assumptions [Robins et al., 2000].
IPW for balancing multivariate covariate distributions

▶ Correct IPW-based weights (wgt) are expected to hold:

wgt × P(X|R = 1) = P(X).

▶ X is the p multivariate covariates X1,X2,...,Xp.

▶ wgt is to weight and make the multivariate distribution

P(X|R = 1) approximate P(X).

▶ Causal inference and missing data methodologies that use

IPW is based on this notion [Miguel A. Hern ́an, 2020].

▶ wgt has a clear purpose to make a subset of the data set

become representative of a target sample with respect to the

multivariate distribution of X, not a univariate distribution of

each variable in X = X1, X2,...,Xp.
An important condition for wgt to correctly balance

▶ E(wgt × R × X1) = E(X1).

The weighted average of the single covariate X1 should be the

average of X1 for the entire sample (R = 1 and R = 0).

▶ wgt aims to balance the multivariate covariate distributions

between R = 1 and the entire sample, but checking this

balancing property has been done in an univariate sense

[Austin, 2011].

▶ Even though balancing multivariate distribution of X is a

principal task of IPW-based weights wgt, it is based on a

response propensity model P(R = 1|X), which is a single

scalar value and a one-dimensional summary of X.

▶ Because IPW weights wgt require a very ‘good’ response

propensity score, careful modeling is needed.
Machine-learning method: generalized boosted

model(GBM)

▶ IPW weights heavily rely on the correctness of the response

propensity model and have been built with machine-learning

methods including the GBM.

▶ R package twang [Cefalu et al., 2021], managed by RAND

institute’s staff, builds GBM-based responses propensity

models in causal inference.

▶ R package twang automates the process of creating IPW

weights with the GBM-based response propensity model.

▶ GBM can handle high dimensionality, colinearity, nonlinearity,

missing data, and complex interactions among covariates.

▶ GBM does not require an assumption that the logistic

regression model needs, and hence it is nonparametric (a

black box problem).

▶ GBM reports the relative influence of each covariates at a

percent scale in a nonparametric way.
What if wgt is incorrect?

▶ wgt (from IPW) is expected to hold the univariate balancing

condition

E(wgt × R × Xk ) = E(Xk ),

for k=1, . . . , p.

▶ If this does not hold, then the response propensity is

considered incorrectly specified for wgt.

▶ A calibration is needed for wgt to hold the equality above.

▶ Survey statisticians’ raking calibration creates such additional

weight to enable the balancing equality.
The entropy balancing (ebal) technique [Hainmueller,

2012]

▶ Ebal has been extensively used in causal inference

▶ Ebal aims to attain the p-univariate balancing conditions

wgt × P(X1|R = 1) = P(X1),

wgt × P(X2|R = 1) = P(X2),

. . .

wgt × P(Xp|R = 1) = P(Xp),

▶ Recall that the IPW method aims to meet the multivariate

balancing condition for the multivariate p covariates

X1, X2, ..., Xp

wgt × P(X|R = 1) = P(X)
The entropy balancing (ebal) technique: continued

▶ R package ebal [Hainmueller, 2022] minimizes the following

objective function to re-weight ‘working’ IPW weights wgt by

creating a new weight wgtebal = wgt × a(x), where a(x) is an

additional factor that helps wgt meet univariate balancing

properties.

X

respondents



wgtebal log(wgtebal/wgt) +X

p

j=1

λj[wgtebalXj − mj] + ...





▶ wgtebal aims to attain the p univariate balanced distributions,

while the IPW-based weights wgt aim to attain the

multivariate version of balanced distribution of all covariates

X.

▶ Ebal has been extensively used as a raking method for causal

inference in Economics.
Kang and Schafer [2007] (KS) simulation study design as a

check point to evalute methods

▶ KS designed the following simulation to critically evaluate

methodologies for causal inference and missing data analysis.

y = 210 + 27.4z1 + 13.7z2 + 13.7z3 + 13.7z4 + ε,

π = 1 − (1 + exp(−z1 + 0.5z2 − 0.25z3 − 0.1z4))−1,

x1 = exp(z1/2), x2 = z2/(1 + exp(z1)) + 10, x3 = (z1z3/25 + 0.6)3,

x4 = (z2 + z4 + 20)2, and ε and z’s were from standard normal

distributions.

▶ z’s are not available but x‘s are available to analysts.

▶ x’s are reasonable predictors for y and r in lieu of z.
KS simulation study: continued

▶ KS design

ID R X Y Y 1

1 1 X1 Y1 Y1

2 1 X2 Y1 Y1

. . . . . . . . . . . . . . .

m 1 Xm Ym Ym

m + 1 0 Xm+1 Ym ??

. . . . . . . . . . . . . . .

n 0 Xn Yn ??

n: sample size, m: Σn

i=1(Ri = 1),

??: missing Y

▶ Y 1 is a potential outcome that could have been observed if a

unit had responded (R = 1).

▶ E(Y 1) = 210 is the value to be inferred with X’s and Y

within the subgroup of R = 1.
KS study results for 1000 samples

Weights MBias VBias RMSE MAE RNG

wgt ebal 1.881 2.310 2.418 1.894 -1.005:4.966

wgt gbm1 2.997 1.806 3.285 2.973 0.36:5.666

wgt gbm2 3.394 1.745 3.642 3.383 0.789:5.972

wgt gbm1ebal 1.271 1.879 1.869 1.365 -1.342:4.024

wgt gbm2ebal 0.880 1.754 1.590 1.098 -1.726:3.514

ols 0.725 2.263 1.669 1.159 -2.213:3.693

bias = 210 − est

MBias =

1

103

Pbias

VBias =

1

103−1

P(bias − MBias)

2

RMSE =

q

1

103

Pbias2

MAE = median of abs(bias)

RNG = 2.5% − 97.5%

▶ ‘ols’ indicates ordinal least square regression estimates in

fitting a regression for units with R = 1 and use its estimated

coefficients to predict the entire sample.
Ebal results

Weights MBias VBias RMSE MAE RNG

wgt ebal 1.881 2.310 2.418 1.894 -1.005:4.966

wgt gbm1ebal 1.271 1.879 1.869 1.365 -1.342:4.024

wgt gbm2ebal 0.880 1.754 1.590 1.098 -1.726:3.514

ols 0.725 2.263 1.669 1.159 -2.213:3.693

bias = 210 - est; MBias= 1

103

P bias; VBias = 1

103−1

P(bias − MBias)

2

; RMSE = q 1

103

P bias2; MAE =

median of abs(bias); RNG = 2.5% - 97.5%

▶ The simple ols method was reported to be the most unbiased

method by Kang and Schafer [2007].

▶ Ebal’s MBias was improved with GBM.

▶ wgt gbm2ebal outperformed the ols in RMSE, MAE, and

RNG!

▶ ebal alone is inferior to ebal-raked GBM IPW.
GBM results

Weights MBias VBias RMSE MAE RNG

wgt gbm1 2.997 1.806 3.285 2.973 0.36:5.666

wgt gbm2 3.394 1.745 3.642 3.383 0.789:5.972

wgt gbm1ebal 1.271 1.879 1.869 1.365 -1.342:4.024

wgt gbm2ebal 0.880 1.754 1.590 1.098 -1.726:3.514

ols 0.725 2.263 1.669 1.159 -2.213:3.693

bias = 210 - est; MBias= 1

103

P bias; VBias = 1

103−1

P(bias − MBias)

2

; RMSE = q 1

103

P bias2; MAE =

median of abs(bias); RNG = 2.5% - 97.5%

▶ GBM, if with ebal, worked better than itself.

▶ GBM worked worse when interaction.depth increased from 1

to 2.

▶ GBM, with ebal, worked better when interaction.depth

increased from 1 to 2.
1000 simulated samples for balancing results I

Table: Averages of variables

method X ̄1 X ̄2 X ̄3 X ̄4 E(Y (1))

all 1.134 10.000 0.219 401.985 210.023

wgt ebal 1.134 10.000 0.219 401.985 208.119

wgt gbm1 1.083 10.020 0.217 403.406 207.003

wgt gbm2 1.068 10.025 0.218 403.224 206.606

wgt gbm1ebal 1.134 10.000 0.219 401.985 208.729

wgt gbm2ebal 1.134 10.000 0.219 401.985 209.120

▶ GBM methods alone (wgt gbm1 and wgt gbm2) did not

completely balance covariate distributions.
1000 simulated samples for balancing results II

Table: Averages of standardized difference of variables

method x1 x2 x3 x4

wgt ebal 0.394 0.530 0.050 0.443

wgt gbm1 2.877 3.236 1.331 1.348

wgt gbm2 4.165 3.894 1.051 1.189

wgt gbm1ebal 0.440 0.438 0.265 0.426

wgt gbm2ebal 0.031 0.032 0.032 0.045

▶ A standardized difference:

d =

∆ ̄

q

B

PB

b=1(∆b − ∆ ̄

B )

2/(B − 1)

, B=1000

where ∆ ̄

B =

1

B

PB

b=1 ∆b, ∆b is ( ̄x1b − x ̄b),  ̄x1b indicates the

sample mean of covariate among R = 1 and  ̄xb is for all units

in the b

th bootstrap (simulation) sample.

▶ d < 0.1 implies that univariate balancing conditions were met

by negligible differences in the mean of a covariate between

the subgroup with R = 1 and all [Austin, 2011].
Summary

▶ Calibrated machine learning-based IPW weights produced as

reasonable estimates as the ols method in inferring the average

of an outcome with missing data via the KS simulation study.

▶ The GBM machine-learning method was used for IPW to

meet the multivariate balancing condition.

▶ The ebal method was used to meet the univariate balancing

conditions.

▶ Either GBM alone or ebal alone is inferior to the combined

method.

▶ The combined method used 1) GBM to meet the multivariate

condition, which is hard to check, and 2) ebal as ‘insurance’

to complete the univariate conditions.
References

Jens Hainmueller. Entropy balancing for causal effects: A

multivariate reweighting method to produce balanced samples in

observational studies. Political Analysis, 20(1):25–46, 2012.

Reducing bias on benchmarks. https://www.pewresearch.org/

methods/2018/01/26/reducing-bias-on-benchmarks/.

Accessed: 2022-08-30.

J. M. Robins, M. A. Hern ́an, and B. Brumback. Marginal

structural models and causal inference in epidemiology.

Epidemiology, 11(5):550–560, Sep 2000.

James M. Robins Miguel A. Hern ́an. Causal Inference: What If.

Chapman & Hall/CRC, Boca Raton, 2020.

P. C. Austin. An Introduction to Propensity Score Methods for

Reducing the Effects of Confounding in Observational Studies.

Multivariate Behav Res, 46(3):399–424, May 2011.

Matthew Cefalu, Greg Ridgeway, Dan McCaffrey, Andrew Morral,

Beth Ann Griffin, and Lane Burgette. twang: Toolkit for

Weighting and Analysis of Nonequivalent Groups, 2021. URL

https://CRAN.R-project.org/package=twang. R package